Wrong orientation?

[paulgrassler]

Hey, I have been trying out combining the SURREACT dataset with the STAR model and can not seem to get satisfying results. From what I can see this is very similar to the problem described here: gulvarol/surreal#10 (comment)

However, I implemented the code regarding the rotation and still the generated meshes seems to have the wrong orientation. Tested this with a number of different examples but the rotation always seems to be off (without me being able to notice any similarity in the way they are oriented). Included the used code below, would appreciate any kind of help on this matter.

---

import math
from star.pytorch.star import STAR
import numpy as np
import torch
import open3d
import scipy.io
import transforms3d

def rotateBody(RzBody, pelvisRotVec):

pelvisRotVec = pelvisRotVec.cpu().detach().numpy()
angle = np.linalg.norm(pelvisRotVec)
Rpelvis = transforms3d.axangles.axangle2mat(pelvisRotVec / angle, angle, True)
globRotMat = np.dot(RzBody, Rpelvis)
R90 = transforms3d.euler.euler2mat(np.pi / 2, 0, 0)
globRotAx, globRotAngle = transforms3d.axangles.mat2axangle(np.dot(R90,globRotMat))
globRotVec = globRotAx * globRotAngle
return torch.cuda.FloatTensor(globRotVec)

num_betas=10
batch_size=1
m = STAR(gender='male',num_betas=num_betas)

mat = scipy.io.loadmat('/SURREACT_DATA/a37_d1_p019_c1_color.avi_v225_r00_info.mat')

betas = torch.cuda.FloatTensor(mat['shape'])
trans = torch.cuda.FloatTensor(np.zeros((batch_size, 3)))

max_frame = torch.cuda.FloatTensor(mat['joints3D']).shape[2]

for x in range(0, max_frame+1, 2):

pose = torch.cuda.FloatTensor(mat['pose'][:, :, x])
betas = torch.cuda.FloatTensor(mat['shape'])

zrot = mat['zrot_euler'][0][0]
zrot = math.radians(zrot)
RzBody = np.array(((math.cos(zrot), -math.sin(zrot), 0),
                   (math.sin(zrot), math.cos(zrot), 0),
                   (0, 0, 1)))
pose[0, 0:3] = rotateBody(RzBody, pose[0, 0:3])

model = m.forward(pose, betas, trans)
shaped = model.v_shaped[-1, :, :]

# 3d visualization with open3d
mesh = open3d.geometry.TriangleMesh()
vertices_list = np.array(model[0].tolist())
faces_list = np.array(model.f.tolist()).astype(np.int32)
mesh.vertices = open3d.utility.Vector3dVector(vertices_list)
mesh.triangles = open3d.utility.Vector3iVector(faces_list)
open3d.visualization.draw_geometries([mesh])

[gulvarol]

I unfortunately may not be helpful to provide support for the STAR model as I have not used it before. Maybe you can contact the authors of the paper.