How to generate a SMPL body aligned with the scan?

[sunshineatnoon]

Hi, Thanks for providing this awesome dataset.

I am trying to use the SMPL parameters and scan mesh in my project. I tried to produce a SMPL mesh that aligns with the scan

mesh, the code is like this

def create_smpl(pose, betas, trans, device, smpl_scale_in=None):
    bm_dir_path = "./models/"
    body_model = smplx.create(
        model_path=bm_dir_path, gender="neutral", model_type='smpl', batch_size=1, num_betas=10
    )
    body_model = body_model.to(device=device)

    # Prepare inputs
    pose_np = np.asarray(pose)
    betas_t = torch.from_numpy(np.asarray(betas).reshape(1, 10).astype(np.float32)).to(device=device)
    body_pose = torch.from_numpy(pose_np[3:].astype(np.float32).reshape(1, 69)).to(device=device)
    global_orient = torch.from_numpy(pose_np[:3].astype(np.float32).reshape(1, 3)).to(device=device)
    smpl_trans = np.asarray(trans)  # numpy (3,)
    if smpl_scale_in is None:
        smpl_scale = np.array(1.0, dtype=np.float32)  # default scale if not provided
    else:
        smpl_scale = np.asarray(smpl_scale_in, dtype=np.float32)

    # Full pose for saving
    full_pose = torch.cat([global_orient, body_pose], dim=-1)[0].detach().cpu().numpy()

    # Forward SMPL without translation; apply scale+trans like prep_scan.py
    body_model_output = body_model.forward(
        betas=betas_t, body_pose=body_pose, global_orient=global_orient
    )
    smpl_verts = body_model_output.vertices  # [1, V, 3]
    smpl_verts = smpl_verts * torch.from_numpy(smpl_scale).to(device=device)
    smpl_verts = smpl_verts + torch.from_numpy(smpl_trans.astype(np.float32)).to(device=device)

    # Faces from the SMPL model
    smpl_faces_np = body_model.faces_tensor.detach().cpu().numpy()

    return smpl_verts, smpl_scale, smpl_trans, full_pose, betas_t, smpl_faces_np

I found the smpl mesh and scan are misaligned, is there any more transformation I need to apply? Thanks for your time!

[sunshineatnoon]

Oh, I figured it out, the order of translation and scaling is wrong, it should be

    smpl_verts = smpl_verts + torch.from_numpy(smpl_trans.astype(np.float32)).to(device=device)
    smpl_verts = smpl_verts * torch.from_numpy(smpl_scale).to(device=device)

However, I am seeing a lot penetration between the body and scan, is this expected?

[umariqbal-nvidia]

@anticdimi we are having the same issue. Do you have any example code to reproduce the dataset. Currently, SMPL and garments are not aligned on our end.

[anticdimi]

Hi @sunshineatnoon @umariqbal-nvidia ,

In certain scans, as the one you show above, SMPL-to-scan registration indeed produced params which are not the best possible fit.

As a quick workaround, maybe you could try using this method and see if it will produce better fits?

We will try to investigate this further.

Best,
Dimitrije

[sunshineatnoon]

Hi @anticdimi Thanks for the quick reply. As you mentioned here, there is a gender label for each scan. I couldn't find it in the npz files. Are the gender labels available somewhere and are they important for the body and garment alignment?