2. Customizing SynthDa
This section outlines the various customization options available across the AutoSynthDa pipeline. From adjusting model parameters and camera angles to modifying default prompts and rendering styles, these settings provide flexibility to fine-tune the behavior and output of each component to better suit your use case.
The accuracy of how closely the joints are fitted can be adjusted using the number of iterations. Higher values lead to more accurate results but also increase the computation time. A value of 15 is recommended for a balance between speed and quality.
You can modify this using the --num_smplify_iters parameter when running the command, or by manually editing the fit_seq.py file within the joints2smpl repository.
When the LLM (e.g. OpenAI's ChatGPT) fails to recognize or generate a valid action from the input video, a fallback action prompt is used. This fallback is defined by the action_class variable in :
./video_processing/models.py
It is then injected into the system prompt when calling the LLM API:
If your video files are in formats other than .mp4 (e.g., .mov, .avi), you must update the file extension references in the pipeline scripts.
Replace all occurrences of .mp4 in process_real2synth_pipeline.py with your desired file format:
Do the same for process_real2real_pipeline.py:
.mp4 files by default.
The model used in the StridedTransformer-Pose3D repository is trained on the Human3.6M (H36M) dataset, which contains approximately 3.6 million frames. This dataset defines 17 joints for 3D human pose estimation.
However, both SMPL and text-to-motion require 24 joints, which means a mapping between the 17-joint H36M format and the 24-joint SMPL format is necessary.
You can adjust this mapping in the following function:
./AutoDa/optimisation/optimisation_utils.py → map_h36m_to_smpl()
SMPL Joint Structure (Reference) defined in joints2mpl/src/config.py
📄 You can find the official documentation of the 17 Human3.6M joints here.
The interpolation of pose points between the two sources—whether from synthetic data or a second real video—is handled in the compute_P_opt() function.
A key parameter, alpha determines the weight of interpolation:
- A higher alpha biases more towards the second input (synthetic or real).
- A lower alpha gives more emphasis to the first real video’s data.
Currently, alpha is fixed at 0.5 in both the real_syn and real_real optimization scripts.
This creates a balanced contribution between the two data sources.
You can find and adjust this parameter in the respective calls to compute_P_opt() within:
AutoDa/optimisation/optimisation_real_synth.pyAutoDa/optimisation/optimisation_both_real.py
Optimisation logic is defined in compute_P_opt() within ./optimisation/optimisation_utils.py, where the real and synthetic poses are blended.
This optimisation function is invoked in both the real-synthetic and real-real pipelines, the alpha that controls the interpolation strength can be adjusted in here.
The number of video pairs for real-to-real processing can exceed the number of individual uploaded videos. This is because the combinations are formed using pairwise comparisons.
The both_real_main() function accepts a parameter number_of_videos, which determines how many combinations will be generated. The number of valid pairs is based on combinatorial logic.
For example, given three videos:
[video_1, video_2, video_3]
You can generate the following 3 combinations:
[video_1, video_2][video_2, video_3][video_1, video_3]
This logic ensures flexible scaling when testing or training across multiple real video inputs.
Blender is used to generate the video from the .ply objects generated by the joints2smpl repository. The custom Python script, animation_pose.py, imports these .ply files, places them according to the frame they represent (e.g., 0001 for the first frame, 0002 for the second, and so on), and exports them as a video.
Further adjustments can be made directly within the animation_pose.py script to suit specific visualization or rendering needs.
The environment and visual settings in Blender can be adjusted within the scene_addition() function. This includes:
- 👤 Character Color (line 207)
- 🪵 Floor Color (line 222)
- 🌫️ Background Color (line 230)
You can modify these RGBA values to customize the appearance of the rendered animation.
You can customize the number and type of angles rendered in Blender by editing the angleInput.txt file located in the Blender folder.
The available camera angles you can specify are:
frontbackleftfront-leftfront-rightupper-leftupper-right
These values are processed by the set_camera_view() function during rendering to set up the appropriate camera perspectives.
The following parameters can be adjusted to fine-tune the camera setup:
-
LeftRight-adjust: Modifies the camera's placement along the X-axis -
closeness-adjust: Controls how close the camera is to the subject -
height-adjust: Adjusts the camera's height along the Z-axis -
camera-angle: Changes the tilt angle of the camera
These variables are also handled within the set_camera_view() function to give more flexibility in rendering the scene.
The FBX export feature enables the animation and 3D object to be baked together into a single file. This file is automatically saved within the Blender generation folder after rendering.
FBX files are widely supported and can be seamlessly imported into other tools such as:
- 🎮 Unity
- 🎥 Unreal Engine
- 🎨 Autodesk Maya
- 🌐 NVIDIA Omniverse
This enhances interoperability across pipelines for animation, simulation, or game development workflows.
The .fbx file generated will be saved inside the folder named after the video, which contains the .ply objects produced by joints2smpl.
Export settings such as resolution, frames per second (fps), and video file format can be adjusted in the export_video function.
This provides control over the output quality and compatibility of the rendered videos. The .mp4 file generated will be saved inside the folder named after the video, which also contains the .ply objects produced by joints2smpl.