[v0.1.0] 6. Before Rigging

MYTY Kit’s Motion Tracking Solution

Understanding the overall Motion Tracking process is required before the Bone Rigging, Controller, and MotionAdapter steps. It is recommended that you first read the Prerequisites page.

MYTY Kit supports motion tracking using Google's MediaPipe, a machine learning solution for live and streaming media. In particular, MYTY Kit utilizes MediaPipe's Face Mesh solution to track facial expressions in real-time. Since machine learning is employed to infer the position of facial and upper body landmarks as 3D coordinates, motion tracking requires just a single camera input, without the need for a dedicated depth sensor.

The process of taking this 3D coordinate information and connecting it to the 2D NFT Avatar can be broken down into smaller steps of bone rigging, Controller, and MotionAdapter. Before you proceed further, please familiarize yourself with the following key terms.

1️⃣ Here’s an overview of how motion tracking data moves the avatar.

How does MYTY Kit handle motion data?

2️⃣ Find out which types of motion tracking data can be collected.

Motion tracking modules (included in MYTY Kit)

3️⃣ Find out how collected motion tracking data is sent to the avatar.

The Controller-MotionAdapter Relationship

4️⃣ Choose the appropriate Controller and MotionAdapter for the type of animation you want to create.

Usage Example

How does MYTY Kit handle motion data?

MYTY Kit tracks the user's facial expression and movement using MediaPipe and delivers this information to the MYTY Avatar through the Motion Adapter and Controller tools.

  • MediaPipe tracks the user’s movement and presents it as raw motion data.

    → This raw motion data is processed by the Motion Adapter and delivered to the Controller.

    → The Controller uses the processed motion data to affect applicable bones and sprites.

    → A MYTY Avatar that mirrors the user’s movements is created.

Motion tracking modules (included in MYTY Kit)

  • The MYTY Kit provides the following motion tracking modules. Any movements within a module can be tracked and applied to MYTY Avatar.

With the MYTY Kit v0.1, only upper body movements (Head & Face, Chest) can be applied to the MYTY Avatar. If you’d like to use more tracking modules, please contact us through the Community Channel.

Modules included in MYTY Kit v1.0

  • Head & Face: Module for tracking head and facial muscle movements

  • Chest: Module for tracking torso movement

Modules to be included in future versions

  • Hips: Module for tracking movement of the hip joints

  • Neck: Module for tracking movements of cervical spine joints

  • LeftUpperArm & LeftLowerArm: Module for tracking movement of joints in the left arm

  • RightUpperArm & RightLowerArm: Module for tracking movement of joints in the right arm

  • LeftUpperLeg & LeftLowerLeg: Module for tracking movement of joints in the left leg

  • RightUpperLeg & RightLowerLeg: Module for tracking movement of joints in the right leg

  • LeftHand: Module for tracking movement of finger joints in the left hand

  • LeftWrist: Module for tracking movement of joints in the left wrist

  • RightHand: Module for tracking movement of finger joints in the right hand

  • RightWrist: Module for tracking movement of joints in the right wrist

You can check the input values in real-time: Play Unity Project, select MediaPipe > RiggingElement > Module Object. Input values are displayed on the Inspector panel.

👉 Related Document: How to add MediaPipe prefab to Hierarchy


The Controller-MotionAdapter Relationship

Which Controllers and MotionAdapters will you need to make your desired animation?

  • The MYTY Kit tools Controller and MotionAdapter connect the user’s Motion tracking data to their MYTY Avatar.

    • MotionAdapter is a tool that processes raw Motion input and delivers it to the related Controller.

    • Controller is a tool that uses the processed Motion input to control an avatar’s Bones and Sprites.

  • Let’s find out more about the four types of Controllers included in the MYTY Kit: what they do, and which MotionAdapters they can be used with.

👉 Please refer to 8. MYTY Kit Controllers for more information on the Controller.

👉 Please refer to [v0.1.0] 9. Apply Motion Tracking for more information on the MotionAdapter.

Usage Example

There are two ways to making an eye-blinking animation.

You can make it using the Bone 1D Controller and EyeBone Adapter

  1. Rig the EyelidBone to the Eyelid Sprite.

  2. Create an empty GameObject (Naming example: EyeBoneController) within Controller Group, and add the Bone 1D Controller Component.

  3. Open Bone 1D Controller Window. Specify EyeBoneController, and add EyelidBone Object to the list in the Hierarchy window.

  4. Save the EyelidBone’s positions for when the eye is open (Max) and closed (Min).

  5. Add EyeBone Adapter Prefab to Hierarchy. Specify Face model and EyeBoneController to the Model and Controller, respectively.

  6. Press the Project play button to check that the animation works as intended.

You can make it using the Sprite 1D Controller and EyeSprite Adapter.

  1. Create an empty GameObject (Naming example: EyeBoneController) within Controller Group, and add the Sprite 1D Controller Component.

  2. Open Sprite 1D Controller Window. Specify EyeBoneController, and add Eyelid Object to the list in the Hierarchy window.

  3. Press Auto Label to automatically bring up Sprite Library Label.

  4. Assign Min and Max values to the Sprite Label for when the eye is open, to determine the data range within which an open-eye Sprite will be sent.

  5. Assign Min and Max values to the Sprite Label for when the eye is closed, to determine the data range within which a closed-eye Sprite will be sent.

  6. Add EyeBone Adapter Prefab to Hierarchy. Specify Face model and EyeBoneController to the Model and Controller, respectively.

  7. Press the Project play button to check that the animation works as intended.

Last updated