[v0.1.0] 6. Before Rigging
MYTY Kit’s Motion Tracking Solution
Understanding the overall Motion Tracking process is required before the Bone Rigging, Controller, and MotionAdapter steps. It is recommended that you first read the Prerequisites page.
MYTY Kit supports motion tracking using Google's MediaPipe, a machine learning solution for live and streaming media. In particular, MYTY Kit utilizes MediaPipe's Face Mesh solution to track facial expressions in real-time. Since machine learning is employed to infer the position of facial and upper body landmarks as 3D coordinates, motion tracking requires just a single camera input, without the need for a dedicated depth sensor.
The process of taking this 3D coordinate information and connecting it to the 2D NFT Avatar can be broken down into smaller steps of bone rigging, Controller, and MotionAdapter. Before you proceed further, please familiarize yourself with the following key terms.
| Terms | Definition |
|---|---|
MediaPipe | Google’s machine learning solution that tracks user’s real-time movements on camera as 3D Motion tracking data. |
Motion tracking data (Motion data) | Movement data tracked by MediaPipe. |
Motion tracking module | A module for tracking a subset of movements with MediaPipe. |
Motion Adapter | A MYTY Kit tool that processes motion tracking data and connects it to another MYTY Kit tool, the Controller. |
Controller | A MYTY Kit Tool that controls the Bones and Sprites (Image) of MYTY Avatars. |
MYTY Avatar | An avatar created using the MYTY Kit. It is both a Live Avatar and an NFT Avatar, and is compatible with the MYTY Ecosystem. |
Bone rigging | The process of connecting Bones to the 2D Avatar Image to enable joint movements. |
1️⃣ Here’s an overview of how motion tracking data moves the avatar.
→ How does MYTY Kit handle motion data?
2️⃣ Find out which types of motion tracking data can be collected.
→ Motion tracking modules (included in MYTY Kit)
3️⃣ Find out how collected motion tracking data is sent to the avatar.
→ The Controller-MotionAdapter Relationship
4️⃣ Choose the appropriate Controller and MotionAdapter for the type of animation you want to create.
How does MYTY Kit handle motion data?
MYTY Kit tracks the user's facial expression and movement using MediaPipe and delivers this information to the MYTY Avatar through the Motion Adapter and Controller tools.
MediaPipe tracks the user’s movement and presents it as raw motion data.
→ This raw motion data is processed by the Motion Adapter and delivered to the Controller.
→ The Controller uses the processed motion data to affect applicable bones and sprites.
→ A MYTY Avatar that mirrors the user’s movements is created.
Motion tracking modules (included in MYTY Kit)
The MYTY Kit provides the following motion tracking modules. Any movements within a module can be tracked and applied to MYTY Avatar.
With the MYTY Kit v0.1, only upper body movements (Head & Face, Chest) can be applied to the MYTY Avatar. If you’d like to use more tracking modules, please contact us through the Community Channel.
Modules included in MYTY Kit v1.0
Head & Face: Module for tracking head and facial muscle movements
Chest: Module for tracking torso movement
Modules to be included in future versions
Hips: Module for tracking movement of the hip joints
Neck: Module for tracking movements of cervical spine joints
LeftUpperArm & LeftLowerArm: Module for tracking movement of joints in the left arm
RightUpperArm & RightLowerArm: Module for tracking movement of joints in the right arm
LeftUpperLeg & LeftLowerLeg: Module for tracking movement of joints in the left leg
RightUpperLeg & RightLowerLeg: Module for tracking movement of joints in the right leg
LeftHand: Module for tracking movement of finger joints in the left hand
LeftWrist: Module for tracking movement of joints in the left wrist
RightHand: Module for tracking movement of finger joints in the right hand
RightWrist: Module for tracking movement of joints in the right wrist
You can check the input values in real-time: Play Unity Project, select MediaPipe > RiggingElement > Module Object. Input values are displayed on the Inspector panel.
👉 Related Document: How to add MediaPipe prefab to Hierarchy
The Controller-MotionAdapter Relationship
Which Controllers and MotionAdapters will you need to make your desired animation?
The MYTY Kit tools
ControllerandMotionAdapterconnect the user’s Motion tracking data to their MYTY Avatar.MotionAdapter is a tool that processes raw Motion input and delivers it to the related Controller.
Controller is a tool that uses the processed Motion input to control an avatar’s Bones and Sprites.
Let’s find out more about the four types of Controllers included in the MYTY Kit: what they do, and which MotionAdapters they can be used with.
| Bone1D Controller | Bone2D Controller | Sprite1D Controller | Sprite2D Controller | |
|---|---|---|---|---|
Input | The float value of 1 MotionAdapter output. | The float values of 2 MotionAdapter outputs. | The float value of 1 MotionAdapter output. | The float values of 2 MotionAdapter outputs. |
Output | Transformation of connected bone(s). | Transformation of connected bone(s). | Selection of appropriate sprite image(s) to display. | Selection of appropriate Sprite image(s) to display. |
Related MotionAdapter | Bone2DZAdapter EyeCoverAdapter | Bone2DAdapter EyeBoneAdapter | EyeSpriteAdapter | EyeBrowAdapter MouthSpriteAdapter |
Usage Example |
|
|
|
|
👉 Please refer to 8. MYTY Kit Controllers for more information on the Controller.
👉 Please refer to [v0.1.0] 9. Apply Motion Tracking for more information on the MotionAdapter.
Usage Example
There are two ways to making an eye-blinking animation.
1. Using Bone Animation
You can make it using the Bone 1D Controller and EyeBone Adapter
Rig the EyelidBone to the Eyelid Sprite.
Create an empty GameObject (Naming example: EyeBoneController) within Controller Group, and add the Bone 1D Controller Component.
Open Bone 1D Controller Window. Specify EyeBoneController, and add EyelidBone Object to the list in the Hierarchy window.
Save the EyelidBone’s positions for when the eye is open (Max) and closed (Min).
Add EyeBone Adapter Prefab to Hierarchy. Specify Face model and EyeBoneController to the Model and Controller, respectively.
Press the Project play button to check that the animation works as intended.
2. Using Frame Animation
You can make it using the Sprite 1D Controller and EyeSprite Adapter.
Create an empty GameObject (Naming example: EyeBoneController) within Controller Group, and add the Sprite 1D Controller Component.
Open Sprite 1D Controller Window. Specify EyeBoneController, and add Eyelid Object to the list in the Hierarchy window.
Press Auto Label to automatically bring up Sprite Library Label.
Assign Min and Max values to the Sprite Label for when the eye is open, to determine the data range within which an open-eye Sprite will be sent.
Assign Min and Max values to the Sprite Label for when the eye is closed, to determine the data range within which a closed-eye Sprite will be sent.
Add EyeBone Adapter Prefab to Hierarchy. Specify Face model and EyeBoneController to the Model and Controller, respectively.
Press the Project play button to check that the animation works as intended.
Have a question or an idea? If you have a question to ask or an idea to share, participate in the MYTY Kit Community. We’d love to hear from you.😄
Last updated
