[v0.1.0] 6. Before Rigging

MYTY Kit’s Motion Tracking Solution

Understanding the overall Motion Tracking process is required before the Bone Rigging, Controller, and MotionAdapter steps. It is recommended that you first read the Prerequisites page.

MYTY Kit supports motion tracking using Google's MediaPipe, a machine learning solution for live and streaming media. In particular, MYTY Kit utilizes MediaPipe's Face Mesh solution to track facial expressions in real-time. Since machine learning is employed to infer the position of facial and upper body landmarks as 3D coordinates, motion tracking requires just a single camera input, without the need for a dedicated depth sensor.

The process of taking this 3D coordinate information and connecting it to the 2D NFT Avatar can be broken down into smaller steps of bone rigging, Controller, and MotionAdapter. Before you proceed further, please familiarize yourself with the following key terms.

Terms
Definition

MediaPipe

Google’s machine learning solution that tracks user’s real-time movements on camera as 3D Motion tracking data.

Motion tracking data (Motion data)

Movement data tracked by MediaPipe.

Motion tracking module

A module for tracking a subset of movements with MediaPipe.

Motion Adapter

A MYTY Kit tool that processes motion tracking data and connects it to another MYTY Kit tool, the Controller.

Controller

A MYTY Kit Tool that controls the Bones and Sprites (Image) of MYTY Avatars.

MYTY Avatar

Bone rigging

The process of connecting Bones to the 2D Avatar Image to enable joint movements.

1️⃣ Here’s an overview of how motion tracking data moves the avatar.

→ How does MYTY Kit handle motion data?

2️⃣ Find out which types of motion tracking data can be collected.

→ Motion tracking modules (included in MYTY Kit)

3️⃣ Find out how collected motion tracking data is sent to the avatar.

→ The Controller-MotionAdapter Relationship

4️⃣ Choose the appropriate Controller and MotionAdapter for the type of animation you want to create.

→ Usage Example

How does MYTY Kit handle motion data?

MYTY Kit tracks the user's facial expression and movement using MediaPipe and delivers this information to the MYTY Avatar through the Motion Adapter and Controller tools.

  • MediaPipe tracks the user’s movement and presents it as raw motion data.

    → This raw motion data is processed by the Motion Adapter and delivered to the Controller.

    → The Controller uses the processed motion data to affect applicable bones and sprites.

    → A MYTY Avatar that mirrors the user’s movements is created.

Motion tracking modules (included in MYTY Kit)

  • The MYTY Kit provides the following motion tracking modules. Any movements within a module can be tracked and applied to MYTY Avatar.

With the MYTY Kit v0.1, only upper body movements (Head & Face, Chest) can be applied to the MYTY Avatar. If you’d like to use more tracking modules, please contact us through the Community Channel.

Modules included in MYTY Kit v1.0

  • Head & Face: Module for tracking head and facial muscle movements

  • Chest: Module for tracking torso movement

Modules to be included in future versions

  • Hips: Module for tracking movement of the hip joints

  • Neck: Module for tracking movements of cervical spine joints

  • LeftUpperArm & LeftLowerArm: Module for tracking movement of joints in the left arm

  • RightUpperArm & RightLowerArm: Module for tracking movement of joints in the right arm

  • LeftUpperLeg & LeftLowerLeg: Module for tracking movement of joints in the left leg

  • RightUpperLeg & RightLowerLeg: Module for tracking movement of joints in the right leg

  • LeftHand: Module for tracking movement of finger joints in the left hand

  • LeftWrist: Module for tracking movement of joints in the left wrist

  • RightHand: Module for tracking movement of finger joints in the right hand

  • RightWrist: Module for tracking movement of joints in the right wrist

You can check the input values in real-time: Play Unity Project, select MediaPipe > RiggingElement > Module Object. Input values are displayed on the Inspector panel.

👉 Related Document: How to add MediaPipe prefab to Hierarchy

The Controller-MotionAdapter Relationship

Which Controllers and MotionAdapters will you need to make your desired animation?

  • The MYTY Kit tools Controller and MotionAdapter connect the user’s Motion tracking data to their MYTY Avatar.

    • MotionAdapter is a tool that processes raw Motion input and delivers it to the related Controller.

    • Controller is a tool that uses the processed Motion input to control an avatar’s Bones and Sprites.

  • Let’s find out more about the four types of Controllers included in the MYTY Kit: what they do, and which MotionAdapters they can be used with.

Bone1D Controller
Bone2D Controller
Sprite1D Controller
Sprite2D Controller

Input

The float value of 1 MotionAdapter output.

The float values of 2 MotionAdapter outputs.

The float value of 1 MotionAdapter output.

The float values of 2 MotionAdapter outputs.

Output

Transformation of connected bone(s).

Transformation of connected bone(s).

Selection of appropriate sprite image(s) to display.

Selection of appropriate Sprite image(s) to display.

Related MotionAdapter

Bone2DZAdapter EyeCoverAdapter

Bone2DAdapter EyeBoneAdapter

EyeSpriteAdapter

EyeBrowAdapter MouthSpriteAdapter

Usage Example

  • Using the user’s eyebrow movements in the vertical axis to set movements of the avatar’s eyebrow bone.

  • Using the user’s face movements in the vertical and horizontal axes (face direction) to set movements of the avatar’s FaceBone.

  • Using the user’s chest movements in the vertical and horizontal axes to set movements of the avatar’s BodyBone.

  • Using the user’s eye size in the vertical axis to display an appropriate eye-shape sprite.

  • Comparing the width and height of the user’s mouth to display an appropriate mouth-shape sprite.

👉 Please refer to 8. MYTY Kit Controllers for more information on the Controller.

👉 Please refer to [v0.1.0] 9. Apply Motion Tracking for more information on the MotionAdapter.

Usage Example

There are two ways to making an eye-blinking animation.

1. Using Bone Animation

You can make it using the Bone 1D Controller and EyeBone Adapter

  1. Rig the EyelidBone to the Eyelid Sprite.

  2. Create an empty GameObject (Naming example: EyeBoneController) within Controller Group, and add the Bone 1D Controller Component.

  3. Open Bone 1D Controller Window. Specify EyeBoneController, and add EyelidBone Object to the list in the Hierarchy window.

  4. Save the EyelidBone’s positions for when the eye is open (Max) and closed (Min).

  5. Add EyeBone Adapter Prefab to Hierarchy. Specify Face model and EyeBoneController to the Model and Controller, respectively.

  6. Press the Project play button to check that the animation works as intended.

2. Using Frame Animation

You can make it using the Sprite 1D Controller and EyeSprite Adapter.

  1. Create an empty GameObject (Naming example: EyeBoneController) within Controller Group, and add the Sprite 1D Controller Component.

  2. Open Sprite 1D Controller Window. Specify EyeBoneController, and add Eyelid Object to the list in the Hierarchy window.

  3. Press Auto Label to automatically bring up Sprite Library Label.

  4. Assign Min and Max values to the Sprite Label for when the eye is open, to determine the data range within which an open-eye Sprite will be sent.

  5. Assign Min and Max values to the Sprite Label for when the eye is closed, to determine the data range within which a closed-eye Sprite will be sent.

  6. Add EyeBone Adapter Prefab to Hierarchy. Specify Face model and EyeBoneController to the Model and Controller, respectively.

  7. Press the Project play button to check that the animation works as intended.

Previous[v0.1.0] 4. Import MYTY KitNext[v0.1.0] 9. Apply Motion Tracking

Last updated