6. Before Rigging

This document has been updated along with MYTY Kit v1.0 release on October 2022.

TL;DR

MYTY Kit’s Motion Tracking Solution

Understanding the overall Motion Tracking process is required before the Bone Rigging, Controller, and MotionAdapter steps. It is recommended that you first read thePrerequisites.

MYTY ecosystem aims to enable Avatar Creators and metaverse app developers to create and use various live avatars easily. Accordingly, MYTY Kit solves the following needs of Avatar Creators and metaverse app developers.

Needs of Avatar Creators :

  1. Even if I am not familiar with the technical background of motion capture, I want to apply motion capture data to my avatars freely.

  2. I want my avatar works well with compatibility across various metaverse apps.

Needs of Metaverse app developers :

  1. I want to use various motion capture solutions to fit my multiple platform applications and apply them to MYTY Avatars.

  2. I want to use live avatars without compromising the Avatar Creator's motion design intent.

To achieve the above purpose, MYTY Kit provides DefaultMotionTemplate and MotionAdapter. The overall motion data pipeline covered by MYTY Kit is shown below.

Motion Capture Data Pipeline

MYTY Kit's motion capture data pipeline is divided into a part handled by metaverse app developers and a part handled by Avatar Creators.

  1. When we move our body or face in front of the camera, that movement is captured and transmitted as data.

  2. The captured data is categorized through TemplateBridge of MYTY Kit and flow to MotionTemplate.

  3. Motion data delivered to MotionTemplate is used as input for MotionAdapter and flows to the Controller connected to the adapter.

  4. The Controller expresses the movement by transforming the bones or sprites of the connected avatars.

  5. An avatar that follows you in real-time is complete.

DefaultMotionTemplate include in MYTY Kit

MYTY Kit provides a MotionTemplate that can be used as a sample by utilizing Google's MediaPipe and Mefamo. This is called DefaultMotionTemplate. By using this, Avatar Creators can obtain stable motion data automatically without applying a separate motion capture solution.

What is Google's MediaPipe ?

MYTY Kit's DefaultMotionTemplate supports motion tracking using Google's MediaPipe, a machine learning solution for live and streaming media. In particular, MYTY Kit utilizes MediaPipe's Face Mesh solution to track facial expressions in real-time. Since machine learning is employed to infer the position of facial and upper body landmarks as 3D coordinates, motion tracking requires just a single camera input, without the need for a dedicated depth sensor.

What is Mefamo?

MeFaMo calculates the facial keypoints and blend shapes of a user. Instead of using the built in IPhone blend shape calculation (like LiveLinkFace App does), this uses the Googles Mediapipe to calculate the facial key points of a face. Those key points will then be used to calculate several facial blend shapes (like eyebrows, blinking, smiling etc.).

Check the motion data in real-time

1. Drag and drop Assets > MYTYKit > MotionTemplate > DefaultMotionTemplate into an empty Hierarchy window.

2. Drag and drop Assets > MYTYKit > MotionTemplate > Motion Source Samples > MediapipeMotionPack into an empty Hierarchy window.

3. In the Inspector window of MediapipeMotionPack > MediapipeMotionSource object, set Motion Template Mapper to DefaultMotionTemplate.

4. Select DefaultMotionTemplate > Parametric >SimpleFaceParam and press the ▶️(Play) button to play the project.

5. You can see the facial motion data value tracked in real-time in the inspector window.

Get familiar with making live avatars

Key terms

The process of taking this 3D coordinate information and connecting it to the 2D NFT Avatar can be broken down into smaller steps of bone rigging, Controller, and MotionAdapter. Before you proceed further, please familiarize yourself with the following key terms.

What kind of motion data can be tracked?

In DefaultMotionTemplate of MYTY Kit, motion data is classified into 47 templates.

MYTY Kit v1.0 provides a stable motion tracking environment only for Face parameters and Head. Other motion data for full-body tracking will be provided in a later version.

The Controller-MotionAdapter Relationship

Which Controllers and MotionAdapters will you need to make your desired animation?

  • MYTY Kit tools Controller and MotionAdapter connect the user’s Motion tracking data to their MYTY Avatar.

    • MotionAdapter is a tool that processes raw Motion input and delivers it to the related Controller.

    • Controller is a tool that uses the processed Motion input to control an avatar’s Bones and Sprites.

  • Let’s find out more about the four types of Controllers included in MYTY Kit: what they do, and which MotionAdapters they can be used with.

👉 Please refer to 8. MYTY Kit Controllers for more information on the Controller.

👉 Please refer to 9. Apply Motion Tracking for more information on the MotionAdapter.

Usage Example

There are two ways to make an eye-blinking animation.

You can make it using the Bone 1D Controller and (Bone)EyeBlink Adapter

  1. Rig the EyelidBone to the Eyelid Sprite.

  2. Create an empty GameObject (Naming example: EyeBoneController) within Controller Group, and add the Bone 1D Controller Component.

  3. Open Bone 1D Controller Window. Specify EyeBoneController, and add EyelidBone Object to the list in the Hierarchy window.

  4. Save the EyelidBone’s positions for when the eye is open (Max) and closed (Min).

  5. Add (Bone)EyeBlink Adapter Prefab to Hierarchy. Specify the SimpleFaceParam template and EyeBoneController to the Template and Controller, respectively.

  6. Press the Project play button to check that the animation works as intended.

You can make it using the Sprite 1D Range Controller and (Sprite)EyeBlink Adapter.

  1. Create an empty GameObject (Naming example: EyeBoneController) within Controller Group, and add the Sprite 1D Range Controller Component.

  2. Open Sprite 1D Range Controller Window. Specify EyeBoneController, and add Eyelid Object to the list in the Hierarchy window.

  3. Press Auto Label to automatically bring up Sprite Library Label.

  4. Assign Min and Max values to the Sprite Label for when the eye is open, to determine the data range within which an open-eye Sprite will be sent.

  5. Assign Min and Max values to the Sprite Label for when the eye is closed, to determine the data range within which a closed-eye Sprite will be sent.

  6. Add (Sprite)EyeBlink Adapter Prefab to Hierarchy. Specify the SimpleFaceParam template and EyeBoneController to the Template and Controller, respectively.

  7. Press the Project play button to check that the animation works as intended.

Last updated