6. Before Rigging
This document has been updated along with the MYTY Kit v1.0 release.
The MYTY ecosystem aims to enable Avatar Creators and metaverse app developers to create and use various live avatars easily. Accordingly, MYTY Kit solves the following needs of Avatar Creators and metaverse app developers.
- 1.Even if I am not familiar with the technical background of motion capture, I want to apply motion capture data to my avatars freely.
- 2.I want my avatar works well with compatibility across various metaverse apps.
- 1.I want to use various motion capture solutions to fit my multiple platform applications and apply them to MYTY Avatars.
- 2.I want to use live avatars without compromising the Avatar Creator's motion design intent.
To achieve the above purpose, MYTY Kit provides DefaultMotionTemplate and MotionAdapter. The overall motion data pipeline covered by MYTY Kit is shown below.
Motion Data Pipeline of MYTY Kit v1.0
MYTY Kit's motion capture data pipeline is divided into a part handled by metaverse app developers and a part handled by Avatar Creators.
- 1.When we move our body or face in front of the camera, that movement is captured and transmitted as data.
- 2.The captured data is categorized through TemplateBridge of MYTY Kit and flow to MotionTemplate.
- 3.Motion data delivered to MotionTemplate is used as input for MotionAdapter and flows to the Controller connected to the adapter.
- 4.The Controller expresses the movement by transforming the bones or sprites of the connected avatars.
- 5.An avatar that follows you in real-time is complete.
MYTY Kit's DefaultMotionTemplate supports motion tracking using Google's MediaPipe, a machine learning solution for live and streaming media. In particular, MYTY Kit utilizes MediaPipe's Face Mesh solution to track facial expressions in real-time. Since machine learning is employed to infer the position of facial and upper body landmarks as 3D coordinates, motion tracking requires just a single camera input, without the need for a dedicated depth sensor.
MeFaMo calculates the facial keypoints and blend shapes of a user. Instead of using the built in IPhone blend shape calculation (like LiveLinkFace App does), this uses the Googles Mediapipe to calculate the facial key points of a face. Those key points will then be used to calculate several facial blend shapes (like eyebrows, blinking, smiling etc.).
1. Drag and drop
Assets > MYTYKit > MotionTemplate >
DefaultMotionTemplateinto an empty Hierarchy window.
2. Drag and drop
Assets > MYTYKit > MotionTemplate > Motion Source Samples >
MediapipeMotionPackinto an empty Hierarchy window.
3. In the Inspector window of
MediapipeMotionPack > MediapipeMotionSourceobject, set
Motion Template Mapperto
DefaultMotionTemplate > Parametric >
SimpleFaceParamand press the
▶️(Play)button to play the project.
5. You can see the facial motion data value tracked in real-time in the inspector window.
The process of taking this 3D coordinate information and connecting it to the 2D NFT Avatar can be broken down into smaller steps of bone rigging, Controller, and MotionAdapter. Before you proceed further, please familiarize yourself with the following key terms.
In DefaultMotionTemplate of MYTY Kit, motion data is classified into 47 templates.
- The MYTY Kit tools
MotionAdapterconnect the user’s Motion tracking data to their MYTY Avatar.
- MotionAdapter is a tool that processes raw Motion input and delivers it to the related Controller.
- Controller is a tool that uses the processed Motion input to control an avatar’s Bones and Sprites.
- Let’s find out more about the four types of Controllers included in the MYTY Kit: what they do, and which MotionAdapters they can be used with.
There are two ways to make an eye-blinking animation.
An example of making the eyes blink using Bone animation.
You can make it using the
Bone 1D Controllerand
- 1.Rig the EyelidBone to the Eyelid Sprite.
- 2.Create an empty GameObject (Naming example: EyeBoneController) within Controller Group, and add the Bone 1D Controller Component.
- 3.Open Bone 1D Controller Window. Specify EyeBoneController, and add EyelidBone Object to the list in the Hierarchy window.
- 4.Save the EyelidBone’s positions for when the eye is open (Max) and closed (Min).
- 5.Add (Bone)EyeBlink Adapter Prefab to Hierarchy. Specify the SimpleFaceParam template and EyeBoneController to the Template and Controller, respectively.
- 6.Press the Project play button to check that the animation works as intended.
An example of making the eyes blink using Frame animation.
You can make it using the
Sprite 1D Range Controllerand
- 1.Create an empty GameObject (Naming example: EyeBoneController) within Controller Group, and add the Sprite 1D Range Controller Component.
- 2.Open Sprite 1D Range Controller Window. Specify EyeBoneController, and add Eyelid Object to the list in the Hierarchy window.
- 3.Press Auto Label to automatically bring up Sprite Library Label.
- 4.Assign Min and Max values to the Sprite Label for when the eye is open, to determine the data range within which an open-eye Sprite will be sent.
- 5.Assign Min and Max values to the Sprite Label for when the eye is closed, to determine the data range within which a closed-eye Sprite will be sent.
- 6.Add (Sprite)EyeBlink Adapter Prefab to Hierarchy. Specify the SimpleFaceParam template and EyeBoneController to the Template and Controller, respectively.
- 7.Press the Project play button to check that the animation works as intended.