---
description: >-
  This document has been updated along with MYTY Kit v1.0 release on October
  2022.
---

# 6. Before Rigging

## TL;DR

1. [MYTY Kit's Motion data pipeline](#motion-capture-data-pipeline)
2. [Check the motion-tracking data](#check-the-motion-data-in-real-time)
3. [What kind of motion data can be tracked?](#what-kinds-of-motion-data-can-be-tracked)
4. [The Controller-MotionAdapter relationship](#the-controller-motionadapter-relationship)
5. [Usage Example](#usage-example)

## MYTY Kit’s Motion Tracking Solution

{% hint style="info" %}
Understanding the overall Motion Tracking process is required before the Bone Rigging, Controller, and MotionAdapter steps. It is recommended that you first read the[prerequisites](../before-you-begin/prerequisites "mention").
{% endhint %}

MYTY ecosystem aims to enable Avatar Creators and metaverse app developers to create and use various live avatars easily. Accordingly, MYTY Kit solves the following needs of Avatar Creators and metaverse app developers.

#### Needs of Avatar Creators :

1. Even if I am not familiar with the technical background of motion capture, I want to apply motion capture data to my avatars freely.
2. I want my avatar works well with compatibility across various metaverse apps.

#### Needs of Metaverse app developers :&#x20;

1. I want to use various motion capture solutions to fit my multiple platform applications and apply them to MYTY Avatars.
2. I want to use live avatars without compromising the Avatar Creator's motion design intent.

To achieve the above purpose, MYTY Kit provides <mark style="color:purple;">**DefaultMotionTemplate**</mark> and <mark style="color:purple;">**MotionAdapter**</mark>. The overall motion data pipeline covered by MYTY Kit is shown below.

### Motion Capture Data Pipeline

<figure><img src="https://667382355-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZT6KDsWarcdCAiEJZ3TV%2Fuploads%2FRpG3RoHI3ptlMuFyAAxC%2FImage%20-%2025-%20mocap%20pipeline.png?alt=media&#x26;token=6d7474ec-8888-4a46-9133-45df4f9a4154" alt=""><figcaption><p>Motion Data Pipeline of MYTY Kit v1.0</p></figcaption></figure>

MYTY Kit's motion capture data pipeline is divided into a part handled by metaverse app developers and a part handled by Avatar Creators.

1. When we move our body or face in front of the camera, that movement is captured and transmitted as data.
2. The captured data is categorized through TemplateBridge of MYTY Kit and flow to MotionTemplate.
3. Motion data delivered to MotionTemplate is used as input for MotionAdapter and flows to the Controller connected to the adapter.
4. The Controller expresses the movement by transforming the bones or sprites of the connected avatars.
5. An avatar that follows you in real-time is complete.

### DefaultMotionTemplate include in MYTY Kit

MYTY Kit provides a MotionTemplate that can be used as a sample by utilizing [Google's MediaPipe](https://google.github.io/mediapipe/) and [Mefamo](https://github.com/JimWest/MeFaMo). This is called **DefaultMotionTemplate**. By using this, Avatar Creators can obtain stable motion data automatically without applying a separate motion capture solution.

<details>

<summary>What is Google's MediaPipe ?</summary>

MYTY Kit's DefaultMotionTemplate supports motion tracking using [Google's MediaPipe](https://google.github.io/mediapipe/), a machine learning solution for live and streaming media. In particular, MYTY Kit utilizes [MediaPipe's Face Mesh solution](https://google.github.io/mediapipe/solutions/face_mesh.html) to track facial expressions in real-time. Since machine learning is employed to infer the position of facial and upper body landmarks as 3D coordinates, motion tracking requires just a single camera input, without the need for a dedicated depth sensor.

![](https://667382355-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZT6KDsWarcdCAiEJZ3TV%2Fuploads%2FRvk2D4LAzN3FmIX6fqtV%2Fface_mesh_ar_effects.gif?alt=media\&token=08b6e57a-58bb-433e-8670-ae8c8b4c62cf)

</details>

<details>

<summary>What is Mefamo?</summary>

MeFaMo calculates the facial keypoints and blend shapes of a user. Instead of using the built in IPhone blend shape calculation (like [LiveLinkFace App](https://apps.apple.com/us/app/live-link-face/id1495370836) does), this uses the Googles [Mediapipe](https://github.com/google/mediapipe) to calculate the facial key points of a face. Those key points will then be used to calculate several facial blend shapes (like eyebrows, blinking, smiling etc.).

![](https://667382355-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZT6KDsWarcdCAiEJZ3TV%2Fuploads%2FUdOIrjaVKGDKkGslc2S3%2Fshowoff_2.gif?alt=media\&token=4a42a9f0-5fba-4175-bd8a-32980d98cdb2)

</details>

### Check the motion data in real-time

1\. Drag and drop <mark style="color:purple;">`Assets > MYTYKit > MotionTemplate >`</mark><mark style="color:purple;">` `</mark><mark style="color:purple;">**`DefaultMotionTemplate`**</mark> into an empty Hierarchy window.

2\. Drag and drop <mark style="color:purple;">`Assets > MYTYKit > MotionTemplate > Motion Source Samples >`</mark><mark style="color:purple;">` `</mark><mark style="color:purple;">**`MediapipeMotionPack`**</mark> into an empty Hierarchy window.

<figure><img src="https://667382355-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZT6KDsWarcdCAiEJZ3TV%2Fuploads%2FYjRhXLyqkH9KK28bDP17%2Fcheck_data_1.gif?alt=media&#x26;token=ebbd86c2-47c0-4095-afb1-b305e9ba0868" alt=""><figcaption></figcaption></figure>

3\. In the Inspector window of  <mark style="color:purple;">`MediapipeMotionPack > MediapipeMotionSource`</mark> object, set <mark style="color:purple;">`Motion Template Mapper`</mark> to <mark style="color:purple;">**`DefaultMotionTemplate`**</mark>.

<figure><img src="https://667382355-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZT6KDsWarcdCAiEJZ3TV%2Fuploads%2FnbHsothgqZDneclReI4e%2Fcheck_data_2.gif?alt=media&#x26;token=270d3359-c85b-4409-8a96-0338cfbcf320" alt=""><figcaption></figcaption></figure>

4\. Select <mark style="color:purple;">`DefaultMotionTemplate > Parametric >`</mark><mark style="color:purple;">**`SimpleFaceParam`**</mark> and press the <mark style="color:purple;">`▶️(Play)`</mark> button to play the project.

5\. You can see the facial motion data value tracked in real-time in the inspector window.

<figure><img src="https://667382355-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZT6KDsWarcdCAiEJZ3TV%2Fuploads%2Fd2xUvU3N4b8seTOBpleK%2Fcheck_data_3.gif?alt=media&#x26;token=980e836e-f8d0-4845-a5c1-96efd3e7b315" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}
Is motion tracking not working? The input camera may be configured incorrectly. Refer to [#the-mediapipe-object-cant-track-my-movements.](../../misc/troubleshooting-and-faq#the-mediapipe-object-cant-track-my-movements. "mention")for a solution.
{% endhint %}

## Get familiar with making live avatars

### Key terms

The process of taking this 3D coordinate information and connecting it to the 2D NFT Avatar can be broken down into smaller steps of **bone rigging, Controller, and MotionAdapter**. Before you proceed further, please familiarize yourself with the following key terms.

|                                        |                                                                                                                                                                                                                                                                                |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **MYTY Avatar**                        | An avatar created using MYTY Kit. It is both a [Live Avatar](https://myty.gitbook.io/myty-kit-guide/before-you-begin/myty-kit-glossary) and an [NFT Avatar,](https://myty.gitbook.io/myty-kit-guide/before-you-begin/myty-kit-glossary) and is compatible with MYTY Ecosystem. |
| **Motion capture**                     | Motion capture(Mo-Cap) is the process of capturing the movement of people, animals or objects, transferring it to a 2D or 3D model and animating it with the set of recorded movements.                                                                                        |
| **Motion tracking data (Motion data)** | Movement data tracked by Mo-Cap solution.                                                                                                                                                                                                                                      |
| **Motion Adapter**                     | A MYTY Kit tool that processes motion tracking data and connects it to another MYTY Kit tool, the Controller.                                                                                                                                                                  |
| **Controller**                         | A MYTY Kit Tool that controls the Bones and Sprites (Image) of MYTY Avatars.                                                                                                                                                                                                   |
| **Bone rigging**                       | The process of connecting Bones to the 2D Avatar Image to enable joint movements.                                                                                                                                                                                              |
| **Sprite Mesh**                        | The sprite's mesh type is **all about how Unity will render your sprite in your game**. The traditional way of rendering them is in full rect mode.                                                                                                                            |

### What kind of motion data can be tracked?

In DefaultMotionTemplate of MYTY Kit, motion data is classified into **47 templates**.

{% hint style="danger" %}
MYTY Kit v1.0 provides a stable motion tracking environment only for **Face parameters** and **Head**. Other motion data for full-body tracking will be provided in a later version.
{% endhint %}

<figure><img src="https://667382355-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZT6KDsWarcdCAiEJZ3TV%2Fuploads%2FNvzUHUdIRo1lMK3txyha%2FImage%20-%2020-%20DMT.png?alt=media&#x26;token=aea42317-e84c-4443-8a59-cf8a58230aac" alt=""><figcaption></figcaption></figure>



### The Controller-MotionAdapter Relationship

{% hint style="success" %}
Which Controllers and MotionAdapters will you need to make your desired animation?
{% endhint %}

<figure><img src="https://667382355-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZT6KDsWarcdCAiEJZ3TV%2Fuploads%2F7vIJY6U9WfD781ywulvY%2FImage%20-%2027-%20puppet.png?alt=media&#x26;token=3056ae30-36c5-4d17-8fed-d198e1d6a15f" alt=""><figcaption></figcaption></figure>

* MYTY Kit tools <mark style="color:purple;">`Controller`</mark> and <mark style="color:purple;">`MotionAdapter`</mark> connect the user’s Motion tracking data to their MYTY Avatar.
  * MotionAdapter is a tool that processes raw Motion input and delivers it to the related Controller.
  * Controller is a tool that uses the processed Motion input to control an avatar’s Bones and Sprites.
* Let’s find out more about the four types of Controllers included in MYTY Kit: what they do, and which MotionAdapters they can be used with.

<table><thead><tr><th width="170">   </th><th>Bone 1D Controller</th><th>Bone 2D Controller</th></tr></thead><tbody><tr><td><strong>Input</strong></td><td>The float value of 1 MotionAdapter output.</td><td>The float values of 2 MotionAdapter outputs.</td></tr><tr><td><strong>Output</strong></td><td>Transformation of connected bone(s).</td><td>Transformation of connected bone(s).</td></tr><tr><td><strong>Related MotionAdapter</strong></td><td>👉 BoneTiltOrPosition<br>👉 (Bone) EyeBlink &#x26; EeyBlinkAll<br>👉 (Bone) EyeBrow &#x26; EyeBrowAll</td><td><p>👉 Bone2D</p><p>👉 (Bone) Pupil &#x26; PupilAll</p></td></tr><tr><td><strong>Usage Example</strong></td><td><ul><li>Using the user’s eyebrow movements in the vertical axis to set movements of the avatar’s eyebrow bone.</li></ul></td><td><p></p><ul><li>Using the user’s face movements in the vertical and horizontal axes (face direction) to set movements of the avatar’s FaceBone.</li></ul><ul><li>Using the user’s chest movements in the vertical and horizontal axes to set movements of the avatar’s BodyBone.</li></ul></td></tr></tbody></table>

<table><thead><tr><th width="171">   </th><th>Sprite 1D Range Controller</th><th>Sprite 2D Nearest Controller</th><th>Rigged Sprite 2D Nearest Controller</th></tr></thead><tbody><tr><td><strong>Input</strong></td><td>The float value of 1 MotionAdapter output.</td><td>The float value of 2 MotionAdapter output.</td><td>The float value of 2 MotionAdapter output.</td></tr><tr><td><strong>Output</strong></td><td>Selection of appropriate sprite image(s) to display.</td><td>Selection of appropriate sprite image(s) to display.</td><td>Selection of appropriate bone-rigged sprite image(s) to display.</td></tr><tr><td><strong>Related MotionAdapter</strong></td><td>👉 (Sprite) EyeBlink &#x26; EyeBlinkAll<br>👉 (Sprite) EyeBrow &#x26; EyeBrowAll</td><td>(Sprite)MouthSprite</td><td>(Sprite)MouthSprite</td></tr><tr><td><strong>Usage Example</strong></td><td><p></p><ul><li>Using the user’s eye size in the vertical axis to display an appropriate eye-shape sprite.</li></ul><p><br></p></td><td><ul><li>Using the user's mouth size in the vertical and horizontal axes to display an appropriate mouth-shape sprite. </li></ul></td><td><ul><li>Using the user's mouth size in the vertical and horizontal axes to display an appropriate mouth mesh sprite rigged with bones.</li></ul></td></tr></tbody></table>

{% hint style="info" %}
👉 Please refer to [8.-myty-kit-controllers](8.-myty-kit-controllers "mention") for more information on the Controller.

👉 Please refer to [9.-apply-motion-tracking](9.-apply-motion-tracking "mention") for more information on the MotionAdapter.
{% endhint %}

## Usage Example

There are two ways to make an eye-blinking animation.

### 1. Using [Bone Animation](../../getting-started/1.-prepare-illustration-psb/basics-of-avatar-animation#bone-animation)

![An example of making the eyes blink using Bone animation.](https://667382355-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZT6KDsWarcdCAiEJZ3TV%2Fuploads%2FqwfLJlDiujlZBhsUSXVg%2Feyebone.gif?alt=media\&token=e487f7b2-ccc3-4659-bc13-965cdd8d8f5f)

You can make it using the <mark style="color:purple;">**`Bone 1D Controller`**</mark> and <mark style="color:purple;">**`(Bone)EyeBlink Adapter`**</mark>

1. Rig the EyelidBone to the Eyelid Sprite.&#x20;
2. Create an empty GameObject (Naming example: EyeBoneController) within Controller Group, and add the **Bone 1D Controller** Component.&#x20;
3. Open Bone 1D Controller Window. Specify **EyeBoneController**, and add EyelidBone Object to the list in the Hierarchy window.&#x20;
4. Save the EyelidBone’s positions for when the eye is open (Max) and closed (Min).&#x20;
5. Add **(Bone)EyeBlink Adapter Prefab** to Hierarchy. Specify the **SimpleFaceParam template** and EyeBoneController to the Template and Controller, respectively.&#x20;
6. Press the Project **play button** to check that the animation works as intended.

### 2. Using [Frame Animation](../../getting-started/1.-prepare-illustration-psb/basics-of-avatar-animation#frame-sprite-animation)

![An example of making the eyes blink using Frame animation.](https://667382355-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZT6KDsWarcdCAiEJZ3TV%2Fuploads%2F68jhNbhjiN7LCMhmIHc0%2Feyesprite.gif?alt=media\&token=b316b7b2-c76a-42ff-b18b-c7c5c92e2b7c)

You can make it using the <mark style="color:purple;">**`Sprite 1D Range Controller`**</mark> and <mark style="color:purple;">**`(Sprite)EyeBlink Adapter`**</mark>**.**

1. Create an empty GameObject (Naming example: EyeBoneController) within Controller Group, and add the **Sprite 1D Range Controller** Component.&#x20;
2. Open **Sprite 1D Range Controller** Window. Specify EyeBoneController, and add Eyelid Object to the list in the Hierarchy window.&#x20;
3. Press **Auto Label** to automatically bring up Sprite Library Label.&#x20;
4. Assign Min and Max values to the Sprite Label for when the eye is open, to determine the data range within which an open-eye Sprite will be sent.&#x20;
5. Assign Min and Max values to the Sprite Label for when the eye is closed, to determine the data range within which a closed-eye Sprite will be sent.&#x20;
6. Add **(Sprite)EyeBlink Adapter Prefab** to Hierarchy. Specify the **SimpleFaceParam template** and EyeBoneController to the Template and Controller, respectively.
7. Press the Project **play button** to check that the animation works as intended.



{% hint style="success" %}
**Have a question or an idea?** \
If you have a question to ask or an idea to share, participate in the [MYTY Kit Community](https://discord.com/invite/myty). We’d love to hear from you.:smile:
{% endhint %}
