Lip Sync and Performance

Lip Sync Task Brief

For this challenge, I had to make a short (one line), 3-10 second, lip syncing performance animation. Beginning with the mouth animation, then the face, head and body animation will follow. By doing this I will be taking the audio clip (from a movie) out of the context that it was in and creating my own character and staging design.

The concept I designed for the performance is where the character is talking to his morning cup of coffee (See ref BLEH). The character pleads or even challenges the cup to “make my day”. As though a cup of coffee can typically improve the mood of someone, but in this scenario, the character is asking the for the cup to make his day. The emotions I attempted to portray were: depressed; intimidating, joyful, irritated, disgust, indifferent or casual. All of these, were portrayed by the tone or volume of voice, facial expression, body language, gestures and the volume of movement. I also made many tests to create a variety of possibilities. While this performance is short and simple, where is much that I would like to input into these small moments. For any animation, even if it seems simple, there is so many details that go into a small expression. Directing and translating my concept idea into the animation is my aim for this task.

• Step 1: Find an audio clip.
• Step 2: Act out (yourself or a friend) a performance to the audio (lip syncing).
• Step 3: Convert the audio (if needed) and match it to your performance footage.
• Step 4: In Maya, create a lip sync (jaw bounce and visemes only).

Audio Clip & Rig: Source/find a short audio clip (3 -10 seconds approx of one person talking) which will form the basis of a lip sync and facial animation sequence in the first instance and then later, a full body acted performance sequence (combining both face/lip syncing animation and body performance animation). It is therefore important that you…

• 1) Consider carefully which audio clip and rig you choose – Listen carefully to the audio clip and try to imagine it as an animated character.
• 2) Ignore the source of the clip – Which actor or film for example. The goal is to create your own performance.
• 3) Select a rig which suits the audio and is capable of lip syncing.
• 4) Consider the clip as potential ‘full performance’ beyond just lip syncing.
• 5) Record reference footage based upon the audio clip – Either yourself or a friend (no online downloads).
• 6) Prepare the reference footage and audio clip using Premiere and After Effects to use in Maya.

Research

“Unlike live action films, where the dialog is simultaneously recorded with the action, in animation it must be recorded beforehand so that the movement can be fitted precisely. It is an essential pre-production operation, which cannot be left until after the completion of animation. No two dialog performances are the same, even single words (…) can vary substantially when analysed in terms of separate frames. Firstly, listen carefully to the soundtrack and in particular to the feeling behind the way in which the words are spoken. Then listen to the phrasing and rhythm of the speech and find the positions of the main emphasis and key words. The facial expressions, head movement and body gestures can underline and add to the meaning and interest of the dialog. It is not essential to animate all vowels and consonants in terms of single frames, about eight positions of mouth and tongue are adequate.”

This animation, I think , is brilliant. It shows great emotion and character in the facial features which make the performance look natural, expressive and aesthetically pleasing. Yet, the performance is not an exact replica. That is a good thing, considering that the differences align with the character rig that the animation is made with – it fits the character of the curmudgeon old man.

Chosen Audio

“Go ahead… make my day”

Reference Footage

Sentence Mouth Shape:

Using the reference video and researched secondary references I made myself a guide to help make each mouth shapes and facial expressions,

Performance Concept

The idea I had for the performance is where the character is talking to his morning cup of coffee. The character pleads or even challenges the cup to “make my day”. As though a cup of coffee can typically improve the mood of someone, but in this scenario, the character is asking the for the cup to make his day.

Maya Scene – Lip Syncing

Step 1: Jaw Bounce

There was slight jaw chattering on the word ‘Go. Before the next character ‘A’, I thought that the jaw would move up slightly on the ‘O’. This did not look good, and in the second playblast, I decided to try and take out his extra frame. Then, I would will only show the ‘O-w’ sound in the lip movement and leave the jaw for now.

Step 2: Viseme

Viseme means how short, wide and curled the mouth is. This should have the same rhythm as the jaw bounce.

I noticed that to improve the animation at this stage, then I should align the jaw bounce with the key frame sounds of the viseme.

If you look at the play blast for this section, you will see different versions of tests. The ‘make my day’ seemed too fast, so in the last play blast for viseme you can see where I removed the key frame on the ‘Y’. Taking out less essential key frames smooths out the animation slightly. Although, the essential frames must stay to provide the structure. I chose the ‘Y’ frame since it seemed less essential, even though I intended to shape the mouth at that moment.

Step 3: Mouth Position and Up/Down

At this step, I looked closely at the reference footage. I created these still frames to help with the mouth movement the facial expression in the later steps.

Step 4: Teeth

For the best teeth position, I closely used my reference images to guide their position. As well as this, I found that watching the jaw movement is imperative to make the teeth feel correct.

Step 5: Lower Face Deformer Controls

Step 6: Upper Hemisphere (Eyes and Brows)

I added one blink at the start, following the video. The outcome was very natural, and at the right timing, since the head should move at that point. Looking at the lecture video, it seems that a blink indicates a moment of thought in the character. Therefore, I attempted to add another blink in the pause between the words. This one felt too quick and forced after the other. I considered adding a blink elsewhere, like at the end instead.

The eye direction is only roughly done. I will have to edit them again to look directly at the subject after I animation the head.

Step 7: Body

I struggled with problems with thumb and arm positions.

After many attempts to fiddling with these problems using FK, I changed the arms to IK.

I will have to figure out how to position the hand onto the hip throughout the animation. Perhaps if I use a locators. If I snap the locator onto the hip and make the locators animation. Then move the into the distance that the hand controller must be away from the hip? Would that work, I think I may need to use parent to snap the locator onto the hip, then to move it, it may mess up the animation. I will have to do some tests. I do not want to use my tutor’s method of making the arm animation in FK, since I had so much trouble with that already. Perhaps if I simply copy the hip animation using the locator. I can just copy the curves in the graph editor. I’ll ensure that the z-axis of the locator is in the exact position that I want the hand controller, then just move the x-axis curve of the locator after I have made the animation. This will place the locator and the IK hand control the distance I need it away from the hip.

Cup Parenting

I made a cup model for the performance, as well as its own control. I tested out if it would be better to have the parent as the hand or the cup. I think the hand would be best, since if it was the cup, then if I move the cup too far away from the hand, the cup will escape the grip of the hand. Either way, it makes sense for the

This is when I finished placing the fingers around the cup. Then, I parented the cup to the hand.

This is the starting position, although looking at it now I can see a problem. The weight of the cup in the hand is not right. The centre of gravity of the hand hold is wrong, it needs to be shifted to the centre of the palm more, to the right.

Once the left arm movement was completed, I considered that the cup should be held up for longer. So, that in the performance aspect, there is enough time to show that the character is talking to the cup.

I decided that the cup should be held up to the eyes, to show a subject of attention, for longer. I used the animation timeline select ‘squash’ tool (not real name) to do this. The result is as shown in the screenshot below.

This was the first attempt at animating the left hand. I had to add so many key frames to adjust the rotation, as every few frames, it was off. Instead, I restarted the cup animation and manually adjusted the centre pivot original rotation as this is what caused all of the problems. The next left hand animation was much better and smooth.

Play Blasts

Final Animation

Note: I had to reupload the vimeo file, after the submission, as I realised that I didn’t add audio to that clip (See the end of the post).

I ensured that the animation fit more to my concept by extending the time that the cup was in front of the character’s face. The eyes were drawn to the cup for longer and the cup/left arm movement was made wider to draw attention to the focal subject. I ensured that I animated the body movement around the emphasised words or pauses within the audio. The review the final animation, I believe that there are areas that need more improvement, though. Such as where the eyes look all of the time, they should not looked at the viewer for so long. It broke the fourth wall and made the impression that the character was talking to the audience instead of the cup. Also, I believe that I could have chosen a longer audio clip or designed a few more seconds of performance, outside of the audio clip. For example, an exasperated sigh while looking down, and then picking up the cup. While animating, I experimented with difference ways of introducing the cup using various ways of staging. I realised I had to juggle having the audience’s and character’s focus drawn to the cup and still having it low enough to see the lip syncing. In another scenario, I could have made a sideview of the character instead. This would have been better staged this way, the eye line would have clearer and the focus of both aforementioned parties would have been drawn better.

Peer Review:

The arms look strange. Perhaps this was because of the rig, or my insufficient positioning of the limbs. They look slightly twisted or the under the arm pit is squashed on the left hand.

References:

Pluralsight. 2013. Proven Tips for Animating Believable Lip Sync. [online] Available at: <https://www.pluralsight.com/blog/film-games/proven-tips-animating-believable-lip-sync> [Accessed 16 November 2021].

Sito, T., 2013. Timing for Animation. Taylor & Francis, pp.125-130.