00 | Testing & Research phase

The start of the project was quite bumpy for me, as I initially struggled to decide which medium to use. I wavered between live-action and 3D animation, initially considering the idea of combining both components.

However, I quickly realized that my skills weren't sufficient to seamlessly integrate professional 3D animations into live-action footage without making them appear cheap. On a whim, I decided to create the entire film within a 3D environment.

My personal motivation was to enhance my proficiency with the tool Blender and animate my own characters.

01 | Story

02 | Concept

My personal goal was not to take the story or visual style of the film too seriously, but to maintain complexity in the technical execution of animations and movements. It was crucial for me to have control over the nature of the movements in order to choreograph the end product myself.

In my process, I turned to AI-powered tools to both experiment with whether such innovations could introduce new workflows or perspectives to the animation field and to compensate for skills I did not yet possess.

During this phase my research question was formed:

„Can I, as a single person, use AI to my advantage to complete work, that would usually be a studio pipeline/would take multiple people?“

Various AI-technologies were used during this process for brainstorming, structuring, translating, transferring and generating.

03 | Workflow

Nomad Sculpt - Leonardo - Mixamo - Blender

In the first workflow, I roughly and less detailedly modeled the shape and silhouette of the character using Nomad Sculpt. With the assistance of Leonardo AI, I created a character sheet tailored to the proportions and silhouette. This workflow allows for a wide range of possible styles and is particularly useful in the ideation phase to gain various impressions. I find this workflow helpful when a quick prototype of the character is needed, or when one needs to compensate for a lack of illustration skills. But nothing is compared to creating it by yourself.

For the movements, I initially turned to the Mixamo library, as the motions are of high quality, and transferring them to the character is a straightforward process with just a few clicks.

Nomad Sculpt - Rokoko Vision - Blender

In the second workflow, I wanted to try out the AI-powered tool Rokoko Vision. Rokoko Vision allows the generation of a MoCap-rig based on a video source. This rig can later be transferred to any character. In the traditional process, this step usually requires the use of a motion capture suit, motion trackers, and green screens. Depending on the complexity of the movement, actors or stuntmen might also be necessary.

Rokoko Vision bypasses these steps and only requires a well-shot video source. Critical factors for the quality of the rig include:

- Visibility of all limbs ( ideally with distinct clothing, with black consistently working well for me)

- Movement source should stand out from the background ( ideally, a homogeneous background )

- Complexity of the movement ( complex rotations around the body's own axis may lead to limbs not being recognized correctly )

- Stability of the cameraman (movement/zoom can cause unintended rig movements that are challenging to rectify. However,  this potential error source could be leveraged to one's advantage by adding dynamic jumps or „dashes“ through intentional camera movement, creating effects that would be physically impossible)

To evaluate the capabilities of Rokoko Vision, I progressively explored movements, starting with simple gestures and advancing to basic throwing actions, eventually incorporating complex breaking and tricking motions. I was pleasantly surprised by the high translation quality, noting that large movements like somersaults were often better translated than intricate hand gestures. Rokoko Vision excelled in full-body movements, where the finer details of hand movements were less critical. Even simple walking sequences were well translated.

Rokoko Vision has proven to be the greatest enhancement and innovation for movement generation and translation in my project. However, it posed one of the most significant and labor-intensive challenges during the cleaning and post-processing of movements. In very few cases does the rig translate well enough to require no post-processing, highlighting the necessity of having a foundation in rigging principles to exert influence.

Nevertheless, Rokoko Vision didn't necessarily bind me to actors or stuntmen; instead, it allowed me to choreograph and refine my film with my own movements. To diversify the spectrum of movements, I sought out individuals with athletic backgrounds in disciplines such as martial arts, parkour, and dance, whose movements were incorporated into the final product.

For this workflow, I also aimed to model and design the characters myself. While I am satisfied with the process, I am not entirely pleased with the clay-like appearance that resulted. Due to technical issues, I couldn't incorporate the right design into the animation, as the effort required to make extensive changes to most scenes would have been too great.

A significant challenge, which I sometimes managed better and sometimes worse, actually emerged during the transfer of movement to my character. Rokoko Vision operates on the principle that the parameters/movement of bones from the source rig are transferred to the bones of the desired rig. This means that ideally, the character's skeleton should match the number of bones in the Rokoko rig, and the bone labels must be nearly identical for the translation to occur.

There are (to my knowledge) two ways to create the character rig:

- Manual creation of the rig, bone by bone.

- Uploading the character to Mixamo and letting the platform create the rig

I tried both methods, using both the exact same number of bones and a reduced number of bones. This works as long as a bone doesn't carry essential information for the movement and can thus be „ignored.“

The reason for me not using a Mixamo-rig every time, was simply me not knowing better at that point. I hadn't figured out yet, how to take influence on the materials, when I generated through Mixamo. So every time, a character just had to move, with the material not being important, I used Mixamo. Every time when I had to change material in a character, I manually created the rig, reduced, without unnecessary bones.

Later I discovered how to also influence the Mixamo Rigg, but that is the progress I guess.

However, I couldn't explain why the transfer went smoothly in some instances with both methods and completely failed in others, even when all conditions for successful translation were met. My suspicion is that the weighting of individual bones was not properly assigned, leading to bones influencing parts of the mesh they shouldn't affect. I also assume that depending on the movement, sometimes an identical rig and other times a reduced rig are better suited.

For most applications, my reduced rig was sufficient, even for very complex movements.

04 | Animation in Blender

With movements as the focal point of the animations, I created the scenes in Blender. Most ideas for the scenes were independent of each other, which initially posed challenges in establishing meaningful connections. Each scene represented a new tool, which I had the opportunity to learn from week to week. The key scenes were the intro, the flight, and the fight, which is why these scenes were the first to be created.

Each scene was a process on its own. While the first scene, which I am personally most satisfied with, came together relatively quickly and intuitively, the close-combat scene proved to be very complicated. On one hand, the process of cleaning up and choreographing was the most challenging, and on the other hand, my MacBook reached its limits in this scene. With being one of the most time consuming scenes, it’s one of the scenes I’m less satisfied with, mainly because of the movement not transitioning fluently and the overall aesthetic, which looked kinda cheap.   All 3D assets were modeled in Nomad Sculpt, some specifically for the project, while others were recycled from older projects. The background images were the result of a previous Midjourney project that I could incorporate into the scenes.

05 | Post Production

The editing and post-production were done in Davinci Resolve and Adobe After Effects. I tried Davinci for the first time and am very satisfied with how intuitively the tool works. In Davinci, I handled cutting, sound, subtitles, and color grading, while After Effects was used to add minor effects.

For sound sources, I used free SFX packs, and for music, I relied on anime soundtracks.

I wrote the dialogues myself, translated them into Japanese using Chat-GPT, and then had the dialogues generated using ElevenLabs.

06 | Final Movie

Stillframes

07 | Making of

To shed more light on the application of Rokoko Vision, I wanted to create a small making-of video demonstrating how virtually anyone can produce an animated film from the comfort of their own home, without special equipment.

08 | Fictional Nintendo Switch Campaign

My classmate Lukas Vondran inspired me with his project „Catch me! if you can“ to animate a small fictional campaign conceptualizing my animated film as a game or mini-cinematic for the Nintendo Switch. His project is based on a „product shot“ for the „Fujifilm Instax 90 mini,“ which impressed me with its high level of quality and professionalism, motivating me to produce a small teaser.

09 | Conclusion

This project was the most elaborate endeavor I've ever undertaken. I've always been aware that animation is much more complex and demanding than it appears, but it wasn't until I gained firsthand experience that I realized how much technical knowledge and creative foundations are required to create an animated film. This project pushed me to my limits, and while I'm partially satisfied with the outcome, I believe I could have achieved higher quality if I had approached the process in a more structured and disciplined manner. I've come to understand that certain groundwork such as storyboarding, planning, and brainstorming are necessary to efficiently drive the process forward. I see this as a significant personal weakness on my part. I've recognized that while AI can be helpful in many ways, I only have real control over my project when I truly understand the individual components, tools and fundamentals before turning to AI. It should be supportive, not a replacement. Nevertheless, for my research question, my answer was definitely „Yes“ - with more knowledge and experience, I can as an individual collaborate with AI to do studio work. I believe that especially for freelancers or small studios, it can be beneficial to embrace new workflows.

In conclusion, I'm incredibly grateful for all that I've learned this semester. This project marks a personal milestone for me, and I couldn't ask for more.

Many thanks to Prof. Hartmann for the support, tools, and film-related knowledge I've gained throughout the semester. Thanks also to Prof. Jüsche for the Blender tools without which the film wouldn't have been possible. And a big thank you to the rig actors who helped me explore a new workflow and possibly advance the technology to a new level.

„I THINK I SPIDER“ - Major project „Visual Poetry“ Winter Semester 23/24.