AI-Assisted Costume Creation: Accelerating Production Pipeline

Artistic vision often demands knowledge beyond our immediate reach. In those moments, references and specific expertise become indispensable. If you were asked to draw a bike from memory, it would likely resemble two wheels connected by a confusing mess. However, If you tried drawing it again using a picture as a reference, or with the help of a bicycle designer, the result might actually look like a real bike (congrats!).

The same principles applied in video game, art, and animations. Some 3D artists may excel at creating 3D characters, but struggle with the nuanced art of designing and fitting accurate clothing for them. Others might be highly skilled at crafting intricate garments, but find the process of adapting them across multiple characters time-consuming or uninteresting.

This complexity inspired CDRIN to explore innovative solutions that could bring value to the world of 3D avatars with authentic clothing. But because CDRIN had virtually no experience in garments creation, we sought guidance from a professional pattern maker and fashion expert, Anakim Gosselin. Alongside Shaghayegh (Sherry) Taheri (AI Researcher), Yann Roubeau (R&D Programmer), and myself, Christophe Marois (Technical Artist), we formed a collaborative team to tackle this technical challenge.

The Technical Challenge: Bridging 2D and 3D Garment Design

While draping virtual clothing as if it were a Greco-Roman sculpture is a common practice in the 3D world, it is generally more effective to work with actual 2D patterns. Using patterns makes it easier to achieve a more realistic representation, especially with animating garments. This method also allows the garments to be tailored and adjusted just like in real life. The drawback of this pattern-based approach is that it requires an accurate pattern; without one the result can look like a glue-together cardboard catastrophe.

We sought a solution that would enable 3D artists to extract patterns from their sculpted attire. Our exploration led us to NeuralTailor, an open-source model designed to estimate 2D patterns from scanned 3D garments. The model was functioning within the established parameters 🖖🏻, but those parameters were not great. NeuralTailor was trained on data that did not match the Industry Standard, according to Le Grand Costumier, our fashion police.

We decided that improving NeuralTailor would be a good idea, as it would also work with regular 3D meshes, not just scanned ones.

Methodology: Crafting a Comprehensive Dataset

Dataset Requirements

To understand the intricacies of NeuralTailor, we must first examine the complexity of its required data. TailorNet is trained on a dataset that maps 2D garment patterns to their 3d simulated counterparts, ensuring realistic draping and fit. The patterns must adhere to several key constraints:

• Industry Standards: The 2D patterns should follow established garment design conventions.
• 3D Fit and Stability: When simulated on a 3D body, garments must behave realistically (e.g., skirts should not collapse to the ground)
• Design Variability: The Dataset must represent diverse garments types (shirts, pants, skirts, coats)
• Style Variability: Include various design elements (neckline variations, different sleeve types, and garment lengths).
• Technical Constraints: Patterns must comply with technical model constraints (mathematical and computational limitations of the project)
• Body Shape Adaptability: If NeuralTailor is expected to generalize beyond a single body type, garments must be simulated on multiple 3D body shapes.

Two Innovative Approaches

1. Manual Workflow

To jumpstart the project, I proposed a semi-manual workflow to create a high-quality pattern dataset. This industry-standard initial dataset would include various designs simulated on different 3D body shapes, without being custom-fitted.

This approach leverages Garment-Pattern-Generator, a tool developed by NeuralTailor’s creator, to automate the 3D simulation process.

The workflow followed this structure:

1. Anakim designs new patterns using standard-industry tools, adhering to specific guidelines for NeuralTailor Compatibility. Each pattern is created in two sizes (small and large).
2. I (Christophe) convert patterns to NeuralTailor-compatible format.
3. A custom script augments the dataset by interpolating between sizes.
4. Patterns are simulated in 3D using Garment-Pattern-Generator.

Then, to augment the dataset with style variations:

1. Anakim modifies existing patterns, introducing variations (e.g., converting a V-neck into a U-neck).
2. I convert the updated patterns into a NeuralTailor-compatible format.
3. The script now interpolates between sizes and styles, further expanding the dataset.
4. Garment-Pattern-Generator processes the updated patterns in 3D.

2. Automatic Pipeline

The automatic pipeline was my second approach to generating a larger dataset. It followed the same core requirements as the manual workflow, but introduced new challenges—along with some exciting possibilities.

How it works

1. Select a 3D body and adjust its morphology.
2. Take specific body measurements.
3. Generate a fitted pattern using freesewing.org based on those measurements.
4. Clean and convert patterns to a NeuralTailor-compatible format.
5. Transform the pattern in a 3D meshes.
6. Simulate the meshes on the 3D body to create a draped garment.
7. Repeat the process with randomized body shapes and/or pattern styles/options.

Why This Matters

This pipeline leverages freesewing.org, a free and open source project that generates patterns directly from measurements.Unlike the manual workflow, which primarily produced oversize or undersized garments on a limited number of bodies, this method creates tailored clothing that accurately fits a diverse range of 3D human morphologies.

The automatic pipeline eventually became a prototype called Costumy— one of the most ambitious (and frankly, coolest) things I’ve ever coded.

Costumy: An Open Source Prototype

Costumy, developed as an alternative to the manual pipeline, is a Python prototype that handles everything from measuring 3D bodies to draping garments from 2D patterns. Beyond dataset generation, Costumy has broader potential – from populating movie scenes with well-dressed crowds to inspiring avatar generation.

Few scenarios where Costumy could be used, as a whole or partially:

• Animated Film Production – An studio wants to populate the background scenes with diverse characters wearing well-fitted clothing.
• VFX Production – A team needs to dress the 3D replica of their real-life actor. By inputting the actor’s actual measurements, Costumy generates a 3D garment that fits both their real-world and digital counterpart.
• Indie Game Development – A small studio requires well-fitted clothing that integrates seamlessly with a custom character creator during production.
• Interactive Museum Installation – A tech-driven exhibit estimates visitors’ body measurements and uses Costumy to generate made-to-measure 3D avatars wearing a specific style of clothing.
• Independent 3D Artist – An independent artist designing a 3D humanoid character needs realistically tailored clothing to enhance their model.
• Pattern Making & Prototyping – A pattern maker wants to visualize a new design in 3D before physical production. While Costumy can assist, software like Marvelous Designer might be a better fit for this workflow.
• Research & Measurement Analysis – A research project requires precise 3D mesh measurements (e.g., SMPL body models) tailored to custom specifications.

Costumy is an exciting part of this research because it brings together powerful open source projects into a flexible and modular pipeline. It can be used as a whole, or broken down into individual components, making it highly adaptable.

And the best part? Costumy is open source and available on Github! Anyone can swap out different parts—whether it’s the 3d body model or the pattern generation method—to tailor it to their specific need.

Collaboration Made This Possible

I’d like to highlight how important collaboration is to me and how it made this project possible.

The entire pipeline was built on open source components—it simply wouldn’t exist otherwise. In fact, while developing the automatic pipeline creation, we made a key decision: we moved away from GarmentPatternGenerator because it relied on Maya and Qualoth, two paid and closed-source technologies. Instead, we built our own fully open-source alternative.

A huge part of this success was thanks to Anakim Gosselin, who played an essential role in helping our team navigate the rules of pattern making. She designed the patterns for the manual workflow and found clever workarounds to the technical limitations we faced.

One major challenge in the prototype phase was that we couldn’t create patterns requiring a single piece of fabric to be stitched to itself (like traditional shirt sleeves). Anakim adapted by designing sleeveless shirts, and even came up with innovative patterns where the sleeves were seamlessly integrated into the front and back panels—a solution that was technically compatible with our prototype and aligned with the industry standards.

NeuralTailor Results

If you remember, one of the main goals of this project was to improve NeuralTailor, whose original dataset didn’t quite align with the industry standards. Sherry trained the model using the manual pipeline dataset, then ran tests to see how well it performed. Check out the image below—it was generated by feeding the AI a 3D garment created from a 2D pattern.

Now, let’s test our initial idea: feeding NeuralTailor a rough 3D mesh made by an artist, converting it into a 2D pattern, then generating a 3D garment from it. Turns out—it works!

That said, we can already see a clear improvement in pattern estimation. Comparing the initial version of NeuralTailor with our retrained model, the difference is obvious—our dataset made it significantly better.

Nevertheless, our research wrapped before we could really push things further. We didn’t have time to train NeuralTailor with a Costumy-generated dataset, and most of our tests were limited to shirts (plus a few pants). So, claiming that NeuralTailor now works perfectly would be a stretch.
What we do know: It works—for shirts. It might do well on pants, but we don’t have enough data to say for sure. There’s still plenty of room for improvement, and a lot more research to be done!

Impact and Future Applications

The biggest takeaway from this research is that we successfully improve NeuralTailor by creating a dataset that aligns with industry standards. On top of that, we now have a Costumy, a working prototype that can generate an even larger dataset of both fitted and unfitted garments. The more we refine Costumy—which already has solid real-world applications—the more we expand the quality and diversity of data compatible with NeuralTailor.

NeuralTailor itself has the potential to evolve into a powerful pattern estimation tool. It could be the backbone of a feature that seamlessly fits garments onto 3D characters or, as we envisioned, a smart tool for 3D artists.