This is the lab notebook: the chronological record of every experiment, kept as it was written. The curated account - method, evaluation, references - is the CharForge paper; open work is in the experiment tracker.
24 September - 5 October 2026 · M5 MacBook, 24 GB shared memory

What CharForge makes now, and what fixing it took

A prompt or a picture becomes a rigged, animated game character on this laptop; a video becomes a move on it, and now so does a sentence. The first round went through every character in the poses that break things and traced each fault to its cause before touching it: Pip's vest no longer lifts into wings when he raises his arms, no texture carries another garment's paint, a hem stays with the hips instead of the thighs, a reference drawn in the wrong pose is drawn again, and a short prompt is written out before anything is drawn. The second round measured instead of looking: every frame of every clip is now audited for aberrations - faces crushed, stretched or sheared, one body part inside another, a shoe through the floor - and what the audit found was fixed. Limbs keep out of each other within joint limits measured on each character's own mesh; the shoe that is planted is the shoe the viewer sees; the second 3D pass, which made models less like their pictures, is off. A new character, Cadet, was made with the pipeline as it stands and given moves the motion library lacks from a sentence each. The third round measured the models against professionally made ones and made them cleaner, and then took the face apart, part by part, on a benchmark set of twelve. CharForge Studio does all of it without a terminal. The page leads with how things work now; what they replaced is in the archive at the end.

48% → 5%
of frames with one body part more than 2 cm inside another, on 13 characters: every clip now held inside joint limits measured on each character's own mesh
48% → 1%
of Cadet's frames with a boot under the floor - the shoe the viewer sees is the one planted now, every vertex of it
5 moves from 5 sentences
the motion library lacks, made for Cadet on his own skeleton by UniMate - about 25 s each on the GPU
1,529 → 9 faces
of Pip's texture sharing texels with another surface - his vest's paint on his sleeves. Every character had 224–2,602; now 0–9
wings → none
Pip's vest stays on his torso with his arms overhead; the rig's rule for it had never acted - its flood walked onto the vest
Studio :8830
make, watch, turn round and walk your own characters in a local app - python charforge.py studio
4 new
characters made with the pipeline as it is now - Knight, Kaito, Bo and Cadet
0.77–0.84 IoU
the rig's silhouette on the video's figure, four moves from video in three styles, every frame
30° → 8°
how far Aoi's hands point from the video's, turned to DWPose's reading - palms the right way 62% → 99% of frames
Licences. The videos of moves on this page come from MiniMax H3 (the Singularity community fine-tune). Its licence excludes the US, EU, UK and South Korea and covers what it generates; they are shown here as research. Qwen-Image 2.1, which draws the references, is under a research-only licence: commercial use needs a licence from Qwen. UniMate, which makes moves from words, is MIT-licensed, and its text encoder Flan-T5 Apache-2.0.
Make your own · a local web app, nothing uploaded anywhere

CharForge Studio

Everything this page describes, run from the browser: describe a character in a few words or drop in a picture, pick a style, press Make. Write it out shows what the image model will be asked for - the few words completed with an age and a build, skin and hair, and each garment in its own colour, by a local model - and lets you edit it; the height follows from it unless you set one. A name already taken is refused on the spot (a new prompt under an old name used to give back the old character). Jobs run one at a time in a queue that says what each is doing and how long it has left - learned from the stages' own logs on this Mac. Each character gets a page: turn it round in 3D, play its clips, compare its moves with the videos they came from, watch any clip from the front, the side and above at once (Angles), download the .glb and .fbx, add moves from video or from words (a sentence each, three samples to choose from - below), rerun from any stage. Walk around opens the playground with every character on this Mac, not only the published ones. The Studio can be closed and opened again while a job runs; it finds the job and waits for it.

cd charforge python charforge.py studio # opens http://localhost:8830 # or double-click "CharForge Studio.command" in Finder
The Studio's character grid: the fourteen characters on this Mac with their styles, heights and moves
Characters - everything in work/ and out/, newest first.
Aoi's page in the Studio: the 3D model on a grid, clip controls, her prompt and numbers, her move from video and the stages
A character - the model playing its clips, its moves beside their videos with their scores, every stage.
The New character form: 'a boy scout' written out as a ten-year-old scout in a striped shirt, khaki shorts and hiking boots, with the height set from it
New character - a few words, Write it out, edit what comes back; style, and a height that follows the description.
The Jobs page: the running job's stage and progress bar with the time left, and the queued jobs with when each starts and how long it takes
Jobs - the stage each job is in, its time left, when the next ones start; logs one click away.
The Angles panel on Mara's page, paused at the top of her roundhouse kick: the video, its camera, the left side and above, with the joint readout under them and play, frame-step and scrub controls below
Angles - any clip from four sides at once, the joint readout under it, and a frame at a time (, and .) to find the worst one. Paused here at the top of Mara's kick.
Cadet's page in the Studio: his model on a grid, his prompt and numbers, and the Moves from words panel with the salute's sentence, the sample in use and a contact sheet of its three samples
Moves from words - a name and a sentence for each new move (New move), three samples of each shown as contact sheets to choose from, and Use the chosen samples to add them to the character's clips.
Few-word prompts · three characters made in the Studio by a user

From a few words to a character

Made from the Studio with short prompts, they came out poor, and each for its own reason. "A boy scout" came back a grown man, at the height slider's default adult 1.72 m, and his model lost the picture's face and hair. "Gray alien that is extremely muscular" came back a grey clay sculpture - no clothes, no colour - and its 3D model a flat board with a body behind it. "A young knight in steel armor" gave back the Knight already made, in 0.0 minutes: the name was taken, every stage was cached, and the new prompt was quietly written over the old character's.

A prompt is written out first

A local language model (Ollama, on this Mac; unloaded as soon as it has answered) fills in what a few words leave open - an age and a build, skin and hair, the face, and the top, the bottom and the shoes each with its own colour - and the sentence is put together from its fields. Asked for free prose under rules, 3B and 7B models broke them one prompt in two ("holding a wooden badge", "a small shield", "standing at 1.5 meters tall"), so a field that names a prop, a pose or a height is dropped, and the age comes from your own words. "A boy scout" now reads: a boy scout, fit and energetic, light skin, short brown hair, cheeky grin and bright blue eyes, wearing a blue scout uniform shirt, khaki scout shorts and brown leather scout shoes, with a yellow scout neckerchief and a small blue badge. The Studio shows it before anything is drawn, to edit; a prompt of 20 words or more is used as written.

The height follows the description

A child 1.2-1.5 m, a teenager about 1.6, an adult about 1.75 - unless you set it. The slider no longer sends 1.72 for everyone.

A name already taken is refused

In the Studio as you type (with a free name offered), and by charforge.py make, which will not write a new prompt, style or height over a character that exists - rerun it from a stage for that.

TRELLIS's first pass seen from the front: Gray's is a flat grey board hiding the figure, Knight's has a board behind him, Kaito's a board through his legs; Bo's is clean
A board built into the model. TRELLIS's first pass from the front, with how its silhouette lies on the picture's: sound models 0.90-0.94, these 0.35-0.44 - the picture's plain backdrop made solid, in front of the figure, behind it or through it. Knight's membranes and Kaito's loose pieces were what was left of it after the later stages. Both passes are now checked from the front, and a model that does not match is made again on a new seed. Made again from the picture with its background cut away, all four match: Knight 0.92, Kaito 0.91, Gray 0.93, the boy scout 0.91.
The boy scout and the grey alien: the picture and the model as they first came out, and as they come out now
Two characters from a user’s few-word prompts, before and now. Made again with each prompt written out. "A boy scout" is a boy now, 1.20 m, in the scout's blue shirt and yellow neckerchief. "Gray alien, extremely muscular" is clothed and coloured, his 3D model whole (0.93 against his picture instead of a board), and his hands are his own grey and as broad as his forearms - at first the hand step gave him a person's peach hand, small on a forearm twice its width. His all-black outfit is the older prompt model's choice (What is left).
How a character is made now · steps marked new changed this round

Two lanes, one skeleton

A character is built once from a prompt or an image. Moves are then added to it from videos - one video per move serves every character, because the clip it becomes is a standard Mixamo skeleton.

Making a character charforge.py make · ~25 min

  1. The prompt, written outa local model completes a few words into a character - age and build, skin and hair, each garment in its own colour, nothing held in the hands - and the height follows from itdescribe.py
  2. Reference image, checkedQwen-Image 2.1 draws it in an A-pose with its own alpha; a pose model reads the arms and it is drawn again, up to three times, while the hands touch the bodyreference · pose_gate.py
  3. 3D model, checked from the frontTRELLIS.2, one pass; a model whose front silhouette does not lie on the picture's (a board built in with the figure) is made again on a new seed. The second pass on repainted side and back views is off: it made the model less like the picture on 12 of 12 characters (archive)generate
  4. Solid, joints, handsclosed into a solid, limbs traced to their tips, modelled hands with 15 finger bones eachsolidify · joints · hands
  5. Arms cut freeof whatever they were generated against below the armpit, then checked slab by slab and cut again - wider and further out - where an arm still reaches the bodyfree_arms.py
  6. Mesh, UVs, texturegame topology; UV islands folded onto themselves unwrapped again; the source images projected back on - not beside a nearer surface's outline - the hands in the face's skin toneretopo · texture
  7. Rigweights measured through the body; the arm's weight on the arm alone below the armpit; a vest's armhole with the collarbone and a jacket's hem with the pelvis, told apart by colourweights · rig
  8. Facejaw and five shapes, where eyes and a mouth can be foundface
  9. Clips, planted and kept apart18 Mixamo clips retargeted; the shoe the viewer sees planted, every vertex of it; each clip held inside joint limits measured on the character's own meshanimate · limits · clearance
  10. PackageglTF + FBX with LODs and a manifest; every clip can then be audited for aberrationspackage · aberrations.py

Adding a move from video charforge.py move · ~40 min each

  1. Write the movetimed beats - ready, the move, recovery - in MiniMax's six-section prompt formatmotion_prompts.json
  2. Generate the videoMiniMax H3 in ComfyUI: the reference image in, 5 s of the character doing it outmotion_video.py
  3. Read the poseViTPose's 17 points on every frame; DWPose's 21 per handvideo_motion.py · dwpose.py
  4. Nearest capture, fittedthe closest of 2,453 Mixamo clips, bent toward the video only where it misses by more than the pose model's jitter (-34% error on moves the library lacks)match · --fit
  5. Clip onto the rigretargeted, the character's own limbs, spine and head aimed along the fit, feet plantedretarget.py
  6. Refine against the videothe character's mesh re-posed frame by frame until its silhouette, joints, face and hands lie on the videorefine_pose.py
python charforge.py move --name mara --move punch_combo,roundhouse_kick python charforge.py move --name juno3 --move punch_combo@mara,spell_cast@aoi # other characters' videos

Adding a move from words tools/unimate_moves.py · about a minute each

  1. Write a sentencethe way UniMate's training captions are written - "An object marches in place."--move name=text
  2. Sample it on the character's own skeletonUniMate reads the packaged rig's T-pose and topology; three samples, 50 Euler steps on the GPU, ~25 s eachunimate_sample.py
  3. Choose onea contact sheet of each sample; the chosen one becomes an extra clip, like a move from video--pick name=n
  4. Onto the rigretargeted, planted and held inside the joint limits like every clip--rebuild
python tools/unimate_moves.py --name cadet --move "salute=An object stands at attention and salutes." python tools/unimate_moves.py --name cadet --pick salute=1 --rebuild # or the Studio's Moves from words
Toward a professionally made model · 3 October

Measured against models people made, then made closer to them

The question for this round was what makes a model look professionally made, and how to get ours there. A model-quality audit runs the same way on every character and on Blender Studio's Human Base Meshes and Ellie from Sprite Fright; the professional bodies score surface noise 0.74-0.83° and lumps 0.2-0.27 mm, ours were two to four times that. Then every character was looked at close up - face, hands, shoulders, back, feet - because a whole model's averages moved little between builds that look very different up close. The roster was rebuilt on what came out of it: over 9 characters the median noise went 3.07 → 2.92°, lumps 1.05 → 0.94 mm, loose pieces 13 → 1.

Knight2, Cadet and Gray as shipped, grey under a raking light, v0.11 beside v0.12: faceted plates and a jagged slot before, smooth plates with crisp edges after
the shape

A finer surface, filtered

15.66 → 14.85° median triangle tilt the normal map corrects (professional meshes 2.4-8.4)

The generator was asked for 200,000 faces, and simplifying to that turned gently curved armour into flat facets with sawtooth edges. Every later stage kept them - the solid is put back on the generated surface to keep its detail - so they reached the normal map as a triangle pattern under any grazing light. It now keeps a million faces, and the surface is filtered by bilateral normal filtering before anything is built on it: facets a few degrees apart become one surface, a plate's edge, tens of degrees sharp, stays. Quad remeshing refused our meshes outright.

Cadet's shoulder at rest, idle and walking: slivers and torn armour under the pauldron with the T baked in; the plate whole with the modelled pose kept
the rig

The pose it was modelled in, as its rest

3.44 → 1.446% Cadet's crushed faces over every clip; frames with a limb 2 cm inside another 40 → 28%

Every character was skinned up from the pose it was generated in to a T and saved in it, because the motion library's rest is a T. Skinning is linear blending and cannot be undone: Cadet's pauldron sheared into shards on the way up, and every arms-down frame came back down from that T with slivers. Now the T is only recorded, for the retargeting maths, and the rest stays as modelled. The clips are the same - foot slip identical in all nine gaits. Arms raised above the shoulder still stretch armour fused to the arm.

Knight2's back under a raking light: noise and floating shards in v0.11, a clean back in v0.12
the mesh

No loose shards

250 → 123 loose pieces over the roster

Every low-poly carried 2-94 floating shells of a few faces - where the modelled hands were unioned on, and where the decimation's repairs cut round bad edges - and they held nearly all its open edges. Pieces under a quarter of a percent of the faces are dropped now; a strand or a garment kept apart by design is hundreds of faces.

Cadet's hand from above and hanging, before and after: a pale patch across the back of the hand before, an even skin tone with soft shade between the fingers after
the texture

Hands shaded by their own shape

two-tone → even the hands' light and dark from their own occlusion

The modelled hands took their light and dark from a colour bake of the generated mesh - which has the old hand, or nothing, where the modelled one stands - so most hands carried pale and dark patches. They take it from their own ambient occlusion now: shade between the fingers, nothing elsewhere.

Mara's and the boy scout's faces at three-quarter view before and after: the skin evener after, the features unchanged
the texture

One skin

dE 16.6 → 1.5 the colour step at Mara's bare wrist

The skin's colour came from three places - the picture with its shading on the front, the generator's colours on the sides, one tone on the hands - so the same skin was three colours: a ring at every bare wrist, and on faces the picture's shading lying on a shape not quite the picture's. Skin, found by its hue and on the parts the parser calls skin, has its colour at the 1-2 cm scale of shading replaced by one tone and keeps its freckles and pores. It evens a face; it cannot move a feature onto the right part of one.

Shipped textures beside Hunyuan3D-Paint's views: a face-like pattern on the back of the boy scout's head, Bo's checks turned to dots, Knight2's back plate darker with a pale patch
what did not work

Backs painted by a multi-view model

6 of 8 characters worse by eye, though the texture audit scored them better

Hunyuan3D-Paint 2.1 paints a mesh's views from its own normals, consistent with the picture and each other, and cleaned Pip's blotchy back. On eight more characters it invented what the picture does not show and lost fine pattern: at the 256 px this Mac can run it at, it guesses. It is an option now, used for Pip - and the texture audit, which scored it better, needs a likeness measure beside its cleanliness measures.

Boyscout's head under a raking light: from the whole figure, ragged holes for eyes and a slit for a mouth; made on its own from a crop of the picture, brows, lids, lips and ears
the face

A head made on its own

7-8 min Hunyuan3D 2.1's shape model on a crop of the head

The most visible fault left is the face. In a whole-figure model the head is a few dozen voxels across, and the picture is projected onto a face shaped unlike it: seen from the side, its features and shading slide off. Made again on its own from a close crop of the picture, the head has the picture's shape. It is placed on the body through the picture both were made from, replaces the old head above the neck, and the picture is projected onto it; joining and painting it cleanly is still in progress (--head hunyuan). The next chapter takes the generated face apart instead, and mends each part.

Faces, part by part · 4 October

The face taken apart: eyes, triangles, features and skin, each mended on its own

After the shape and texture round, faces were what looked sloppy - eyes hollow or smeared, features not where the face has them, patches on a cheek. Taken one part at a time, each fault had its own cause, and each part its own fix. Instead of rebuilding every character, a benchmark set of two per style and gender - realistic Mara, Juno, Rowan and Knight, stylised Vex, Rivet, Pip and Cadet, anime Aoi, Hana, Kaito and Ren - is scored before and after: landmark error against the picture, and how alike the face's parts look by a vision model's features, robust to lighting but not to a smeared eye. Over the 10 with a v0.12 build to compare, the eyes' likeness went 0.633 → 0.688, the mouth's 0.527 → 0.637. The gain is the realistic faces' (eyes 0.514 → 0.628), where every part had been misplaced and the eyes were pits; drawn faces held, and those left wrong are wrong in shape - Ren's head is longer than his picture's.

Mara's head in plain grey at five stages: ragged almond eye openings as generated, streaked slabs in the solid, crumpled pits in v0.12's low-poly, and smooth lids over the eyes once rebuilt
the shape

Eyes rebuilt as a smooth cap under the lids

17 → 28.35% of the triangles on the head; eyes rebuilt on 11 of 12 benchmark characters

The generator models every eye as an almond opening with shards inside it. The solid filled it with streaks, and the decimation, which gave the whole head 17% of 60,000 triangles, crumpled it into a pit - and no texture laid on a pit reads as an eye. More texels on the head changed nothing you could see. The openings are found by a pose model on the generator's own paint (on a grey render of a drawn face it only guessed), and rebuilt in the solid as a smooth cap across the opening that follows the lids and bulges a little like an eyeball under them - an eyeball set behind the opening left a slot that rendered as goggles. The head now keeps 28% of the triangles. Anime eyes the pose model cannot read stay as generated.

Mara's face lit from the front: the picture; v0.12 with hollow, smeared eyes; the eyes rebuilt but squinted by the old face flow; laid on in parts with dark patches on one cheek; v0.13 with the picture's face and clean skin
the texture

The face laid on in parts

8 of 12 benchmark faces laid on in parts

The picture's face had been projected larger than the sculpted one, its eyes 10-30 pixels off, and the optical flow meant to line it up squinted and smeared them. Now each part - each brow, each eye, the nose, the mouth, the jaw - is moved by its own small fit onto the model's own features, and the moves are spread smoothly over the face. Two more sources of patches went: the generated side views, whose faces never sit quite where the picture's does, no longer paint the face; and the skin-evening step, whose colour bands cut the picture's soft cheek shadow into blotches, leaves the face alone.

Knight's generated head in its own colours from the side: crackled with short grey lines as generated, dark flecks when mended in the atlas, clean skin and hair when rebuilt through the mesh
the texture

The generator's colours without the crackle

1024 px for a million faces: most triangles smaller than a texel

Every surface the picture does not paint - the side of a face, an ear, the hair, the back - carried a crackle of short grey and dark lines. They were not holes in the mesh and not dark specks: the generator paints its colours into an atlas of thousands of tiny charts, and along their edges leaves texels sampled off the surface. Mending them from the atlas's neighbouring texels borrowed other charts' colours. Rebuilt through the mesh instead - each vertex given the median colour of its triangles, outliers given their neighbours', the atlas painted again at four times its size - the skin and hair come out clean.

Before and after: Mara's and Rivet's right arms ending in a stump, then with a hand; the back of Vex's head with a sculpted face, then with hair; Bo's back with two rectangular holes, then with thin lines
what a glance found

A missing hand, a second face, holes in the back

5 characters had shipped without a right hand; all 17 now have both, checked by fingertip

You saw it at once and no measure had: five characters had no right hand. When one hand's join to the arm failed, it was attached as an overlapping shell, and the other hand's join, run on that tangled body, quietly returned it unchanged - the only check was that the body did not shrink. Every hand is now joined to the clean body first and checked by its fingertips, and a build with a missing hand stops. Vex had a second face sculpted on the back of her head, which the generator invented; every new model is now looked at from behind and made again if a face is there. And the cut that frees the arms from the body had run through eight characters' backs; what it took beyond the arm now goes back. The vision models in the local library can tell a hand from a stump every time, but asked for any defect they flag too much to be a gate - they say which close-ups to open first.

The twelve benchmark characters' faces from the front in soft light, v0.12 beside v0.13
the benchmark

Twelve faces, before and after

0.032 → 0.027 landmark error against the picture, as a share of the eye spacing

The same twelve characters from the front, v0.12 beside v0.13. Landmark error alone missed the improvement - a smeared eye is still where an eye should be - which is why the likeness of each part is measured too. Still open: the hairline (the hair shell's ragged edge), anime eyes, and the face from the side, where it is still the generator's shape.

Polish · every character in the poses that break things, front-left and back-right

What broke when the characters moved, and what fixed it

Each character was rendered at the extremes of its clips and moves - the jump's crouch, the landing, the wave, the fall, arms overhead - and every fault traced to its cause before anything was changed. Twice the obvious cause was the wrong one, and the fix that looked right changed nothing; those are in the archive. Left of each pair: this morning. Right: now.

Pip from behind in five poses, before and after: before, the yellow vest flares out like wings and a pocket juts from his thigh; after, the vest stays on his torso
the rig

Pip's vest no longer lifts into wings

1,190 + 1,236 vest vertices now let go of the arms (before: 0 and 5)

Three causes, stacked. The 3D model glued his arms to the vest's sides - now cut free below the armpit. The rig's rule that keeps an arm's weight on the arm floods the arm's own surface from the sleeve's outside, and it started at the very slab where the cut opens, where sleeve and vest are still one surface: it walked onto the vest on both arms and changed nothing. It starts a little lower now. Above the cut, the vest's armhole is the arm's surface and took half its weight, as any shoulder does; the vest's yellow and the sleeve's blue tell them apart, and the armhole now moves with the collarbone.

The cargo pocket that stood off his thigh like a plank: the part labeller calls it an accessory, and accessories rode the torso. Below the hip joints one now stays on its leg (off-thigh vertices in the landing 122 → 25).

Pip's victory cheer with both arms straight up, before and after: before, the vest's sides rise with the arms; after, the vest stays down and the armpits stretch
the rig

Arms overhead

vest down armpit stretches

With both arms straight up the vest stays on his torso. What is left is the stretch between vest and sleeve at the armpit: they are one surface, and a surface joining a garment that stays to a sleeve that rises has to stretch somewhere. The fix for that is the vest as its own layer.

The back of Pip's upper arm: before, patches of the vest's yellow on his blue sleeve; after, clean blue
the texture

No garment carries another's paint

224–2,602 → 0–9 faces sharing texels, per character

The patches looked like a projection error, and three rounds of fixes to the projection changed nothing. The cause was the UV layout: unwrapped on a smoothed copy of the mesh, islands folded onto themselves wherever it curls, and two surfaces painted the same texels - 1,529 faces on Pip, 2,602 on Vex. Repacking cannot part an island from itself; the overlapping faces are unwrapped again, finer each time, and every island brought back to one texel density (unwrapped alone, they took 42% of Pip's atlas).

Aoi's trouser texture before and after: before, pale slashes across the dark fabric; after, clean
the texture

Aoi's trousers lost their light streaks

10–20 px her side view's hands stood off the mesh's

Not the generated hands the modelled ones replaced (masking them changed 12,513 texels and left the streaks): in the generated side view her hands hang a little lower than the mesh's, and the trouser texels just past the mesh's hand read the picture's hand. A generated view now fades out beside the outline of anything nearer - a depth jump between neighbouring pixels. Not in the reference, which the mesh was made from: half of Aoi's face lies behind a lock of her hair.

The same pass found the modelled hands' region reaching into her hip (36,482 texels painted skin) and 161,553 texels of Pip: a texel is the hand's now only if it stands outside the body the hand was joined to.

Pip's hands before and after: brown and grey before, peach like his face after
the texture

Hands the colour of the face

(137, 95, 71) → (222, 152, 122)

The modelled hands took the median of every skin-like colour on the body - on Pip that is his orange hair, khaki shorts and the vest's shading as much as skin. The face's own texels decide it now.

Juno crouching, before and after: before, the red jacket's hem is pulled down between her thighs; after, it hangs level over her hips
the rig

A jacket's hem follows the hips, not the thighs

1,453 vertices of Juno's jacket moved from the thighs to the pelvis

The lowest part of Pip's vest carried half its weight on his thighs (median 0.50), and spreading the legs in a jump pulled its hem into a W; Juno's windbreaker wrapped round her thighs in a squat. Where the torso's colours and the upper thighs' differ, the upper garment near the hips follows the pelvis, and a long jacket still rides on the seat below it; one colour top to bottom - Mara's olive and tan are 5 apart - nothing changes, and her kick still drags her jacket along the raised thigh.

Knight waving and jumping, before and after: before, grey sheets hang from his arms to his hips; after, none
the solid

Webs between an arm and the body - most of them a board

8,286 + 10,349 → 1,877 + 2,326 voxels of web between Knight's arms and body

Waving stretched grey sheets from Knight's arms to his hips. Cutting thin webs out of the solid before the arm is freed took some (anything under about 9 mm goes, and pieces the cuts split off are dropped), but fins and a strap remained - because most of it was not a web: TRELLIS had built the picture's backdrop into his model as a board behind him (see Few-word prompts), and the later stages cut it into sheets. Made again through TRELLIS's own background remover, his model has no board, the web cut finds a fifth as much, and the sheets, fins and strap are gone.

Kaito crouching: before, a hair spike floats beside his head, left where the rest pose had it; after, it moves with his head
the rig

Loose pieces ride with the body

37 loose pieces on Kaito, now each riding with the body part nearest it

Kaito's mesh came as 40 pieces: the body, the two modelled hands, and 37 bits the generator left detached - the inner toes of his split-toe boots, hair spikes. The weights follow paths through the body and never reached them, so they stayed where the rest pose left them. Each now rides rigidly with the body part nearest it.

Aoi's spell cast at 1.5 and 2 seconds: the video; before, at 1.5 seconds her left jacket sleeve sticks far out sideways past an elbow folded back on itself; after, both elbows out as in the video with her hands in front of her chest. At 2 seconds the elbow was already within range and nothing changes
the move

Elbows no longer fold back through the arm

176° → 144° her most folded elbow in the spell; the motion library's reach 159° at most

In her spell cast Aoi presses her hands together in front of her chest, elbows out. One camera cannot say how far a bone reaches toward it, so the fit takes each bone's depth from its length in the image - up to a sign - and it took the matched capture's side for her upper arm: tipped back. The elbow came out folded to 176°, the forearm back through the upper arm, and her jacket sleeves, which move with the upper arm, stuck straight out sideways past the fold. The hand score caught it: the pose model read those sleeve ends as her hands.

The fit now chooses an upper arm's side and its forearm's together. An elbow folded past 150-160° costs - the motion library's pass 150° in 0.05% of frames - and so does a wrist within the chest's width and height put behind the torso, as 1% of the library's such wrists are. That alone brought this elbow to 145°: where the video shows an upper arm at its full length there is no side left to choose (the pose model puts her elbow at the outside of a puffy sleeve). So the refine stage also holds every elbow under 150°: 144°, with the silhouette and the hands where they were. On 40 library moves it had never seen, the new fit is as close as the old (0.212 → 0.211 torso lengths).

Aoi's spell cast from the side at 0, 1 and 2 seconds: before, both boots hover above the grid's floor line; now they stand on it
the move

Feet back on the floor after the refine

3.1 → 0.6 cm Aoi's lowest shoe above the floor, most of her spell cast

Seen from the side for the first time, Aoi's spell cast stood 2-6 cm in the air. The retarget had put her planted soles on the floor (0.02 cm off it on average); the refine then turned her hips, knees and ankles toward the video, which lifted them, and nothing put them back. The correction step now measures each key's lowest shoe before the turn and after, and moves the pelvis by the difference - the lowest foot goes back to where the retarget had it, on the floor while planted, as high as it was in a jump. Where her feet run off the bottom of the video's frame, the video cannot say where the floor is, and the retarget's planting is kept: that stretch is the 1.9 cm left.

Bo the chef: the first reference with his fists on his hips, the model it made with shredded hands at his belt, and the reference drawn again with his arms clear
the reference

A picture in the wrong pose is drawn again

0.33 torso lengths from Bo's wrists to his hips; good references 0.60–0.79

Asked for an A-pose in so many words, the image model drew a chef's habitual stance - fists on hips - and 32 minutes later the fused hands came out in shreds at his belt. A pose model now reads the arms of every new reference, and below 0.45 it is drawn again on a new seed. A picture you give is only warned about.

Aberrations · every frame of every clip, measured on the mesh the viewer sees

Measured instead of looked for

The polish pass found faults by rendering the poses that break things and looking, which finds what someone thinks to render. The aberration audit (tools/aberrations.py) poses every frame of every clip with dual quaternion skinning, as the playground does, and counts what a viewer would object to: faces crushed to under half their area, stretched past double, or turned more than 60° from their bone; one body part more than 2 cm inside another; a shoe through the floor; a surface that pops for one frame. Before any of it was believed it was run on a synthetic character with aberrations of known size: 22 of 22 checks pass under both skinnings, and the three that failed at first were errors that would have misled every number below - quads re-split by the pose, a depth search that stopped at 3.1 cm, and "inside" read past the open end of a sleeve.

2.72 · 1.52 · 2.71 %
of faces crushed, stretched and sheared, averaged over every clip of 13 characters - most of the crushed ones out of sight, under an arm
48% → 5%
of frames with one body part more than 2 cm inside another - the commonest fault - before and after the joint limits
3.89 → 1.27 cm
the deepest one part goes into another, averaged over clips; crushed, stretched and sheared shares unchanged
Bo in idle, strafe and sprint before and after: before, his forearms sink into the sides of his chef's jacket and his raised heel folds into the back of his thigh; after, his arms hang just clear of the jacket and the raised leg bends less
the clips

Limbs keep out of each other

7–53° the largest single correction per character; armour takes the most - Cadet's sprint, his knee plates meeting at 105°

A capture was performed by one person of one build. On a thicker character the same joint angles put a forearm inside a puffy sleeve and a heel into a thigh, and hang the arms inside a wide torso. Each character's elbows, knees and shoulders are now bent on its own mesh, 5° at a time, until one part goes 2 cm into another, and every clip is held inside those limits: an arm past its limit moves to the nearest allowed direction, an elbow or knee opens about its own hinge - a knee only while its foot is off the floor, so planted feet stay put. The last 10° of each limit are soft, so no correction has a corner in time. At 1 cm Mara's boxing guard was forced open by 25°; at 2 cm it is left alone.

What it cannot see is what one joint's limit cannot: hands swinging into thighs that are themselves moving, legs brushing past each other in strafes, arms through long hair.

Cadet walking and crouch-walking, from the front and from behind, before and after, with every face more than 1 cm below the floor painted amber: before, a boot goes through the floor in each view; after, none
the feet

The shoe the viewer sees is the one planted

48% → 1.3% of Cadet's frames with a boot more than 1 cm under the floor

Cadet's armoured boots went through the floor - in a walk, in a crouch walk - while the solve reported his feet planted. It planted a copy of the sole carried rigidly on the ankle bone, which on an ordinary shoe is the shoe. Cadet's ankle joint sits high inside a big boot, the upper boot blends with the shin, and the copy stood level on the floor while the boot on screen went under it. The planted shoe is now posed the way the viewer deforms it - each vertex by its own skin weights, with dual quaternions, matching Blender's own to 0.01 mm - and the whole shoe below the ankle counts, toe cap included: a planted key pivots on the flat of the sole, a key in the air keeps all of the shoe above the floor.

The first version thinned the shoe to 400 points. The median slip fell, but backpedals and strafes slid more - Juno walking backwards 6.0% → 11.4% of ground speed, Pip strafing 15.3% → 20.5% - at the same ground speed. Thinned, the shoe's lowest point jumped from one sample to the next as the foot rolled, and the planted pivot jumped with it. With every vertex (1,016–3,266 a foot) Juno is at 6.0% and Pip 7.8%; across 13 characters and nine gaits the worse foot slides a median 4.0% → 2.9% of ground speed.

New characters · made start to finish by the pipeline as it is now

Knight, Kaito, Bo and Cadet

Four new characters in the three styles, each from one sentence, made by the pipeline as it stood - the reference checked, the arms cut free and checked, the UVs repaired, the rig's rules for garments applied; Cadet, the last, also with every clip audited, the joint limits and the shoe planted as the viewer sees it. Each card: the video is the game rig walking, waving and turning; below it the reference, the model at rest and the model mid-jump.

Knight

realistic

"a young knight in dented steel plate armour over a blue surcoat, short blond hair, leather gloves and brown leather boots"

height
1.80 m
triangles
61,927
made in
about 35 min
notes
Three failures on the way, each now a rule: macOS stopped his 3D model on the GPU (it is evaluated in smaller pieces now); his armour's texture failed the web build's quality gate (it steps up the quality instead); and TRELLIS built the picture's backdrop into his model as a board, which the later stages cut into grey sheets hanging from his arms - made again through TRELLIS's own background remover, it has none. Last, his face: his picture's would not lie on his model's, and went on unaligned with every feature twice; his generator's own face is kept now, and with eyes it can find, the face stage gave him blinks and a jaw.
Knight's reference image
reference
Knight at rest
model
Knight mid-jump
jump

Kaito

anime

"an anime ninja boy with spiky black hair, a long red scarf, a fitted dark blue jacket with a belt, black trousers wrapped at the shins and split-toe boots"

height
1.70 m
triangles
98,781
made in
49 min, sharing the GPU
notes
His first model was 40 pieces: the picture's backdrop built into it as a board through his legs (front match 0.35), his boots' inner toes and hair spikes detached. Made again from the picture with its background cut away (0.91), it is one solid - 7 specks of 80 vertices ride with the part nearest them - and his open sandals have whole toes.
Kaito's reference image
reference
Kaito at rest
model
Kaito mid-jump
jump

Bo

stylised

"a cheerful cartoon chef with a tall white chef's hat, round glasses and a thick moustache, a white double-breasted chef jacket, black-and-white checked trousers and red clogs"

height
1.60 m
triangles
59,894
made in
about 35 min
notes
His first reference had his fists on his hips; the model fused them and shredded his hands at his belt. The new reference check rejects that picture and draws him again.
Bo's reference image
reference
Bo at rest
model
Bo mid-jump
jump

Cadet

stylised

"a fresh-faced young space marine, about 19 years old, short messy brown hair, in chunky rounded sci-fi power armour: teal-blue armour plates with white trim, a dark grey flexible undersuit showing at the elbows, knees and waist, big rounded shoulder pads, sturdy armoured boots, plain smooth armour with no emblems, no insignia, no logos, no skulls and no markings"

height
1.75 m
triangles
96,880
made in
about an hour
notes
A young space marine with the emblems left off, from a few words in the Studio, written out by the local model. Two faults on the way, each now fixed for every character: "about 19 years old" was read as an elder (the word "old"), and his armoured boots went through the floor while the solve reported them planted (Aberrations). The texture stage took 20 of his minutes, slowed by memory swapping. The first character with moves from words (below).
Cadet's reference image
reference
Cadet at rest
model
Cadet mid-jump
jump
Moves from words · UniMate on the character's own skeleton

A move from a sentence in half a minute

The motion library has no salute, no boxing guard, no march. A move from video works (below), but each costs half an hour of video generation. UniMate generates a skeletal motion from a sentence for a skeleton it has never seen: a flow-matching model conditioned on the skeleton's topology and T-pose as well as on the text, which Flan-T5 encodes. So it runs on each character's own rig, and the move then goes through retargeting, foot planting and the joint limits like any captured clip. Cadet got five: a salute, a ready stance, a fist pump, a march and a look around.

Cadet, a young space marine in teal armour, in five moves generated from text: a salute with his right hand at his brow, a wide combat-ready stance with fists raised, a raised fist, a high-knee marching step, and standing upright looking to one side
Cadet's five moves from words, each the best of three samples: salute, ready stance, fist pump, march, look around.
How it runs on the Mac

UniMate's own preprocessing reads the packaged character - its canonical T-pose, its topology, its clips - through Blender 4.0 as a Python module, the version it was written for. Its sampler integrates the flow with an adaptive solver that needs double precision, which Apple's GPU does not have; a fixed grid of 50 Euler steps runs on the GPU in about 25 s per two-second move (the adaptive solver on the CPU took over 10 minutes for three). Three samples of each sentence, a contact sheet of each, one chosen, about 11 s to export. tools/setup_unimate.sh installs it - code, weights, the text encoder and two Python environments, about 3.2 GB.

Write it the way its captions are written

3 of the five came out as asked on the first wording. "March in place" and "look around" needed UniMate's own caption style: "An object marches in place.", "An object looks to the left and then to the right.".

Two things broke, quietly

The face rig's shapes were not moved into UniMate's frame with the mesh, and tore even Cadet's own clips: they are left out of UniMate's copy now. And UniMate's pruned skeleton ends at the toe, which the FBX importer drops as a leaf - so foot planting, which needs it, switched itself off without a word, and the generated moves stood 3-19 cm into the floor, sliding. The toe is now recovered from the foot bone's tail, and retargeting says so when it cannot plant.

On the surface the generated moves fall inside the range of his captured clips: 3.18–4.09% of faces crushed against 1.47–4.96%, and no pops. Standing still, the salute's feet slid at 0.065 m/s and the ready stance's at 0.579 m/s before the toe was recovered and the shoe planted as the viewer sees it; now 0.002 and 0.088 m/s. The march still scrapes its toe at every lift and landing (0.861 m/s): planting a contact does not remove a stomp. Choosing among the three samples is still done by eye.

Moves from video · 4 runs

The video, and the rig doing it

Left of each clip: H3's video, generated from the character's single reference image and the written move. Right: the character's game rig playing the clip read off that video, from the same camera angle - as the pipeline makes it now, with every fix in the next section.

Mara · punch combo

realistic
video
35.6 min for 5.2 s
nearest capture
cost 0.167
image error
0.130 → 0.034 torso lengths
vs the video
joints 0.054 · IoU 0.843

Guard, jab, cross, hook, back to guard. The closest of the four now: joints about 5% of a torso length from the video's on average.

Mara · roundhouse kick

realistic
video
48.3 min (more swapping)
nearest capture
cost 0.254
image error
0.194 → 0.043 torso lengths
vs the video
joints 0.071 · IoU 0.831

A head-height kick with the torso leaning 30°. At its top the rig's leg is up at head height as in the video (four sides); the pose model misreads the render of that leg for 7 frames, most of this score - without them, joints 0.049.

Aoi · spell cast

anime
video
32.0 min
nearest capture
cost 0.294 (no real match)
image error
0.213 → 0.035 torso lengths
vs the video
joints 0.119 · IoU 0.771

H3 kept her cel-shaded look from one reference. No clip in the library casts a spell; her long legs and short torso are what bent the old fit.

Pip · victory cheer

stylised
video
34.6 min
nearest capture
cost 0.263
image error
0.151 → 0.042 torso lengths
vs the video
joints 0.068 · IoU 0.818

His elbows were rigged 4 cm too far down the arm - fixed. With his arms overhead his vest now stays on his torso (Polish).

Every move from four sides at once · tools/make_angles.py

The same frame from the video's camera, the side and above

One camera cannot see depth: from the video's angle a clip can look right while an arm folds back through itself or the feet float. Each move is now rendered from several cameras at once, side by side in one video with the video itself - every panel orthographic and at one scale, on a 10 cm grid, so what lies in front of what, and how far, reads straight off it - and each elbow's and knee's bend and each shoe's height above the floor are read out under the panels, red where they go further than the motion library ever does. The Studio has it for any clip of any character (Angles).

Aoi · spell cast click to pause
Mara · punch combo click to pause
Mara · roundhouse kick click to pause
Pip · victory cheer click to pause

It found what the front view hid - Aoi's elbow folded back through her arm, and the feet of every move from video standing off the floor after the refine (Polish). And two things about the checking itself. The video beside the rig ran 25% fast: the 24 fps video was stretched onto the renders' 30 fps by taking each frame once, never twice, so Mara's jab seemed to land 0.2 s late and her last guard held while she kept punching. Every compare video here has been remade in time. And the dip in her kick's score at its top is the pose model misreading the render, not the rig: from the side and above her leg is up at head height as in the video.

How close to the video · the same pose model on both, every frame

Making the rig move the way the video does

The measure: the character's clip is rendered at the video's timing from the video's camera angle, and one pose model reads both. Joints - their mean distance from the video's, in torso lengths. Silhouette - how much of the character's outline overlaps the video's figure (IoU, 1 is identical). Face - the nose, eyes and ears about their own centre: which way the head turns. At rest the 3D model already lies on its reference picture at 0.90-0.93 IoU, so nearly all of what was missing was in the motion.

Silhouette overlap with the video

Mean over 124 frames; grey at the start of the day, blue as the pipeline makes it now.

startnow
00.250.50.751 Mara · punch0.750.84Mara · kick0.620.83Aoi · spell0.610.77Pip · cheer0.620.82

Joints, silhouette, face - step by step

Joints in torso lengths (lower is better), silhouette IoU (higher is better). Each scored from the fit's own camera estimate - no search for a flattering angle.

movestartown body + aimed limbsnowface now
Mara · punch0.086 · 0.7510.073 · 0.7960.054 · 0.8430.012
Mara · kick0.187 · 0.6220.121 · 0.7470.071 · 0.8310.013
Aoi · spell0.214 · 0.6090.167 · 0.6510.119 · 0.7710.032
Pip · cheer0.127 · 0.6160.114 · 0.7140.068 · 0.8180.020

How the rig follows the video

  1. Fit on the video's own bodyeach bone as long as the video shows it at its longest. On the Mixamo actor's lengths, Aoi's short torso - her legs 1.03 torso lengths to Mixamo's 0.80 - came out as a 28° lean back. Fit error 0.061 → 0.035.video_motion.py
  2. Aim the character's own limbscopying rotations carried the two skeletons' rest-pose differences into every frame - thighs 4-12°, collarbones 16-27°. Aimed along the fit, the rig lands within 0.5° of it.retarget.py
  3. Turn the head to the facethe capture's head looked down and aside; the video's nose, eyes and ears, held against where the pose model finds them on the character's own head, cut face error 2-3 times.calibrate_joints.py
  4. Refine against the videothe character's skinned mesh, posed in PyTorch and seen by the video's camera, has 20 bones nudged each frame until its silhouette, joints and face lie on the video: +0.07-0.10 IoU on its own.refine_pose.py
  5. Put Pip's elbows backrigged 63% of the way down the arm - measured along a centre line that wandered through his puffy sleeve. Now 55%, joint to joint, where the video shows him bend.refine_joints.py

Before and now

Aoi in frames 24 and 72: the video, the old rig with her head tipped down and arms low, and the rig now standing upright with arms raised
Aoi, frames 24 and 72. Before: head tipped down, body twisted, arms low. Now: upright, head level, arms up. Her palms face the camera in the video and down on the rig - hands are not tracked yet.
Pip in frames 24 and 60: the video, the old rig with his head thrown back, and the rig now facing forward with arms overhead
Pip, frames 24 and 60. Before: head thrown back, arms short of overhead. Now: facing forward, arms up as in the video. (The vest's wings in the before picture are gone too - see Polish.)
The top of Mara's kick before the refine: in the video the leg points up at head height while the pose model's leg points are bunched near her hip; the rig's leg is folded

What is left

The pose model's blind spots. This picture is from before the refine, when the rig followed the pose model's folded reading of the kick. Now the rig's leg is up at head height as in the video, seen from the side and above (four sides), and the skeleton is 0.12-0.14 torso lengths off the video at the top, as through the rest of the kick. What dips is the score: reading the render, the pose model puts her right ankle 2.9 torso lengths from where her skeleton has it, in frames 61-67. Without those frames the kick scores 0.049 instead of 0.069. What the rig still does wrong there: it lowers the leg a little early - at 2.8 s her knee is bent 70° while the video's leg is still up.

Parts fused to the body. Aoi's hair is one surface with the back it lies on, so it cannot swing apart from it. Pip's vest now moves with his torso (Polish).

Tried: DWPose, a whole-body model (133 points - body, feet, face and 21 per hand; 134 MB, Apache-2.0). It does not rescue the kick: at the apex it folds the leg too, the ankle at her fists at confidence 0.16-0.21, where ViTPose's three good frames (62-64) follow it to the foot at 0.58-0.68. It does read hands, at median confidence 0.75-0.88 - and the rig's hands now follow them (Hands).

Three panels: Mara at the top of her kick with ViTPose's skeleton reaching the raised foot; the same frame with DWPose's skeleton folding the leg at her fists; Aoi with DWPose's finger points on both open palms
The top of Mara's kick (frame 63) read by both pose models, and Aoi's hands (frame 72). ViTPose follows the raised leg to the foot, ankle confidence 0.68; DWPose folds it, the ankle at her fists, 0.16. On Aoi's open palms DWPose places every finger - what ViTPose's 17 points never reach, and what the rig's hands should turn to next.
Hands · DWPose's 21 points a hand, on the video and on the render

Turning the hands to the video

ViTPose's 17 points stop at the wrists, so every clip kept the motion capture's hands: Aoi's palms faced down where the video's face the camera, and Pip's hands hung half open where the video makes fists. DWPose reads each hand as 21 points - the wrist, three joints a finger and the tips - and the refine stage now turns the wrists, the forearms about their length and the thirty finger joints until the character's hands lie on that reading. Each finger joint bends only as a finger does - toward the palm, and apart at the knuckles - within a finger's range. The same model reads the render to score it.

Aoi's spell cast at frames 52 and 72: the video, the rig before with curled palm-down hands, and the rig now with open hands, palms to the camera
Aoi's spell, frames 52 and 72. Before, the capture's hands: curled, palms down. Now: open, palms to the camera and fingers spread as the video draws them - the hand points 8° off the video's, from 30°.

The hands, scored

DWPose on the video and on the render, every hand it is sure of: the points about the wrist in palm lengths (lower is better), how often the palm faces the video's way, how far off the hand points. Re-scored on 26 September after the rebuilds, with the median beside each mean. Aoi's mean is carried by 21 frames (53% of it) where the pose model reads the light cuff of her jacket sleeve, sticking out past a strongly bent elbow, as her hand - her own hands are where the video's are; see What is left. Body scores held: joints, silhouette and face within 0.006 of before on all four moves.

movehand errorpalm facingpointing
Mara · punch0.24 → 0.20 median 0.1793% → 97%10° → 8°
Mara · kick0.24 → 0.14 median 0.1251% → 100%12° → 5°
Aoi · spell0.64 → 0.30 median 0.1562% → 97%30° → 16°
Pip · cheer0.23 → 0.20 median 0.1999% → 99%7° → 6°

What it took

  1. Only the hand turnsallowed to swing the arm, the hands lifted Aoi's to her chin where the video holds them at her chest (silhouette −0.005); held to the wrist, the forearm's twist and the fingers, the arm stays where the body's terms put itrefine_pose.py
  2. Weighted up, since it can no longer bend the bodyten times the first weight: Aoi's hands 0.32 → 0.24 palm lengths, her body's scores the same--w-hands 0.2
  3. Where DWPose sees a wriston Pip's cartoon hand, at his sleeve's cuff - 4 cm up from his wrist bone. Measured on his own wave, idle and jump and trusted as far as it holds from pose to pose: his hands 0.21 → 0.18calibrate_hands.py
Running it on the Mac · MiniMax H3 is a 33B-parameter model

What it took to run H3 on 24 GB

H3 is built for data-centre GPUs. The fine-tune ships in ComfyUI's 4-bit format; three problems stood between it and this laptop.

Apple's GPU has no float8
11.8 → 15.4 GB, lossless

The fine-tune's per-group scales are float8, which PyTorch on Apple GPUs cannot hold - the first layer failed. They are rewritten as float32 once (f8_scales_to_f32.py, 21 s); a real layer then matched the original on the CPU to 0.036%.

ComfyUI's LoRA patching swapped the Mac to a halt
20 min stuck → loads in 1 min

On a Mac ComfyUI loads a model whole and keeps a backup of every weight a LoRA patches: 15 GB became 30. It also requantizes each weight - 4.5% error to carry the turbo LoRA's 0.02% change. Baked in offline instead, on each layer's own 4-bit grid: 0.1% of codes moved, 0.5% noise, 7.5 min on the CPU (the GPU took 53 s a layer to the CPU's 1.1).

Time per clip
32–48 min / 5 s

Text encoder (Qwen3-VL 32B) on the CPU: 87–108 s. Video model load: ~1 min. Four sampling steps: 29 min at 6–7.5 min each - attention over ~13,000 video tokens is the cost. Decoding: ~4 min. Everything else on the Mac slows down while it runs.

A bug the preview caught
elbows 173° → 54°

The first clip on Mara's rig had its arms stuck out in a T-pose: Blender had exported the boxing guard as the skeleton's rest, so every guard read as "no change from rest". The FBX now exports with the capture's T-pose showing, and the round trip checks it.

macOS stops a GPU job that holds the display up
6 blocks → 2 a buffer

Knight's and Kaito's 3D models died at TRELLIS's 1024 stage (~8,000 tokens) with "Impacting Interactivity": macOS kills a GPU command buffer that runs too long while the screen needs the GPU. The flow model now evaluates two transformer blocks at a time, and a killed pass is retried once a block at a time with fewer operations per buffer. Both then built.

What is left · measured, ranked by what it costs a character

Open problems

Archive · methods and experiments this page used to lead with

What was replaced, and what was tried

Kept for the numbers and the reasoning: each of these was measured, and several obvious-looking ideas in them are worse than what ships. Open one to read it.

A second 3D pass on repainted viewsoff since 30 September: it made the model less like its picture

A second 3D pass on repainted views

Until 30 September TRELLIS was run twice. The first model's renders from the side and the back were repainted by the image model, and a second pass took those as extra views. Against a generated 360° video of Mara it had moved the mean silhouette IoU a little; timed, it was the slowest stage (16.1 of 41.3 minutes, median). Compared on 12 finished characters, the second model fitted the reference's front worse on 12 of 12 (mean IoU 0.923 → 0.908), and looked at against the pictures it follows the repainted views, not the reference: the boy scout grew knee socks and lost his grin, Vex's trainers became boots. Repainting views the reference does not show invents them, and the second pass builds what was invented. It is off by default; --quality best keeps it.

The boy scout's reference beside his two 3D models from the front, left, back and right: the second pass gives him knee socks and loses his grin
The boy scout's reference and his two models: the first pass, and the second on repainted views.
Planting a rigid copy of the solereplaced on 2 October by the shoe posed the way the viewer sees it

A rigid sole on the ankle bone

Until 2 October the foot planted was a copy of the sole - the shoe's vertices within 3 cm of the floor at rest, at most 160 - carried rigidly on the ankle bone, its lowest point held on the floor while the foot was down. That is the shoe, on an ordinary shoe. On Cadet's armoured boots, which blend with the shin above a high ankle joint, the copy stood on the floor while the boot on screen was under it in 48% of his frames. Across 13 characters and nine gaits the worse foot slid a median 4.0% of ground speed with it, against 2.9% with the shoe posed as the viewer sees it.

Foot slip as first measureda 1 m/s floor under the speed - replaced by the true-speed audit

Foot slip as first measured

The first foot audit divided each gait's slip by max(manifest speed, 1 m/s): below 1 m/s the ratio was understated, and standing still it is undefined, so slow entries were left out. On 9 characters it put the worse foot's median slip at 3.5% of ground speed (worst 18.2%, 78 pairs). The audit now divides by each gait's true ground speed and reports metres per second as well; on 14 characters, every gait, see the table in the paper. The old table, as it was:

gaitAoiJunoKaitoKnightMaraPipRowanVexWren
walk1.0%2.7%1.3%1.5%2.5%4.5%3.6%4.6%7.5%
jog1.7%3.8%1.6%1.3%2.7%3.3%2.0%2.5%2.6%
run0.7%2.3%1.2%1.0%2.7%2.9%3.3%2.2%2.2%
sprint1.1%2.9%4.3%1.1%5.4%8.8%5.1%9.4%4.6%
walk back4.2%6.0%2.6%1.2%7.4%10.3%5.3%6.6%
jog back1.0%2.4%2.0%1.4%5.3%5.6%5.1%6.9%2.8%
strafe left2.0%6.9%8.8%4.5%5.0%6.2%4.3%4.6%
strafe right3.8%3.4%7.5%4.8%5.9%7.5%4.2%5.3%
crouch walk3.5%2.5%1.5%1.4%2.6%3.9%3.3%18.2%11.5%
deepest shoe through the floor0.2 cm0.3 cm0.5 cm0.0 cm0.5 cm0.5 cm0.3 cm0.7 cm0.5 cm
Dead ends, and the problems open before this roundthe table of what was tried and dropped, now with this round's
Dead ends & open problems

What did not work (until 25 September)

IdeaMeasuredStatus
Lift the video's joints to 3D directly4× worse on moves the library has (0.022 → 0.088)dropped
Depth from the best of several captures0.186 vs 0.178 torso lengthsdropped
A three-quarter camera in the video promptno better than front or side in 40 testsnot adopted
Let ComfyUI apply the turbo LoRA20 min of swapping, no load; 4.5% requantizing errorreplaced
Fade the upper-arm weight off clothing on the body's side of the shoulderno visible change - the vest panels ride the collarbonereverted
Fit the video with the capture's bone lengthsbent short torsos into leans - Aoi 28° backreplaced
Calibrate the arms on a T-pose renderthe pose model took Aoi's cuffs for her wristsdropped
Tie Pip's vest to the chest, found by its colourfound the vest exactly; tore the fused surface into streaks where it meets the sleevedropped
Skip foot planting on clips from videoworse: Pip 0.114 → 0.128 jointsdropped
DWPose, a whole-body pose model, for the top of the kickfolds the leg too: ankle confidence 0.16-0.21 at the apex, ViTPose 0.58-0.68not adopted
Fill unsure joints from the parent's motion; weight the fit's smoothing by confidenceclose to neutral: kick 0.083 → 0.079, the rest within 0.001kept
Let the hands swing the arm to point themhands 0.64 → 0.24 on Aoi, but silhouette −0.005 and her hands at her chinreplaced
Move all 21 hand points to where DWPose sees them on the rigworse where the offsets were small: Aoi 0.24 → 0.29, Mara's punch 0.20 → 0.22wrist only
Turn the wrists and fingers to DWPose's handspalms facing the video's way 51-99% → 95%-99%kept
Refine the pose against the video's silhouette+0.07-0.10 IoU on every movekept
Knee hinge rule0.178 → 0.177; blocks a visible faultkept
Let a vest's side go below the armpit wherever its colour says vestjoins between arm and vest stretched into sheets up to the raised elbow - worse than the flapdropped
Colour an arm's unseen surface with the arm's colour at its heightno visible change once the folded UV islands were found and repairedremoved
Mask where the source images show the old generated hands12,513 texels changed; Aoi's trouser streaks stayed - the side view's hands were the causenot wired in
Check the arm cut in slabs of the nearest sectiontorso voxels went to the armpit's sections; Vex's joins went unseenreplaced
Cut more directions where a join survivesthe ring ran inside a bulging sleeve (Vex, right arm at 0.41) - nothing changed; the ring moves outward insteadreplaced
Repair folded UV islands on the shipped mesh, down to an island a faceatlas coverage 0.58 → 0.42 - a third of the texture wasteddropped
Tell a jacket from the trousers by the human parser's classes where their colours are too closethe parser gave the lower back of Mara's, Juno's and Rowan's jackets to their trousers; the hem rule split each jacket across the seat, and a crouch showed a pale banddropped
Fill the arm cut's new surfaces from the nearest texel facing the same waythrough the air, the back of a sleeve and the back of the vest beside it both face backwardsreplaced
Pip with both arms overhead, front and side: the yellow vest's side panels lifted into wings

Was open: Pip's vest lifts into wings (fixed 25 September - see Polish)

A move from video found what the 18 standard clips never reach: with his arms overhead, Pip's puffy vest rises with his collarbones. The vest and his hoodie sleeve are one fused surface, both labelled clothing. Told apart by colour, the vest is found exactly - but tied to the chest it tears where it meets the sleeve, and faded in from its edge the wings stay. It needs to be its own layer, generated or cut that way.

Also open then

Poses the pose model rarely saw (the top of a high kick - DWPose misses it too); hands DWPose cannot be sure of - blurred by a fast move, under hair, overhead at the frame's edge (Pip's); joints that leave the frame; H3's licence for anything public; a licence-clean video model (Wan 2.2, Apache-2.0) as the alternative.

Where the error in moves from video came fromthe first measurement of the old pipeline, stage by stage - replaced by the fit on the video's own body and the refine stage

Where the error came from

The same score at each stage of the old pipeline (joints, torso lengths). Fitting the capture to the video and writing it as an FBX lost nothing; moving it onto the character's own rig doubled or tripled the error.

movefitFBXcharacter's rigrender
Mara · punch0.0410.0420.0720.080
Mara · kick0.0350.0370.0990.178
Aoi · spell0.0700.0710.1960.219
Pip · cheer0.0630.0640.1200.129
Bending the nearest capture toward the video: how the fit was builtthe self-test, the steps that made it, and solving it into bones - the fit itself is still a step
The fit, measured · library clips as ground truth

Bending the nearest capture toward the video

The test: take a library clip, show it to the matcher from a random camera angle with pose-model noise, a 3 s stretch and a speed change, and measure the recovered 3D pose against the true one, in torso lengths. Run once with the clip in the library, and once with it and its near-twins taken out - a move the library does not have. The way the body faces is not counted: a game's controller turns it.

Pose error, 40 tests each

Lower is better. The fit only runs where the capture misses the video by more than the pose model's jitter.

capture as matchedfitted to the video
0 0.1 0.2 0.3 0.048 0.037 0.269 0.178 move in the library move not in the library

Of the remaining 0.178, depth is 0.143 - the part one camera cannot see, still borrowed from the capture.

Does the camera angle matter?

The 40 moves not in the library, split by where the camera stood. A three-quarter view was expected to help; it did not.

capture as matchedfitted
0 0.1 0.2 0.3 0.261 0.163 0.271 0.184 0.271 0.176 front / back · 7 three-quarter · 21 side · 12

So the video prompt keeps its near-front camera, which shows both arms most clearly.

How the fit got there

ChangeWhyIn libraryNot in libraryStatus
Capture alonethe retrieval the pipeline had0.0220.234baseline
Take each bone's image from the video, depth from its lengththe obvious fit0.088—4× worse
Keep the capture where it agrees within the pose model's noisedepth = √(L² − l²) swings with noise near the image plane0.0550.165kept
Blend whole bones, not depths recomputed from a blendnoise alone moved flat bones0.0450.157kept
Smooth only the correction, not the capturea jab lasts three frames at 15 fps0.0360.156kept
Gate: fit only if the capture misses by > 1.2× jittermatched clips leave 0.66–0.84 jitters, novel ones 1.5–60.037*0.178*kept
Depth from the best of the top four capturesagreeing in the image ≠ agreeing in depth—0.186*dropped
Knees only bend one wayguards against visible backward knees—0.177*kept, neutral

Torso lengths, mean of 12 tests (seed 5); * 40 tests (seed 21), whose capture-alone baseline is 0.048 / 0.269.

Solving the fit into bones

The fit gives 13 points per frame; Blender turns the capture's bones to reach them, then exports and re-imports the FBX to measure. The kick's 30° lean showed the spine was solved wrong. Worst arm error per move, cm:

Spine solvePunchKickSpellCheerVerdict
One bone aimed along the hips-to-shoulders line1.815.651.510.75pivots differ: a lean slides the shoulders
The same bone aimed from its own head1.291.882.74—fixes the kick, hurts the spell
Three spine bones bring the shoulders' midpoint home1.812.433.910.70midpoint right, twist wrong
Three bones fitted to both shoulders (Kabsch)0.453.491.570.39best arms - but the head drifts 17 cm
…with the head at half weight0.472.771.450.28chosen: 12 fitted joints within 0.2–1.7 cm on average
Does a turntable video make the model more accurate?an experiment: not worth 30 minutes of H3 per character
More angles for the 3D model · a 360° video as the test

Does a turntable video make the model more accurate?

The reference shows only the front, so the model's sides and back had no measure. H3 was asked for one more move - turn slowly through a full circle - and turned Mara cleanly and consistently from her single reference image. Each frame's figure is held against a model's silhouette from the angle that fits it best.

Twelve frames of the generated turntable: Mara turning through a full circle, front, side, back, side and front again

Silhouette over the turn

97 frames none of the models was built from; IoU, higher is better.

built frommeanfrontsidesback
the reference alone0.7780.8050.711 / 0.7760.780
+ repainted side and back (now)0.7810.8020.731 / 0.7780.771
+ three turntable frames0.7860.8180.724 / 0.7680.773

More views barely move the shape. What limits the score is pose: she steps through the turn while the models stand in the reference's A-pose.

Verdict

Not worth ~30 minutes of H3 per character in the standard build. The turntable frames do pull the unseen sides' colours toward the video's, but where the views disagree the model averages them: Mara's braid hangs over her shoulder in the reference and down her back in H3's back view, and the model built from both lost it at the back.

Kept as a check: charforge.py move --move turntable and tools/turntable_fidelity.py show any model from every side against the video.

Mara from behind and three-quarter behind: the video, then the model from the reference alone, with repainted views, and with turntable frames
From behind, two angles the models were not built from. Left to right: the video, the reference alone, with the repainted views, with turntable frames.
Krea 2 against Qwen-Image 2.1the comparison that made Qwen-Image 2.1 the default image model
Image models · same prompt, same seed, 832 × 1216

Krea 2 against Qwen-Image 2.1

Each character's reference drawn by both models from the pipeline's own prompt. Qwen's images are shown on a checkerboard: the background is its own alpha, not a colour cut out afterwards.

Krea 2Qwen 2.1
Broken figures1 of 6 (Rowan)0 of 6
Whole figure in frame5 of 66 of 6
Backgroundflat grey, removed by rembgalpha, 1.2–1.7% soft edge
Hands from the hips (torso lengths)0.61–0.720.28–0.37 → 0.58–0.70
Time per image (M5)3.9–5.1 min · 8 steps5.9–6.6 min · 30 steps
LicenceKrea's termsresearch only

The one thing Krea did better

Qwen reads "arms slightly away from the body" literally, and hands that close to the thighs can fuse to them in the 3D model. The prompt now spells out the angle.

Mara by Qwen with the first prompt: hands near the thighs
first prompt · hands 0.30
Mara by Qwen with the angle spelled out: arms well clear
angle spelled out · hands 0.70

Mara 0.30 → 0.70, Pip 0.37 → 0.58 torso lengths (Krea: 0.63, 0.61), edges still clean. With that, Qwen-Image 2.1 is the pipeline's default; --image-model krea2 keeps Krea for commercial work.

Mara

realistic
Mara drawn by Krea 2
Krea 2 · 250 s · hands 0.63
Mara drawn by Qwen-Image 2.1, transparent background
Qwen 2.1 · 397 s · hands 0.29 · edge 1.4%

Juno

realistic
Juno drawn by Krea 2
Krea 2 · 232 s · hands 0.67
Juno drawn by Qwen-Image 2.1, transparent background
Qwen 2.1 · 355 s · hands 0.28 · edge 1.3%

Rowan

Krea: no head or legs
Rowan drawn by Krea 2
Krea 2 · 259 s · hands 0.66
Rowan drawn by Qwen-Image 2.1, transparent background
Qwen 2.1 · 355 s · hands 0.30 · edge 1.2%

Wren

Krea: figure touches the edge
Wren drawn by Krea 2
Krea 2 · 289 s · hands 0.72
Wren drawn by Qwen-Image 2.1, transparent background
Qwen 2.1 · 356 s · hands 0.34 · edge 1.3%

Aoi

anime
Aoi drawn by Krea 2
Krea 2 · 304 s · hands 0.69
Aoi drawn by Qwen-Image 2.1, transparent background
Qwen 2.1 · 358 s · hands 0.33 · edge 1.6%

Pip

stylised
Pip drawn by Krea 2
Krea 2 · 274 s · hands 0.61
Pip drawn by Qwen-Image 2.1, transparent background
Qwen 2.1 · 365 s · hands 0.37 · edge 1.7%

The automatic checks passed Krea's Rowan - a jacket with no head or legs still has "arms" to a pose model. Looking at the pictures is still part of the check.