CRASH LAB
A research program of CRASH LAB · Bolas

PAELEO

Promptable Augmented Environments LabEkphrastic Operations

From describing worlds to operating within them.

One research program at three scales of operation. AEL is the place, the laboratory and its lineage. PEO is the method: Prompt, Ekphrasis, Operation, how descriptions become consequential actions. PAELEO is the language and studio where the method operates across environments, archives, games, films and agents.

Six letters, read two ways
P
Promptable
A
Augmented
E
Environments
L
Lab / Language
E
Ekphrastic
O
Operations

The L is a deliberate double reading. As a studio, PAELEO is a Lab. As an artifact, it is a Language: Promptable Augmented Environments Language for Ekphrastic Operations. AEL always keeps its official expansion, Augmented Environments Lab.

It sounds like paleo: sediment, earlier media, and futures buried in the past.

Three scales

Not three competing acronyms. One intellectual and technical system, nested like crates.

AELthe place · institution

Augmented Environments Lab

Where situated worlds are investigated and made. The laboratory and its historical lineage.

PAELEOthe language and the studio · research artifact

Promptable Augmented Environments Lab / Language for Ekphrastic Operations

The language and working environment in which descriptions can be authored, executed, contested and evaluated. PAELEO Studio is its practice: film, games, XR, archives, animation and multi-agent production.

PEOthe method · theory and authoring grammar

Prompt · Ekphrasis · Operation

How descriptions become consequential actions. Its object is not simply a prompt, but a situated operation with observable consequences.

PPromptintent
EEkphrasisdescription
OOperationconsequence

↻ recursive: the consequence changes the situation from which the next prompt is made.

Operationchanges a state of affairs. A single person can perform one. This is the primary research concept: operative ekphrasis is the phenomenon, ekphrastic operations the umbrella.
Orchestrationarranges participants, agents, tools and sequences. It is one way of coordinating many operations: an important technique inside the program, not its name.

Imagetext · environment text · worldtext

The program's lineage, and the first thing its first film shows.

1 · representation

Imagetext

text beside a picture · the pageA picture of the porch on a page, with the caption: A dog at a door.

Text and image remediate each other on a flat plane: the page, the screen, the early interface. The Bolter and Mitchell paradigm.

2 · situatedness

Environment text

text pinned to a place · AELThe caption has become paper tags pinned to the door, the person and the dog.

The imagetext is lifted off the screen and anchored to coordinates, markers and space. The AEL era: DART, Argon, locative media.

3 · execution

Worldtext

text the place runs on · PAELEOThe dog's tag lists what it may do, including refuse, and the dog has risen to watch.

The environment text gains rules and agency. The text does not just sit in a space; it generates and governs it, as a move in a language game.

Why not videotext: a video implies a fixed timeline meant to be watched. A world is inhabited, queried and simulated. Videotext is a closed loop, a broadcast; environment text is an open system, a space; worldtext is a playable language game.

The boxes

Each box is one film made by one SCION swarm, with its case: the question, the trick, the recipe, something to try, what broke. The first box is open.

PAE-00Oct 24 · 1 min 22 s · one swarm: SCION 48, State Machines

The threshold

Can a description make a character that chooses, so that the person who prompts it can understand it but not command it?

Commands push the dog away. Changed circumstances bring it to the sill. Told to come in, it stays where it chose.
The trick

Nobody animates the dog. The prompts change the world (posture, place, door, voice, hand); a five-state machine reads that world through what it perceives, trusts and remembers, and the paper puppet acts out whatever it chose.

states 5 · watch, approach, retreat, lie down, refuseprompts 7, typed on screenchoices the dog made 7tests 2,009,021 checks passedkeyframes on the dog 0
Try it

Earn the approach

You are the operator. You cannot move the dog. You can only change the circumstances, the way the prompts do in the film. It is the same dog: the same machine and the same paper, running in your browser.

The person
Where
The door
Voice
Hand
Start over
the dog is watchingtrust 0.20threat 0pressure 0


  

Recipe

SCION 48 · State Machinesmachine.jsstage.jsChromium canvasnumpy soundindependent critic
  1. Write the dog first as a machine: five states, guards in priority order, a minimum window in each state, and three things it carries between frames: trust, pressure from repeated calls, and how many times it has refused.
  2. Test the machine before drawing anything: 400 random operators, every state reachable, no state it cannot be coaxed out of, never faster than its legs, and commands alone never earn an approach.
  3. Cut the cast from paper in code: limbs as tapered pieces with a fibre texture fixed to each piece, brass brads at the joints, hand-cut edges that stay the same from frame to frame, a cast shadow.
  4. Let springs carry the machine's choice into the pose (ears, tail, crouch, lie, head, turn), so every change takes time; tie the walk to the distance actually travelled.
  5. Score the film as prompts: each one is typed on screen (P), the paper tags pinned to the door, person and dog rewrite themselves (E), and the dog acts (O).
  6. Open with the lineage: a caption on a page is snipped out, carried into the place and pinned there, then becomes the rule the dog runs on.
  7. Synthesize the sound from the render's own cues: keys, a wordless voice with the contour of the words, paws that grow louder as the dog comes near, the door, crickets, a pad under the titles.
  8. Hand each cut to a critic who has not seen the notes; repair the causes it finds; hand the new cut to a fresh critic.

The sheets

The first prompt: the person stands in the doorway and calls. The dog backs away, crouched, ears flat.
Called again and reached for: the dog turns its back and lies down facing away.
Step back, sit down beside the door, say nothing. The dog keeps its back turned.
The door opens. With nothing pressing on it and the doorway lit, the dog turns back to look.

What broke and what changed

v1: told "come in", the dog did nothing you could see. Its choice to stay was told by the closing text, and it lay parallel to the door, not in it.
→
v2: a dog lying down notices any voice: head and ears come up to it, the tail goes, then the chin goes down on its paws. The sill moved into the doorway and the jamb is drawn over its back half, so it lies through the door.
v1: at 33.5 s the dog flipped edge-on while still lying, then stood up, on the same frame as a camera cut.
→
v2: a lying dog gets up before it turns; the turn is slower; the cut moved before the door opens.
The new sill position left a dog standing right at the sill that could never lie down: an absorbing state the recovery test found in 103 of 6,000 snapshots.
→
A guard for it: trusted, unthreatened and already at the sill, the dog lies down there.
v1: the caption vanished under the growing picture and reappeared as a second copy beside its tag.
→
v2: the caption is snipped out as a paper slip and carried in one move to the door, where it becomes the tag.
v1: the dog lay down after the door opened just as the next prompt was being typed.
→
v2: the next prompts wait a second longer.
v2: told "come in", the dog only lifted its head outward, put it down, and raised it again before the end. Refusal could not be told from not hearing.
→
v3: it turns its head in through the lit doorway with its tail sweeping, turns back out, lays its chin on its paws and keeps it there. The sill moved so a third of it lies outside the jamb.
v2: two reframes of about 60 px inside a shot read as jump cuts; the person's sit popped from a squat to a chair-sit in mid-air.
→
v3: the reframes are camera moves; sitting is two phases, down then legs out, with the hands blended to the knees.
v2: a 2 Hz thump under the climax matched no visible tail; near-silence under the lineage.
→
v3: thumps only while the tail really sweeps; a low pad and a paper flick for each pinned tag.
v3: chin down, the neck stood up behind the head like a plank with the collar on it, for the last 5 s of the scene.
→
v4: as the chin goes down the neck base comes forward and down to the chest, so the neck lies along the floor.
v3: only a fifth of the dog lay outside the jamb, so it read as in the doorway, not across the sill. And the jamb stopped being drawn over it, because its condition still used the old sill.
→
v4: the sill moved again, so a third lies outside; the jamb is drawn for any dog within 0.9 m of the threshold.
Moving the sill exposed a speed jump: the dog arrived at 0.12 m/s and stopped dead. The continuity test caught it.
→
The approach slows to 0.06 m/s at the sill.
Carries forwardThe operator, the machine and the critic loop. The next box opens another SCION swarm on the same porch, or takes this dog somewhere else.
PAE-01Oct 24 · 1 min 35 s · all 54 SCION lenses

SCION through the dog

Before a swarm makes its own film, what does it see first in this one?

Fifty-four lenses on one porch: shadow theatre, Sketchpad constraints, GrabCut masks, a halftone print, a provenance graph. The dog underneath is the same machine every time.
The trick

The stage records every paper piece and joint as it draws them. Each swarm gets a lens that reads that record and draws its own formal language over the same running scene: meshes, bones, mattes, weights, state graphs, archive frames.

lenses 54, one per swarmloop 32 s of prompts, sharedlive in the browser, any lens
FINALENov 2026 · a film by Watson Hartsoe · built from PAE-02

It’s All Right

come here · it’s all right · come in

The film

A person speaks to a dog in three sentences. The order fails, the reassurance turns its head, the invitation is answered from the sill. One scene, written once as a score, told at once in every language PAE-02 made it in: cut paper, bones, archive film, print, code.

IT’S ALL RIGHT: every telling at once
PAE-02Oct 24 · 54 swarms · 38 s each · 52 passed by an independent critic

The threshold, handmade

Hold the description and the dog's choices still. Let each swarm build everything else from the bottom up. Do the choices survive, and is each world its own?

The trick

A lens restyles one drawing; PAE-02 asks each swarm to make the whole scene itself, from found footage, silhouettes, constraints, doodles or a torrent of archive tiles, under one score of the dog's choices. The score is checked against every film. Everything else differs.

SCION 01Cut-out collageTerry GilliamEach choice is a substitution, so the dog's decision is also the film's cut.SCION 02Silhouette theatreLotte ReinigerNo colour on any figure: the whole relationship has to read by outline and the gap between shapes.SCION 03Surreal collage grammarMax ErnstAuthority becomes architecture, then the dog's own fur.SCION 04Image torrentFrank MourisThe voice is made of pictures too, and it hits the dog as a ring of alarm clocks.SCION 05Atmospheric multiplaneYuri NorsteinDepth, fog and focus decide what the dog can see, and what we can.SCION 06Harmonic motionJohn WhitneyTrust is a fifth: the dog comes when its breath locks to the house.SCION 07Drawn metamorphosisPeter FoldèsThe door's outline is lifted off the door and laid on the porch as the light.SCION 08Constraint SketchpadIvan SutherlandTHE DOG IS NOT SELECTABLE: the brief's 'understand it, not control it', said in Sketchpad's own terms.SCION 09As-rigid-as-possible puppetryTakeo IgarashiA drawing that bends without ever folding: the dog keeps its shape by solving for it.SCION 10One sheet warped by handlesMLS deformationThere is no skeleton anywhere: the dog bends as one sheet, the way a hide would if handles pulled it.SCION 11Footage cut-outs moved as puppetsVideo puppetryEvery pixel of the dog is the real dog; only the way its pieces are held changes.SCION 12Bounded biharmonic weights on a lace meshLocalized weightsLegs a few millimetres apart on a sheet full of holes move independently because the weights stay local across every gap.SCION 13Paper figures with secondary dynamicsSecondary motionThe secondary motion carries the story: the coat sleeks and stiffens on the refusal, the ears flick to the soft voice, and the coat spreads and settles as the dog lies down.SCION 14Nora Willett lineagePose2Pose castingThe performance is a path through a drawer of drawings: casting, not animating.SCION 15A layered puppet fitted to the archive dogLearned layered puppetThe end card shows what the model makes alone next to what had to be corrected by hand.SCION 16Cast heads on a cut-out bodyJibJabThe performance belongs to the body; the face is a slot anyone could be cast into.SCION 17Restricted graphic grammarSouth Park lineageThe fewer drawings it has, the more each choice of drawing says.SCION 18Painterly puppet disciplineCartoon Saloon lineageDiscipline is what is not rigged: the drawing does the acting, the rig only carries it.SCION 19Smart-Bone rig with switch layersMoho Smart BonesThe turns are not tweened: a view bone switches whole drawings while corrective shapes re-draw the joints it bends.SCION 20Production rig: drawing substitution and node graphToon Boom HarmonyEvery turn is a substitution that lands on a whole drawing, never a dissolve; the end card shows the real xsheet rows of the head-lead turn.SCION 21Photographic actor rigged from stillsPhoto-to-ActorA dog that only ever walked to a bowl is retargeted, from a handful of stills, to perform all six choices of the score.SCION 22Performance puppet with behavioursCharacter AnimatorThe puppet is never animated frame by frame: it is performed, and the performance is a file of takes.SCION 23Skeletal runtime with mixed action clipsSpine runtimeThe head leads because it plays on its own track over the body; the end card draws the real track timeline.SCION 24Portrait volume illusionLive2D lineageGaze, head and body are three separate decisions, and the dog makes them in that order.SCION 25Interactive puppet state machinesRiveThe score is played into the puppet as inputs; the dog's choices come out of its own state machine.SCION 26GPU scene graph of archive fragmentsPixiJS archive stageHundreds of small fragments, texture-batched, are held together into one animal by masks and render-texture passes.SCION 27Declarative film compilerRemotionThe film is a pure function of its description: render it twice and every frame is the same.SCION 28Compositor's film of archive platesMask and plate foundryPlates from different films are pinned, held out and lit until they share one shot, and the dog performs by animating its masks.SCION 29Cutout extractionGrabCutThe cut is the film: every edge of the dog is an argument about where it ends.SCION 30Evidence-bounded mask proposalsSAM 2 proposalsNothing is drawn: when the archive has no evidence of a pose, the film stages around it instead of inventing it.SCION 31Soft-edged matted archive framesAlpha MattingThe dog's edges hold against sky and siding, because the matte, not the outline, decides what is dog.SCION 32Jointed toy moved only by forward kinematicsForward kinematicsThe joint graph you see is the one that moves it: change a hip and exactly its children follow, saddle cloth and all.SCION 33Joints, chains and planted effectorsInverse kinematicsA command is a target the arm cannot reach; the threshold is a target the dog's own chain lays down across.SCION 34Engraved mesh, shape kept by a local–global solveAs-rigid-as-possibleThe dog bends where its triangles can keep their shape and nowhere else.SCION 35Risograph print bent by moving least squaresMLS gesture warpThe warp is honest about itself: wherever the print is bent, its dot screen visibly stretches.SCION 36Auto-rigged archival silhouettesBBW auto-rigNo one paints a weight: the found silhouette rigs itself.SCION 37One skin over a skeleton, weighted per vertexWeighted skinningThe joints keep their girth because of how the skin is blended, and the film shows the version that does not.SCION 38Drawing libraryDynamic pose selectionThe dog that turns its back is the same drawing as the dog that comes.SCION 39Turn labMultiview substitutionIt solves the turns, the place every earlier swarm failed, by never pretending the dog is a solid.SCION 40Paper performance, captured and replayedTabletop captureThe acting exists only as data recovered from fingers moving paper, and the film replays it.SCION 41Inertia and restraintSecondary dynamicsNothing secondary is animated by hand, so stillness has to be earned: the holds are quiet because the energy has gone.SCION 42Parts and joints discovered from a few framesNeural layered puppetNobody drew where the dog's joints are: the fit found them, and the seams stay on screen.SCION 43Typed fragments and rewrite rulesCollage Graph RewritingEach change to the dog is a named rule, so the film can say what it did and why.SCION 44Spatial collage in depth, with a moving cameraMultiplane parallaxThe room behind the door is a real deeper plane, so looking in has somewhere to go.SCION 45Who is in front?Occlusion compositingHalf in, half out is a fact about layers: the door frame is in front of one half of the dog and behind the other.SCION 46View-matched plates on landmarksFace Landmark RegistrationThe dog turns through depth by swapping views, never by mirroring a side.SCION 47Viseme and gestureSound-to-actionThe dog answers sounds, not lines: each choice is a measured lag after a vowel.SCION 48A dog whose refusal is its own decisionState machinesNobody keyed the refusal: the dog decides it, and the board clacks when it does.SCION 49Museum open-access objects, cut and recomposedOpen-content atelierNo line is drawn: every figure is a museum object, recomposed as an actor, with its object number on the card.SCION 50Library of Congress lineageHistorical print stageEvery frame says whether the dog in it was photographed or invented.SCION 51Montage on a measured cutting scoreMouris montage researchThe rhythm of the cuts carries the drama while the dog stays one animal.SCION 52Pinned found prints and substitutionGilliam EphemeraThe collage's logic is the dog's own: each piece moves on a pin, and the absurd beat is a hand from nowhere.SCION 53Civic footage performerAtlanta Moving ImageThe scene says what it is the whole time: COMPOSED SCENE · archive fragments re-performed · not a recorded event.SCION 54Archive-to-puppet lineageProvenance graphThe film can answer, for any pixel, where it came from and what was done to it.
PAE-03Oct–Nov 2026 · one sack · six swarms · ten choices

The Flour Tower, handmade

Hold the story and the sack's ten choices still. Let each swarm build the sack, the tower and the knife from the bottom up in its own language. Do the choices survive, and is each sack its own?

The trick

Disney's flour-sack test with a decision in it: a knife cuts a tower of flour with a penny on top, and the sack holds the tower up, catches the penny and gathers the flour into itself. The same score of ten choices is checked against every film.

The Flour Tower, handmade
PAE-04Nov 2026 · after Italo Calvino · 12 deeds, 4 flour sacks, 1212 films

The Burning of the Abominable House

Four tenants, twelve deeds, an index with the page numbers burned away. Who did what to whom?

The trick

Every deed is animated by a different SCION operation and can be played between any two tenants, so the bank can play any of the 8,916,100,448,256 stories. Each deed leaves a mark and a memory; the tenants can choose with beliefs, plans and memories; and the consistent stories are counted exactly.

Inigo, a patched flour sack, in a dark room with Ogiva's portrait on the wall

The swarm queue

Fifty-four SCION animation swarms, each a standalone prompt studio with its own proof scene and research loop. PAE-00 was made by number 48; every one is shown through the dog in PAE-01. The operator, written up · open the operator.

The names record

Which terms do which work, and where they collide.

CandidateWhat it emphasizesAssessment
Ekphrastic Operationsdescriptions that act upon worldsbest umbrella
Ekphrastic Orchestrationcoordinating agents, tools, media and performancesexcellent subsystem
Operative Ekphrasisthe foundational theoretical conceptbest dissertation term
Prompt Orchestrationcoordinating prompts and model workflowsuseful technique, too narrow for the thesis
Ekphrastic Ontologiesdescriptions constituting objects and worldsuseful research branch

Collisions

  • POML already names Microsoft's Prompt Orchestration Markup Language. A reference or substrate, not the identity of this contribution.
  • PEO is the Person–Environment–Occupation model in occupational therapy. A collision, and also a precedent for a situated, relational understanding of action.
  • The Georgia Tech name is Augmented Environments Lab, singular words exactly as they are. PAELEO never renames the lab.

Formal and public

  • For the dissertation: Ekphrasis as Operation, the theoretical subtitle.
  • For the studio: Ekphrastic Operations. Shorter, more active, broad enough for film, games, archives, XR and agentic performance.
  • Promptable names an affordance of the environment. A place, a character, a film or an archive can become promptable and keep its own constraints and agency.
institutionAELthe laboratory and its lineage
theory + methodPEOPrompt · Ekphrasis · Operation
research artifactPAELEOthe language and authoring environment implementing PEO
creative practicePAELEO Studiofilm, games, XR, archives, animation, multi-agent production

The argument in one breath: operative ekphrasis is the phenomenon investigated. PEO is the theoretical and experimental model. PAELEO is the language and working environment in which descriptions are authored, executed, contested and evaluated. AEL is the institutional lineage from which it develops: a laboratory that develops a language for operating within environments.