*The video below documents the earlier beta version; the system diagram and technical notes on this page reflect the current version exhibited at the CVPR AI Art Gallery 2026.

Project Overview

THE SILHOUETTE SEEKER is an AI powered embodied storytelling experience where players shape a living narrative through hand gestures and hand shadows. Instead of interacting through conventional interfaces, participants communicate with the system using silhouettes formed by their own hands. These gestures function as symbolic acts that invite interpretation rather than fixed commands, allowing human imagination and machine intelligence to collaborate in the creation of story.

‍

‍

‍

Core Concept and Inspiration

The conceptual foundation of the project comes from a simple perceptual phenomenon. A hand shadow is only a rough outline projected onto a surface, yet people immediately recognize animals, characters, or spirits within it. The form itself is incomplete, and meaning emerges only through projection and interpretation. This gap between shape and meaning has historically been a space where imagination operates most actively. Children often invent stories from the simplest objects or shadows, transforming ambiguous forms into living worlds. The project takes this everyday phenomenon as a philosophical starting point and asks how such incomplete signals might become the basis of a new storytelling language in the age of generative artificial intelligence.

Game Mechanics and Interaction

In THE SILHOUETTE SEEKER, shadows are not treated as visual effects but as a ritual language. Certain silhouettes correspond to known creatures preserved in a codex of shadow forms. Others remain ambiguous and must be interpreted by the system. When a player performs a gesture, the AI interprets the silhouette within the current context and transforms it into narrative events, environments, and companions. Because interpretation is probabilistic rather than deterministic, the same gesture may lead to different outcomes in different situations. Each playthrough therefore becomes a unique unfolding of intention, interpretation, and consequence.

Narrative Background and Player Role

The narrative background of the world reflects this idea of unstable meaning. After an undefined disturbance spread across the world, many places lost their balance. Landscapes, communities, and ecological systems remain present but fragmented, as if parts of their stories have been erased. Players enter the world as a Silhouette Seeker, someone who once understood the language of shadows but now remembers it only partially. Through gesture and experimentation, the seeker attempts to restore relationships between places, creatures, and forces that no longer communicate clearly.

The Role of AI and Interaction Dynamics

Artificial intelligence in the system functions simultaneously as storyteller, interpreter, and an unseen spiritual force that responds to human intention. Rather than acting as a passive tool, the AI becomes a participant in the narrative field. It interprets gestures, generates environments, and extends storylines in response to the player’s actions. The relationship between player and system therefore resembles a negotiation rather than control. Meaning emerges in the space between human intuition and machine interpretation. A companion character accompanies the player during the journey. The companion does not control the narrative but acts as witness, interpreter, and recorder. It observes what changes in each encounter, reflects on the consequences of gestures, and helps frame the unfolding events as part of a larger evolving history.

Technical Implementation

The current version runs as a web based installation on an in-gallery monitor, paired with an iPad companion that presents the magical book of shadow forms and flips in sync with the active chapter. Hands are tracked with on-device MediaPipe hand and pose models, and trained silhouettes are recognized by a custom two hand gesture classifier. Invented shapes follow a custom oracle path: a local Qwen vision model running on the installation’s own GPU reads the silhouette’s geometry, and Gemini names the summoned entity and derives its spirit profile, whose silhouette description anchors image generation so the rendered creature echoes the shape the visitor cast. Scenes are generated with FLUX as black and white line illustrations that glow and bloom into color, extended with depth maps for a 2.5D parallax stage. Narration is synthesized locally with the Kokoro voice model, each biome carries a pre generated Lyria music bed that ducks beneath the voice, and a world state engine carries light, balance, trust, disturbance, and resource across chapters, so every cast reshapes the next.

Shared Memory and Data Archiving

Each completed journey can become part of a larger shared memory of the world. When a run ends, it may be transformed into a fog point that appears on an evolving map. A fog point represents a place where a player attempted to intervene in a dilemma and where the world responded with a particular configuration of consequences. These nodes accumulate over time, forming a collective history of interpretations and traces left by different participants.

Future Roadmap

The project’s next iteration moves from the illustrated screen into virtual reality. Wearing a headset, the player becomes a shadow master inside a cave, composing shadows with bare hands on its walls and summoning three dimensional creatures and objects that act within spatial puzzles. Shadow contours condition image generation, which then guides mesh, material, and motion synthesis, while summoning rituals and sigil drawing give generation time an expressive place inside play.

Conclusion

Accordingly, THE SILHOUETTE SEEKER explores how embodied gestures, ambiguous forms, and generative artificial intelligence can produce new modes of storytelling. Instead of presenting a fixed narrative authored by a single creator, the project constructs a system where stories emerge from collaboration between human imagination, machine interpretation, and the evolving memory of a shared world

Technical Workflow & Process:

‍