*The video and system diagram below are provided for reference and are based on the beta version. The latest exhibited version at CVPR AI Art Gallery has since been improved and updated in many ways. [New documentation coming soon.]

Project Overview

THE SILHOUETTE SEEKER is an AI powered embodied storytelling experience where players shape a living narrative through hand gestures and hand shadows. Instead of interacting through conventional interfaces, participants communicate with the system using silhouettes formed by their own hands. These gestures function as symbolic acts that invite interpretation rather than fixed commands, allowing human imagination and machine intelligence to collaborate in the creation of story.

‍

‍

‍

Core Concept and Inspiration

The conceptual foundation of the project comes from a simple perceptual phenomenon. A hand shadow is only a rough outline projected onto a surface, yet people immediately recognize animals, characters, or spirits within it. The form itself is incomplete, and meaning emerges only through projection and interpretation. This gap between shape and meaning has historically been a space where imagination operates most actively. Children often invent stories from the simplest objects or shadows, transforming ambiguous forms into living worlds. The project takes this everyday phenomenon as a philosophical starting point and asks how such incomplete signals might become the basis of a new storytelling language in the age of generative artificial intelligence.

Game Mechanics and Interaction

In THE SILHOUETTE SEEKER, shadows are not treated as visual effects but as a ritual language. Certain silhouettes correspond to known creatures preserved in a codex of shadow forms. Others remain ambiguous and must be interpreted by the system. When a player performs a gesture, the AI interprets the silhouette within the current context and transforms it into narrative events, environments, and companions. Because interpretation is probabilistic rather than deterministic, the same gesture may lead to different outcomes in different situations. Each playthrough therefore becomes a unique unfolding of intention, interpretation, and consequence.

Narrative Background and Player Role

The narrative background of the world reflects this idea of unstable meaning. After an undefined disturbance spread across the world, many places lost their balance. Landscapes, communities, and ecological systems remain present but fragmented, as if parts of their stories have been erased. Players enter the world as a Silhouette Seeker, someone who once understood the language of shadows but now remembers it only partially. Through gesture and experimentation, the seeker attempts to restore relationships between places, creatures, and forces that no longer communicate clearly.

The Role of AI and Interaction Dynamics

Artificial intelligence in the system functions simultaneously as storyteller, interpreter, and an unseen spiritual force that responds to human intention. Rather than acting as a passive tool, the AI becomes a participant in the narrative field. It interprets gestures, generates environments, and extends storylines in response to the player’s actions. The relationship between player and system therefore resembles a negotiation rather than control. Meaning emerges in the space between human intuition and machine interpretation. A companion character accompanies the player during the journey. The companion does not control the narrative but acts as witness, interpreter, and recorder. It observes what changes in each encounter, reflects on the consequences of gestures, and helps frame the unfolding events as part of a larger evolving history.

Technical Implementation

Technically, the current version of the project is implemented as a web based system that combines gesture recognition, generative narrative models, and real time media synthesis. Hand shadows are detected using camera based tracking and machine learning models trained to recognize both known silhouette animals and experimental shapes. A language model generates narrative descriptions, companion dialogue, and encounter structures. Image generation APIs produce stylized black and white line illustrations that function as visual keyframes for the unfolding story, creating a picture book like experience where text, image, and gesture operate together.

Shared Memory and Data Archiving

Each completed journey can become part of a larger shared memory of the world. When a run ends, it may be transformed into a fog point that appears on an evolving map. A fog point represents a place where a player attempted to intervene in a dilemma and where the world responded with a particular configuration of consequences. These nodes accumulate over time, forming a collective history of interpretations and traces left by different participants.

Future Roadmap

Future iterations of the project extend this system beyond static imagery toward animated environments. A local implementation will integrate image to video and text to video generation models to produce short cinematic sequences that respond fluidly to gesture. Combined with spatial audio and real time narrative generation, this approach will allow the storytelling environment to shift from illustrated scenes toward continuously evolving audiovisual worlds while maintaining the same embodied interaction.

Conclusion

Accordingly, THE SILHOUETTE SEEKER explores how embodied gestures, ambiguous forms, and generative artificial intelligence can produce new modes of storytelling. Instead of presenting a fixed narrative authored by a single creator, the project constructs a system where stories emerge from collaboration between human imagination, machine interpretation, and the evolving memory of a shared world

Technical Workflow & Process: