A home system, built for a hackathon

The home system

Lanternina divides execution between a home hub and a cloud API. The hub coordinates displays, printing and returned sheets, and manages activity timing. The cloud API calls models on Microsoft Foundry to generate activities and pages, interpret returned sheets and prepare continuations.

The hub can sit near the router. The displays communicate with it over Wi-Fi; the printer and scanner are also reached over the network.

The displays run on batteries and can be placed where they are useful; more can be added. Displays and portable devices are designed for very low power consumption, aiming for weeks of availability between charges. They spend much of their time asleep and wake to communicate; e-paper retains its image while asleep. Battery life depends on update frequency and use. The hub and printer use a fixed power supply.

The camera and a dragon model beside a display inviting the player to turn three objects in the room into places on a map with a secret passage.

How an activity goes

The game develops through several exchanges until it closes. Lanternina speaks through printed pages and display messages. The adolescent replies by writing or drawing on the sheets and photographing what they made. The returned material shapes the next response, which can bring another sheet.

The pace is a material correspondence. A reply takes time while the photograph is read, a page is prepared and a display refreshes. Instructions remain on paper during the wait. Prompts request a default ceiling of 5 sheets for the whole game, distributed across its exchanges and including any printed closing response. The parent can choose a lower ceiling; the game need not use every sheet.

The parent steers the proposal. They write interests, subjects to avoid, language and current circumstances. They can prepare a brief and set the available hours.
The cloud API requests an activity from a model. It sends household preferences and a selected activity method. The model returns a graph of steps, called moments, with duration options, help and paths to the ending.
The API checks and stores the activity. It validates the response format and activity structure, and applies Content Safety to the text. Failed checks can trigger a repair attempt. The parent reviews the accepted activity and can approve or refuse it.
The hub collects the activity. It polls the device API on its own schedule. The API returns the waiting activity in that HTTPS response, and the hub stores it locally before preparing the first page.
The hub requests a page from the cloud API and prints it. The image model receives the content and drawing instructions for that page. The API returns the generated image; the hub sends it to the printer over IPP.
The camera or scanner returns the completed page. A vision model receives the photograph or scan and the original page image. It compares them and returns a reading of the marks that were added.
The hub selects the next moment from the reading. It follows the corresponding transition in the activity graph and prepares the next message or page. Further replies on paper and photographs continue the game until its ending. At its first timer check within thirty minutes of the end time set by the parent, it starts the current moment's path to the ending.

Returning the sheet with the camera

The adolescent photographs the completed sheet to return it to the activity. The camera uses a XIAO ESP32S3 Sense in a 3D-printed enclosure, with a shutter button, a feedback light and a battery. The prototype produces JPEGs at 1600 × 1200 pixels.

The assembled white camera, with its lens on the front and a green shutter button and yellow feedback light on top. — The assembled prototype, with its shutter button and feedback light.

A press starts acquisition. The light stays on until the selected frame is ready; saving and delivery continue after it goes out. The firmware writes the JPEG to local flash and reads it back before starting Wi-Fi. It sends the image to the home hub over authenticated HTTPS. Two flashes confirm a matching receipt after the hub has stored it. A failed delivery keeps the photograph and its identifier for a later attempt.

For a printed sheet, the reader receives the photograph and the original page kept for printing. It compares them despite changes in perspective and lighting, transcribes legible additions and reports uncertainty. An uncertain reading or ambiguous activity association leaves the activity waiting. The hub chooses one compatible camera or scanner from the parent's assignments and keeps that choice for the return. The family can view, download and delete photographs in the parent's panel.

A construction or arrangement can instead be photographed without a printed original. The model describes relevant visible material. An invited adolescent can also send photographs through the portal; the cloud stores them until the hub retrieves them. Only a photograph associated with a compatible waiting moment enters the activity's reading path.

The open camera enclosure showing the controller board, antenna, wiring, shutter and battery beside it. — The open enclosure shows the board, wiring and battery. The photograph documents the prototype, not a wiring diagram.

Prototype trials verified battery-powered capture, light feedback, sleep, wake and resumed delivery after a receiver interruption. The implementation page describes the checks and remaining limits.

The parent's panel

The parent opens the panel in a browser. Waiting activities show their title, duration, planned sheets and a description of what will happen. The parent can read the script, approve, refuse or work on the idea. Approval makes the activity available to the house without starting it immediately.

The panel showing the Valle Lunga railway activity, its expandable script and the Approve, Refuse and Work on it controls. — Activities awaiting approval, with invented data. The sample activity is in Italian.

Settings cover days and hours, how many activities may begin, interests, subjects to avoid and language. The parent can add a temporary note and set bounds on how a continuation may change. During an activity, they can bring its end time forward or ask it to start closing. The activity log collects what the system generated and showed.

Devices in the house

The panel lists registered displays and cameras, and discovered printers and scanners. The parent names devices, assigns jobs and sees which need attention. Cameras and scanners offer one return function, "shows Lanternina". Several compatible devices can hold it; the hub selects one for each return. Displays can hold several display jobs.

The parent sets each display's connection interval. If all intervals exceed 5 minutes, the panel warns that at least one display assigned to activity instructions should connect every 5 minutes or less, so replies do not take too long. A shorter interval reduces the wait for an update and requires more battery-powered wakeups. The interval does not guarantee the total response time, which also includes processing and printing.

The panel's Devices page with editable names, display status and checkboxes for assigning jobs. — Device names and jobs, with invented data.

Reconstructing a railway

An old railway connected a hut to a cutting, passing between rows of trees and over a bridge. The map preserves these observations but leaves the route beside the pond uncertain. The activity asks for a possible route in the notebook, its direction and an alternative.

The Valle Lunga map: a railway hut, rows of trees, bridge, pond and cutting, with observations written in Italian. — The first sheet contains the map and observations. This example is in Italian.

The second sheet, a notebook filled in by a model with a blue route, north arrow and an alternative hypothesis. — The second sheet is filled in by a simulated hand, not a person. The notebook before it was filled in.

In this synthetic example, the reader recognized the written arrow and direction. The continuation asked for them to be revised after a new indication of the map's orientation.

The system distinguishes a filled sheet from a blank one and uses what it reads to continue. If it cannot read the sheet, it waits. The continuation passes safety checks without a second parent approval.

Memory

The system stores configuration, generated activities, endings, reported durations and reading-derived summaries. A separate call observes scanner page comparisons to calibrate load and writing; photographic reads skip that call. Recent observations help compute the profile that steers prompts. The panel does not expose that profile as a score. The parent can clear activity memory and observations. A complete plain-language memory view is not built.

During development, photographs, returned material, readings and diagnostics may be retained for the authenticated family. The photo archive, generated pages, activity trail and profile observations have separate storage and deletion paths. Original scanner pixels can be archived locally when the scan database is configured; a complete scan-gallery workflow is not built.

An administrator can additionally enable a temporary archive of complete readings for 14 days. Its entries expire 14 days after writing. This is not a 14-day deletion rule for all photographs, traces, summaries or observations. Deleting a photograph does not automatically delete every interpretation derived from it. Retention policy will be reviewed after development.

The stack

In the house

The prototype uses a Raspberry Pi Compute Module 5 on a PoE base board with Raspberry Pi OS, 7.5" displays on XIAO ESP32-S3 with patched TRMNL firmware, and an Epson ET-2870 over IPP and eSCL. In the panel, the parent names devices and assigns display jobs. One display can follow activities, show reminders and hold pictures, or have only some of these jobs.

The prototype hub board in its open enclosure, with a black heatsink and Ethernet port. — The prototype hub with its enclosure open. It stays out of sight on the home network; displays communicate with it over Wi-Fi.

An e-paper panel, an ESP32 board and a battery, before assembly. — What goes into a frame: panel, XIAO ESP32-S3, a 2000 mAh cell, and the ribbon between them.

In the cloud

Static Web Apps serves the site and panel. The API in Container Apps receives requests from the panel and hub, calls models on Microsoft Foundry and applies Content Safety to generated content. Cosmos DB stores configuration and records; Blob Storage stores images. The scan-reading path does not apply inbound-image Content Safety.

The hub also calls the API. The container accesses models and storage; devices in the home do not access them directly.

Boundaries enforced in code

The activity itself provides the reason to continue; the system does not add streaks, daily targets or withheld rewards.
The scheduler responds to approved activities and their deadlines, rather than absence or time-spent metrics.
The review gate makes the profile available only to the models that write and run an activity. User-facing displays, printed sheets and panel pages receive reviewed activity content.
The vision path compares returned sheets or describes photographed objects. It requests visible material and uncertainty, not identity, emotion or attention assessment. Photographs can nevertheless contain people or private information; a prompt does not guarantee that a model will omit every unsupported inference.

Why each boundary exists, and what it costs · every prompt, whole, as the model receives it

Between activities

The displays stay in the rooms between activities. During that time they serve as small framed pictures: instead of an empty screen or an instruction left over from a finished activity, they show an image that belongs on a wall or a shelf. When needed, the same display can show a reminder or accompany a new activity, then return to its picture.

Pictures

The parent assigns the picture job to displays and chooses image themes, hours and how often pictures change. Generated pictures remain available to browse in the panel. A dedicated display can keep showing its picture while another follows an activity.

A mermaid sitting on a rock, drawn in line art on an e-paper display in its white frame. — A mermaid on a rock. Photograph of the picture on the display.

A poodle astronaut on a lunar landscape, shown on the same e-paper display. — A poodle astronaut, on the same display at another time.

Reminders

The parent writes a reminder as a sentence with a time and optional days, such as "On Tuesdays at 5 pm, pack the swimming bag." The house shows it on displays assigned to reminders. A button press dismisses it; otherwise it disappears at the end of its window, normally after thirty minutes. The press is not retained as evidence that the task was done.

An illustrated toothbrush beside an Italian reminder to brush your teeth after lunch, around 1:30 pm. — An illustrated reminder. The Italian text is invented.

A 5 pm reminder in Italian: Pack the swimming bag. — The text-only version, rendered by the display code with an invented Italian sentence.

On a display with several jobs, reminders take precedence over activity instructions, which take precedence over pictures. When the higher-priority content ends, the content underneath becomes visible again.

This site sets no cookies, collects no statistics, and loads nothing from anybody else's server.