Sematica

The Problem

Approximately two million people in the United States live with aphasia, a language disorder most commonly caused by stroke that impairs the ability to speak, read, write, and understand language. For those with Broca's aphasia specifically, the condition leaves cognition largely intact while severely disrupting speech production. Patients know exactly what they want to say but can only produce fragmented, telegraphic utterances: "doctor... go... um... Wednesday" instead of "I have a doctor's appointment on Wednesday."

Current augmentative and alternative communication (AAC) devices remain frustratingly inadequate for this population. Most rely on grid-based symbol boards or pre-programmed phrase banks that require extensive motor dexterity and cognitive load to navigate. They are slow, impersonal, and fundamentally static, unable to adapt to context, interpret fragmented speech, or support the fluid, spontaneous communication that defines human connection. Patients are forced to conform to the device rather than the device adapting to them.

For TreeHacks 2026, I built Sematica to reimagine AAC as an intelligent, voice-first communication partner. The device takes a patient's fragmented speech, however broken or incomplete, and expands it into three natural, full-sentence interpretations in real time. The patient simply speaks, selects the sentence that matches their intent with a single button press, and the device speaks it aloud on their behalf. No grids. No menus. No pre-programmed phrases. Every interaction is contextually generated and unique to the moment.

Demo

How It Works

The system is built around a four-button physical interface mapped to colored hardware buttons on a Raspberry Pi with a 7-inch display. Audio is captured through a USB microphone, streamed to OpenAI's Whisper model for transcription with a prompt tuned specifically for aphasic speech patterns (including filler words, pauses, and false starts), then passed to Anthropic's Claude with a retrieval-augmented generation pipeline that incorporates the patient's personal profile, daily routines, historical communication patterns, and real-time context like time of day. Claude generates three ranked sentence interpretations with confidence scores, displayed as color-coded option cards. Upon selection, the chosen sentence is synthesized through OpenAI's TTS engine and played through the device speaker.

A conversational listening mode extends this further. A caregiver or conversation partner speaks first, their speech is transcribed to provide additional context, and then the patient responds. The system uses both inputs to generate more accurate, contextually grounded interpretations, enabling genuine back-and-forth dialogue rather than isolated requests.

The companion web dashboard, served from the same Next.js backend, provides caregivers with remote monitoring, patient profile management, and an AI-powered clinical analysis tool that identifies communication patterns, behavioral anomalies, and generates actionable care recommendations from aggregated interaction data.

Setup is conducted entirely through voice. A caretaker simply talks to the device, which conversationally gathers the patient's name, key people, medical conditions, and daily preferences, storing them as persistent context that shapes every future interaction. The device learns who the patient is, not just what they say.