Sematica
The Problem
Approximately two million people in the United States live with aphasia, a language disorder most commonly caused by stroke. For those with Broca's aphasia specifically, the condition leaves cognition largely intact while severely disrupting speech production. Patients know exactly what they want to say but can only produce fragmented utterances, like "doctor... go... Wednesday" instead of "I have a doctor's appointment on Wednesday."
Current assistive communication devices fail these users. They act like static digital dictionaries, forcing patients to hunt through complex, frustrating menus to build sentences word-by-word. For someone already struggling to communicate, this is a slow and exhausting process.
The Solution
For TreeHacks, I built Sematica, a portable device that bridges this gap by using AI as a real-time translator.
Instead of relying on static menus, my approach treats communication as a context-prediction problem. Because large language models excel at predicting context, the system listens to the user's fragmented spoken keywords and analyzes their daily routines and conversational history to instantly generate the three most likely intended sentences.
I intentionally designed this as a physical hardware device rather than a mobile app. Many stroke survivors experience physical weakness on one side of their body or a loss of fine motor control, making touchscreens frustrating and inaccurate to use. By pairing a clear display with large, color-coded tactile buttons, the device bypasses the need for precise finger movements, allowing users to confidently select and speak their intended sentence with a simple press.
How It Works
Hardware
- Raspberry Pi with 7-inch display
- Four color-coded GPIO buttons for selection
- USB microphone and speaker
AI Pipeline
- OpenAI Whisper transcribes speech, with a prompt tuned for aphasic patterns: fragmented speech, fillers, and pauses
- Claude generates three ranked sentence interpretations using the patient's profile, daily routines, and real-time context
- A listening mode lets caregivers speak first, giving the system more context for back-and-forth dialogue
- OpenAI TTS synthesizes the selected sentence and plays it aloud
Web Dashboard
- Next.js backend served directly from the device
- Caregiver tools for remote monitoring and patient profile management
- AI-powered clinical analysis identifies communication patterns and generates care recommendations
