An AI-powered virtual patient for medical education. Students practice clinical assessments — like the Glasgow Coma Scale — on a fully embodied 3D avatar that responds with realistic speech, animation, and medical behavior. Built to run on VR, desktop, and browser.
Charlie is a university research project led by Prof. Dr. med. Daniela Becker (concept & medical supervision) and Prof. Armin Grasnick (technical direction). I designed and built the entire technical implementation — from 3D character to animation system to AI pipeline.
Teaching clinical assessment requires real patients or expensive SimMan mannequins (€500k+). Students get limited practice time, no repeatability, and no immediate feedback. Charlie replaces this with an AI patient available 24/7, from any browser, that behaves like a real patient — confused speech, pain responses, involuntary motor reactions.
Charlie isn't just a chatbot — it's a multi-purpose educational AI with distinct operational modes.
GCS/OSCE training with a virtual patient. Students examine, diagnose, and receive scored feedback.
LiveRAG-enhanced Q&A tied to course materials. Answers adapt to student level and curriculum context.
PlannedLive meeting attendance, automated note-taking, and session summaries for remote learning.
PlannedFrom spoken word to animated response in under 500ms. The entire pipeline can run locally — zero cloud dependencies when needed.
16kHz mic capture
Speech to text
Semantic parser
Medical rules
Context generation
Speech synthesis
Avatar response
Whisper.cpp for offline mode (12+ concurrent sessions, zero API cost) or Whisper API for cloud. German and English support.
Custom semantic classifier parsing complex medical commands. Handles multi-intent inputs like "lift left arm and tell me your name" as two separate actions.
Ollama (local) or GPT-4o-mini (cloud) for contextual responses. RAG retrieval from ChromaDB with embedded course materials.
Piper TTS for local synthesis in 50+ languages. ElevenLabs as premium alternative. Natural voice output synchronized with avatar lip-sync.
3-layer animator: base body + action responses + facial blend shapes. Pain responses override all states. Eye behavior follows GCS protocol.
Single codebase targeting Desktop, Quest 3 VR (90 FPS, <20ms latency), PCVR, and Web via Pixel Streaming.
The GCS is the standard for assessing consciousness in clinical settings. Charlie implements all 90 possible combinations with medically accurate responses — from fully oriented (GCS 15) to unresponsive (GCS 3).
| Component | Range | What Charlie Does |
|---|---|---|
| Eye Response (E) | E1 — E4 | No opening → opens to pain → opens to voice → spontaneous. Blend shape animation with smooth interpolation. |
| Verbal Response (V) | V1 — V5 | None → groaning → confused words → disoriented speech → fully oriented conversation. |
| Motor Response (M) | M1 — M6 | No movement → extension → flexion → withdrawal → localizes pain → follows commands. 19 distinct animations. |
Unity (v1.0 shipped) → Unreal Engine 5.4 (v2.0 in development). MetaHuman avatar with FACS facial animation.
Python 3.11 + FastAPI. PostgreSQL for session data, Redis for caching, Docker for deployment.
Whisper (STT), Custom NLU (C++), ChromaDB + Sentence Transformers (RAG), Ollama/GPT-4o (LLM), Piper (TTS).
OpenXR for hardware abstraction. Quest 3 native APK, SteamVR, Pixel Streaming for browser access.
Every clinically valid combination of Eye, Verbal, and Motor responses — implemented and medically verified.
Growing phrase database for medical command recognition. Self-learning: admin can add new patterns at runtime.
Desktop, Quest 3 VR, PCVR, and Web browser — from a single codebase with platform-specific optimizations.
From spoken question to animated response. Fast enough for natural conversation flow in VR.
Piper TTS supports global deployment — Arabic, Chinese, Spanish, and more. NLU currently covers German and English.
UE5 version targeting live demo at Gamescom, August 2026. 12 two-week sprints from concept to showfloor.