Fluo: Multi-Agent Language Learning

My partner is Turkish, and I’m taking classes to learn his language. But there’s a gap that no textbook or classroom can fill—I don’t have a learning companion who actually knows what I’ve learned and can meet me where I am. So I built it.

Architecture Overview
I built a multi-agent conversational system using FastAPI, SQLAlchemy, and OpenAI’s API (GPT-4o + Whisper + TTS). The system maintains persistent state of the learner’s knowledge and dynamically adapts conversation difficulty in real-time.

Core Components

1. Note Extraction Pipeline (NoteExtractorAgent)

Accepts multiple image uploads or raw text input
Uses GPT-4o’s vision capabilities to parse handwritten/printed class notes
Extracts structured data: vocabulary (target word, native translation, example sentences) and grammar concepts (name, description, usage examples)
Supports configurable native/target language pairs

2. Knowledge Persistence Layer

SQLAlchemy ORM with SQLite backend
Per-language vocabulary and grammar tables with fluency tracking (low/medium/high)
Tracks times_practiced and times_correct for spaced-repetition-style prioritization

3. Lesson Planning Agent (LessonPlannerAgent)

Selects vocabulary/grammar items prioritizing low-fluency scores
Generates a structured lesson plan constrained to CEFR levels (A1-C2)
Passes both practice items and all known items to enable natural vocabulary reuse without exceeding learner’s level

4. Dialogue Agent (DialogueAgent)

Maintains conversation state and full transcript
Bidirectional speech support: Whisper for STT, OpenAI TTS for audio responses
Receives lesson plan + full knowledge context to stay within learner’s vocabulary bounds
Detects conversation end signals and handles graceful session termination

5. Analysis Agent (AnalysisAgent)

Post-conversation transcript analysis
Identifies: correctly used target vocabulary, grammatical errors, new words attempted
Updates fluency scores in database based on performance