AI Chat System with RAG

Impact

Built this after leaving my full-time role to go deeper into AI engineering. Core architecture complete and deployed to staging. Project on hold due to client funding constraints.

Key lesson: The architecture is solid but unvalidated by real users. If restarting, I would run a concierge MVP first (manually curating memories for a small group of beta users) before building any infrastructure.

Problem

A client wanted an AI companion chat app with memory. Users chat with an AI character, and the system remembers past conversations, extracts facts, and uses them to give contextually relevant responses over time. The roadmap also called for AI agents that could use tools (Google Search, Maps), and push notifications for proactive check-ins.

Key Decisions

• Why rebuild from a Next.js prototype to a full production stack?
  V1 was a rapid prototype (Next.js + Supabase Edge Functions + Gemini + OpenAI embeddings) to validate the core concept. It worked, but the roadmap required AI agents with tool execution and persistent background job processing. Edge Functions are stateless and short-lived, which does not fit agent workflows or persistent job polling. V2 split into: Fastify API on Cloud Run for request-response, and a pg-boss worker on GCE for background jobs.
• Why Gemini, and why make the LLM layer provider-agnostic?
  The LLM layer is abstracted behind a provider interface, so switching models is a configuration change. Gemini was the starting provider because it offered built-in grounding tools (Google Search, Maps) at a relatively low per-token cost. OpenAI was kept specifically for embeddings (text-embedding-3-small).
• How does memory retrieval work?
  Three-channel hybrid search running in parallel: Channel A is vector similarity (cosine distance via pgvector), Channel B is full-text trigram search (pg_trgm), Channel C is daily summary similarity. Results from A and B are merged by ID, then ranked by a composite score that weights semantic similarity and recency decay. Parameters were tuned through scripted batch testing + manual evaluation, not yet fully optimized at scale. Memory extraction uses LLM-based fact extraction with deduplication and contradiction detection via cosine similarity thresholds.
• Why Flutter instead of staying with Next.js?
  The product needed native mobile apps (iOS and Android) with push notifications via Firebase Cloud Messaging for proactive outreach. Flutter delivered both platforms from a single Dart codebase with Riverpod for state management and native Firebase integration.

Architecture

Project on hold. Live demo not available. Architecture deployed to staging.

Transferable Pattern

Building AI systems with persistent memory and provider-agnostic architecture. Applicable to any product that needs personalized AI interactions: customer service bots that remember past issues, sales assistants that know the client's history, or internal tools that learn from usage patterns.

Tech Stack