Building an AI Assistant with Semantic Memory

Autonomous AI agents are incredibly capable, but they suffer from one fundamental limitation: they are stateless. When a session ends, all context, user preferences, and learnings are lost. To solve this in FRIDAY, I designed a multi-tiered semantic memory architecture using vector embeddings and local retrieval.

The Memory Tier System

Human memory is divided into working memory (short-term) and episodic/semantic memory (long-term). FRIDAY mirrors this structure:

• Short-Term (Working Context): Maintained within the active LLM context window using structured message history.
• Long-Term (Episodic Memory): Past interactions, user decisions, and key accomplishments are logged, vectorized, and retrieved based on cosine similarity.
• Semantic Memory (Knowledge-Base): Project files, local documentation, and reference guidelines retrieved using Local RAG.

Technical Stack & Implementation

I chose ChromaDB as our local vector database because of its lightweight nature and ease of containerization. The embeddings are generated via Ollama (locally) or OpenAI API (cloud), depending on user configuration.

// Example schema of a memory record
interface MemoryRecord {
  id: string;
  timestamp: number;
  content: string;
  metadata: {
    category: "preference" | "episodic" | "fact";
    importance: number; // 1-10
  };
}

The Context Consolidation Pipeline

Every time a user inputs a query, a background thread performs a semantic search over the episodic store. The retrieved records are filtered using a ranking algorithm that balances relevance (similarity score) and recency (decaying weight based on age).

Once every 10 messages, a consolidation routine runs in the background. It takes the message history, feeds it to an LLM with a specialized "consolidation prompt", and extracts new facts and preferences to store in long-term memory. This prevents memory bloat while preserving essential context.