VoiceMem gives voice agents a streaming long-term memory

What this is about

VoiceMem is an open memory component for voice agents. It processes ongoing conversations, separates factual information from emotional and personal context, and passes only relevant memories to a response model. The project appeared in late August 2026; version 0.0.2 on September 1 opened the speech-synthesis layer to interchangeable components, among other changes.

The software targets developers who want to give a voice assistant persistent memory. It is not a finished phone or assistant application. VoiceMem is integrated as a Python package, includes a local web demo, and publishes models, utility files, training data, and evaluation code.

What VoiceMem actually does

VoiceMem divides memories into two areas. Its so-called left brain organizes facts through schemas and entities. The right brain holds information about personality, relationships, and emotional development. During speech, a pipeline processes audio segments, transcription, speaker cues, sounds, and emotion. Speculative retrieval can begin before the user finishes a sentence.

For a request, memories are selected and ranked first. Only the highest-ranked entries enter the response model’s context. Retrieval itself can run locally. Extracting new facts in the default setup, however, uses an OpenAI key; the documentation describes the components as replaceable. The package is licensed under Apache 2.0 and can be installed with pip install voicemem.

Why it matters

Without persistent storage, voice agents either lose conversational context or repeatedly send long histories to a model. Both options are awkward: forgetting harms the experience, while complete transcripts increase cost and privacy exposure. VoiceMem tries to provide only small, relevant excerpts while separating facts from relationships and emotional context.

The repository publishes a technical report, the ChatMem-400K dataset, and reproducible evaluation code. Reported figures, including results on LoCoMo and PersonaMem, come from the project team and should not be read as an independent product comparison. The immediate practical value lies in an inspectable architecture and the ability to replace storage, models, and speech synthesis.

In plain language

Imagine a receptionist with two notebooks. The first contains plain facts such as allergies or appointment restrictions. The second contains relationships and moods, such as the fact that a topic was stressful. Before a new conversation, the receptionist opens only the three to five relevant notes instead of placing the entire archive on the desk.

A practical example

A team builds a voice assistant for appointment scheduling. In the first conversation, a user says she is never available on Tuesdays and prefers morning calls. VoiceMem stores those two details. One week later she asks for an appointment; the assistant retrieves only the relevant memories and suggests Thursday at 10 a.m.

The team tests 100 synthetic conversation sequences and separately checks whether facts are stored correctly, retrieved later, and removed after withdrawal. Voluntary tests with real people follow only after that. Raw recordings are not retained, and sensitive categories are excluded from storage.

Scope and limits

First, version 0.0.2 is a young project. Interfaces, data formats, and operating characteristics may change. Production systems need their own load, failure, and migration tests.

Second, open source does not automatically mean fully local. The default examples use an external API key for fact extraction; depending on configuration, audio, transcripts, or derived data may reach a third party.

Third, memories can be wrong, outdated, or assigned to the wrong person. Speaker recognition and emotion analysis are not reliable proof of identity or truth. Users need access, correction, and deletion, and sensitive decisions must not rely solely on stored inferences.

SEO & GEO keywords

VoiceMem, voice agents, long-term memory, streaming memory, Voice AI, local AI, Apache 2.0, ChatMem-400K, LoCoMo, privacy