AI Record Manager & Vector Database Sync
A workflow that synchronises external content with a Pinecone vector database while maintaining a record manager in MySQL, so embeddings reflect the current state of the source rather than drifting out of date.
- Year
- 2025
- Engagement
- Automation & data engineering
- Focus
- Data Engineering, Generative AI, Automation
- Status
- Open source
01 — The problem
A RAG system is only as current as its index. Source content changes, gets deleted, gets re-added — and a vector store with no memory of what it already ingested will happily serve stale or duplicated answers.
02 — Our solution
A record manager in MySQL tracks what has been ingested and in what state, so each sync run knows what is new, what changed and what should be removed from Pinecone. The vector store becomes a projection of the source rather than an append-only pile.
How it works
The pipeline, step by step — from the first input to the final output.
- 01
Source read
External content is read on a schedule or trigger.
- 02
Change detection
The MySQL record manager is compared against the source to classify each item as new, changed, unchanged or removed.
- 03
Embedding
New and changed content is chunked and embedded.
- 04
Pinecone upsert
Vectors are upserted, and vectors for removed content are deleted.
- 05
Record update
The record manager is updated so the next run starts from an accurate baseline.
What we built
The outcomes
01
An index that stays consistent with its source
02
Incremental syncs rather than full rebuilds
03
A reusable pattern for any RAG ingestion pipeline
Challenges we solved
Deletes are the hard part
Adding vectors is easy; knowing what to remove requires state. The record manager exists precisely to answer that.
Avoiding wasteful re-embedding
Re-embedding everything is simple and expensive. Change detection keeps each run proportional to what actually moved.
Technology stack
Want results like these?
Book a free 30-minute discovery call. We'll map out your project, recommend the right stack and send a fixed quote within 48 hours.