# Load a corpus and keep the index in step with its source

> Load a corpus and keep the index in step with its source: 0 companies fully do it, 14 with a limit, 17 carry the tag. Every listing in Retrieval & vector search with the reason and a source.

- Canonical: https://www.anchorterminal.com/niches/vector-search.ingest-sync/
- Markdown: https://www.anchorterminal.com/niches/vector-search.ingest-sync/index.md (~1,300 tokens)
- Slim: https://www.anchorterminal.com/niches/vector-search.ingest-sync/index.min.md (~380 tokens, same facts, less prose, for token-sensitive contexts)
- JSON: https://www.anchorterminal.com/niches/vector-search.ingest-sync/index.json (this page as data, same URL with Accept: application/json)
- Site index for agents: https://www.anchorterminal.com/llms.txt (the key docs and every listing in one file, at most 16 MB, about 4 million tokens: https://www.anchorterminal.com/llms-full.txt)
- API: https://www.anchorterminal.com/api/v1/index.json
- MCP: https://www.anchorterminal.com/mcp (the whole catalogue as tools, no key)
- DNS: dig +short TXT <slug>.grade.anchorterminal.com (the grade as a DNS TXT record, for a sandbox with no HTTP where names still resolve; also top., best.<category>. and <a>.vs.<b>. under the zone, and the same under grade.letme.dev)
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)
- Updated: 2026-11-15

No company's docs show a full fit yet. The listings below do the job with a limit, or not at all.

| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |
| --- | --- | --- | --- | --- |
| 0 | 14 | 17 | 14 | 8 |

## What counts as done

Every source record is searchable with current metadata, deleted records no longer appear, the agent can tell when a sync or import has finished, and load time grows linearly with batch count.

Measured, for our own tests: 100,000 records load in under 30 minutes and an edited record is searchable within 60 seconds.

How agents ask for it: index this folder; re-sync after the docs changed; delete everything from the old site; import these 100,000 records

## Does the job, with a limit

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

- **Chroma API + MCP (Chroma Inc.)**: Chroma Sync ingests S3, GitHub and web sources with tracked invocations, but it is Cloud only, Cloud writes cap at 300 records, and #78 shows chroma_add_documents slowing as the collection grows.
- **Cloudflare AI Search (Cloudflare, Inc.)**: Managed sync from websites, R2 buckets and D1 runs as jobs whose status the Instances API reports, but the shortest sync interval is 15 minutes (default 24 hours), far past the 60-second edit bar.
- **Elasticsearch (Elasticsearch B.V.)**: Search connectors sync data from third-party sources and the bulk API loads batches, but sync-job status and change delay were not on the pages read.
- **Epsilla Vector Database (Epsilla, Inc.)**: Insert and upsert take records in batches, but there is no import job, source sync or completion status.
- **LanceDB (LanceDB Systems, Inc.)**: Insert, import and delete by predicate are documented, but nothing syncs from a source system or reports a sync job.
- **Meilisearch (Meili SAS)**: Writes are async tasks with a taskUid the agent can poll, but there is no connector that syncs a bucket, site or database, and load time at scale is not stated.
- **Milvus and Zilliz Cloud API + MCP (Zilliz Inc.)**: Bulk import jobs load files from object storage with job status and idempotency keys, but nothing syncs from a source system, and load time against collection size is not stated.
- **OpenSearch (OpenSearch Software Foundation, a project of The Linux Foundation)**: No managed source connector or sync-job status was on the pages read; ingest runs through the owner's own pipeline into a cluster they operate, and accounts report heap pressure past 100M vectors.
- **Pinecone API + MCP (Pinecone Systems Inc.)**: Imports from S3, GCS or Azure Blob run as tracked operations and LSNs show when writes are visible, but import needs a storage integration, and #10 shows the MCP server has no ingest path for a local file.
- **Qdrant API + MCP (Qdrant Solutions GmbH)**: Batch upserts and a bulk-upload guide, but no connector syncs from a source, and #6009 shows the last points staying unindexed with searches past 60 seconds after a large load.
- **Typesense API + MCP (Typesense, Inc.)**: Bulk JSONL import with action=upsert and deletes are documented, but nothing syncs from a source system or reports a sync job.
- **Upstash Vector API + MCP (Upstash, Inc.)**: Upsert takes batches of vectors or text, but there is no import job, no source sync and no stated batch limit on the pages read.
- **Vespa (Vespa.ai AS)**: The feed client and vespa feed load documents in bulk, but nothing syncs from a source system or reports a sync job.
- **Weaviate API + MCP (Weaviate B.V.)**: Batch import is documented, but there is no connector that syncs a bucket, site or database and reports when a sync finished.

## What stops agents finishing

- bulk add slows as the collection grows
- sync from a hosted source lags for hours
- no ingest tool in the MCP server

## Around it

Before: document-extraction.parse, file-storage.list-objects. After: vector-search.rag-over-docs.

## Where this comes from

Weight 14 of 100 in Retrieval & vector search: 8 of 37 accounts (sync pain between Postgres and Qdrant, bulk loads that slow down, unindexed points, slow metadata updates, missing ingest tools)

As data: https://www.anchorterminal.com/api/v1/use-cases/vector-search.ingest-sync.json, or the MCP tool get_use_case with id vector-search.ingest-sync. Draft, read 2026-11-15 from public docs and first-hand accounts, not yet reviewed by a person.

---

Query this catalogue directly over MCP: https://www.anchorterminal.com/mcp (Streamable HTTP, no key). How: https://www.anchorterminal.com/agents/#mcp
