Cohere Embed and Rerank vs OpenAI embeddings

Cohere Embed and Rerank

Rerank 4 Pro and Fast with 32K context and top_n, tracked per model on the status page. Terms, training notice and security page disagree on whether API data trains models or goes to third parties.

OpenAI embeddings

text-embedding-3-small at $0.02 per million tokens, $0.01 through the Batch API. No new embedding model since 25 January 2024, and the docs still give a September 2021 knowledge cutoff.

Cohere Embed and Rerank

1. Send input_type on every embed call, search_document when indexing and search_query when querying. The endpoint rejects a call without it
2. Batch 96 inputs a call, the maximum, stay under 2,000 inputs a minute, and check every batch returns every embedding type you asked for (an open SDK bug drops types missing from the first response)
3. Budget rerank by searches. One query with up to 100 documents is one search, and a document over 500 tokens counts as several
4. Set max_tokens_per_doc on rerank. The default of 4,096 truncates long documents even on the 32K models
5. Ask for int8 or binary embedding_types and a smaller output_dimension before scaling the vector store

OpenAI embeddings

1. Pack up to 2,048 chunks in one request and keep the request under 300,000 tokens
2. Count tokens before sending. An input over 8,192 tokens is rejected, not truncated
3. Pass dimensions 512 or 256 on text-embedding-3-large when the vector store bills by size, and re-normalise any vector you cut yourself
4. Split a Batch API index job into batches of under 50,000 inputs. It's half price with a 24-hour window
5. Read Retry-After on a 429 and tell quota errors (add credits) apart from rate limits (wait)