MongoDB Search Projections with Monstache and Elasticsearch

This is not a database migration

The useful way to describe this architecture is not "move MongoDB to Elasticsearch." MongoDB remains the system of record for writes, transactions, and application state. Elasticsearch receives a derived search projection that is optimized for text queries, filtering, and aggregations.

That distinction changes the operational work. A failed index is inconvenient, but it should not corrupt the source data. A search result can lag briefly behind a write, while a record detail page still reads the authoritative document from MongoDB. The goal is a rebuildable projection with an explicit consistency boundary, not two databases that are expected to behave identically.

Monstache is a Go process that can copy existing MongoDB documents into Elasticsearch and then consume subsequent changes from MongoDB's oplog. It is a practical bridge for this pattern, provided the surrounding deployment makes its assumptions visible.

Start with the replica set and oplog

MongoDB's oplog is the ordered stream Monstache uses for ongoing changes. That means a standalone MongoDB deployment is not enough for live replication: the source must run as a replica set, including a single-node replica set for development.

Before connecting Monstache, check that the replica set is healthy and that the oplog is large enough for the expected interruption window. Oplog retention is a size-and-write-rate question, not a setting that is safely copied from another environment. If the consumer is down longer than the retained history, its saved position cannot be resumed and the projection needs a controlled rebuild or reconciliation.

For a local proof of the prerequisite, initialize the replica set from mongosh after starting mongod with replica-set configuration:

rs.initiate({ _id: "rs0", members: [{ _id: 0, host: "127.0.0.1:27017" }] })

This example is intentionally local-only. Production member addresses, authentication, TLS, storage, and election settings belong in the MongoDB deployment plan rather than in a copied snippet.

Define the projection before running a sync

The first design task is deciding what Elasticsearch document represents. It is often not a byte-for-byte copy of a MongoDB document. A product record might flatten selected nested fields, normalize dates, omit internal notes, and add a combined text field for search. The projection should carry enough identity to trace it back to the source, usually by preserving the MongoDB _id as the Elasticsearch document ID.

Write down the contract for each indexed collection:

Question	Example decision
Source collection	products in the application database
Index name	Versioned alias target such as products-v3
Document ID	MongoDB _id serialized consistently
Searchable fields	name, description, and selected tags
Filter fields	Keyword, numeric, boolean, and date fields with explicit types
Sensitive fields	Excluded before indexing
Read path	Search returns IDs; canonical detail reads remain in MongoDB

Mappings are part of this contract. Letting Elasticsearch infer a mapping during an initial sync is tempting, but a single early value can establish an unsuitable type. Create the index and mapping first, especially for dates, identifiers, nested objects, keyword fields, and text analyzers. Test representative documents against the mapping before copying a large collection.

Keep the projection transform deterministic. Given the same source document and mapping version, it should produce the same indexed document. Time-dependent enrichment and remote lookups make reindexing harder to reason about and should be isolated or recorded if they are necessary.

Initial sync and the handoff to live changes

An initial sync answers a simple question: how do existing documents enter the new projection? Monstache can scan configured namespaces and write them to Elasticsearch in bulk. The harder question is what happens to writes made while that scan is running.

Treat the initial scan and oplog tailing as one handoff, not two unrelated jobs. Run the tool in the mode documented for its version that performs direct reads and then follows changes, and verify the resulting index against MongoDB. Do not declare the index current solely because a process is running.

The following is a deliberately incomplete configuration shape, not a drop-in secret-bearing deployment file. Field names and supported options must be checked against the Monstache version you install. Replace every bracketed value through your normal configuration or secret mechanism.

# config.toml - example shape only; do not copy credentials into this file
mongo-url = "mongodb://[user]:[password]@[mongo-host]/[database]?replicaSet=[replica-set]"
elasticsearch-urls = ["https://[elasticsearch-host]:9200"]
direct-read-namespaces = ["[database].[collection]"]
change-stream-namespaces = ["[database].[collection]"]
resume = true

# Configure index names, authentication, TLS validation, transforms, and mappings
# according to the Monstache release documentation and your environment.

The sample communicates intent: one source namespace, one search destination, an initial read, live change consumption, and persisted resume behavior. It does not guarantee that a particular release accepts every key or that TLS/authentication is correctly configured. Operators must validate both against the deployed version.

Updates, deletes, and resume state

Insert and update propagation gets most of the attention, but delete behavior is where a projection can quietly become misleading. Confirm that a deletion in MongoDB removes the corresponding Elasticsearch document, including deletes caused by application cleanup jobs. Test updates that unset fields as well: an older value must not remain searchable after the source field disappears.

Monstache can persist a resume position so it can continue after a restart. That state is operational data. Keep it durable, scoped to the deployment, and backed up with the same care as the configuration that names the projection. Reusing or deleting it without a deliberate rebuild plan can skip history or replay more data than intended.

Resume only works while the required oplog history is still retained. Monitor the age of the oldest available oplog entry, the consumer's latest processed time or position, and the difference between them. If that margin approaches the planned outage window, increase oplog capacity, reduce the interruption, or prepare a rebuild.

Lag, retries, and backpressure

"Real time" is not a useful promise without a measurement. Define a lag signal that can be compared to a threshold: for example, the source change timestamp versus the time the matching version becomes searchable. Record percentiles and the maximum, not only an average.

Elasticsearch bulk writes can slow down because of indexing pressure, shard recovery, disk limits, mapping errors, or a cluster outage. A correct consumer needs bounded retries and a place to surface failures. Unlimited retries can create an unbounded queue; dropping failed writes creates silent divergence. Choose an explicit policy for transient failures, permanent document failures, and a full destination outage.

Backpressure is not a feature to assume from a diagram. Establish queue limits, alert on growing lag, and load test the expected write rate plus a recovery burst. If a transformation is expensive, measure it separately from Elasticsearch indexing so an operator can identify which stage is limiting throughput.

Decide what evidence is retained for failed documents. A dead-letter record with source ID, change position, error category, and retry history gives an operator a way to repair an exception without guessing which write was lost.

Reconcile instead of trusting the stream blindly

Oplog consumption reduces the window for divergence; it does not eliminate operational mistakes. Mapping changes, bad transforms, failed bulk requests, and an expired resume position can leave the index incomplete even when the process later looks healthy.

Schedule reconciliation that compares source and projection at a level appropriate for the collection. For a small collection, that may be ID and count comparison followed by document-level checks. For a large collection, sample by ID ranges or update windows, calculate checksums for projected fields, and investigate mismatches. Reconciliation should produce an actionable report, not merely a dashboard number.

Keep a repeatable reindex path. A versioned index and alias make this less risky: build products-v4, validate it, then switch the read alias after the acceptance checks pass. The old index becomes a short-lived rollback target rather than an artifact that is overwritten in place.

Cut over deliberately and keep rollback simple

Before an application reads from Elasticsearch, define what it may return. Search should tolerate eventual consistency, and the application should handle a result whose canonical document was deleted between search and detail lookup. For workflows that require strict current state, query MongoDB or use a separate confirmation read.

Run the new search path in shadow mode when possible. Compare result IDs, filters, and relevance behavior without changing the user-facing path. Then enable it behind a flag or an alias switch with a named owner and rollback condition.

Rollback should mean routing reads back to the previous index or MongoDB path. It should not mean writing the derived Elasticsearch data back into MongoDB. The source of truth remains MongoDB throughout, which is the reason the rollback is tractable.

Readiness checklist

Before treating the projection as a dependency, verify the following:

• MongoDB is a healthy replica set and its oplog retention covers the recovery window.
• Index mappings, analyzers, and excluded fields were reviewed with representative documents.
• Initial sync, inserts, updates, field removals, and deletes were tested end to end.
• Resume state is durable, monitored, and understood by the on-call runbook.
• Lag, retry failures, queue depth, and Elasticsearch health have alerts with owners.
• Reconciliation and a versioned reindex procedure have been exercised.
• The read path documents where eventual consistency is acceptable and where it is not.
• Cutover and rollback are named operations, with an alias or feature flag that can be changed safely.

The durable outcome is a searchable projection that can be inspected, rebuilt, and rolled back without redefining MongoDB's role as the authoritative store.