Elasticsearch & OpenSearch

Elasticsearch is a distributed, JSON-document search and analytics engine built on Apache Lucene, originally released by Shay Banon in 2010. OpenSearch is the open-source fork created in 2021 by AWS after Elastic relicensed Elasticsearch under the SSPL/Elastic License; OpenSearch is Apache 2.0 and a drop-in replacement for the Elasticsearch 7.x APIs. Both engines are simultaneously a NoSQL document store, a distributed full-text search engine, and a real-time analytics platform.

OpenSearch is the Apache 2.0 fork of Elasticsearch 7.10 that AWS started in 2021 after Elastic moved Elasticsearch to non-open-source licences. (Elastic later added AGPL as an additional licence option in 2024.) The two engines share the same core design, a cluster of nodes holding Lucene indexes split into shards, and the same basic REST API, so most of this page applies to both.

Cluster architecture

An index is a logical collection of documents divided into a fixed number of primary shards, each a self-contained Lucene index. Replicas are copies of primaries on other nodes; they provide failover and extra read capacity. A search is fanned out to one copy of every shard, each shard returns its local top hits, and the coordinating node merges them into the final ranking. The number of primary shards is set at index creation, so it is usually sized for expected data volume, with indexes rolled over by time for logs.

Key Features:

• Inverted Index. Lucene’s posting lists make full-text search across billions of documents tractable in milliseconds.
• BM25 Relevance Scoring. Industry-standard probabilistic relevance ranking; tunable per-field.
• JSON Documents. Schema-flexible — each document is a JSON blob; mappings are inferred or declared.
• Aggregations. Bucketing, metrics, percentile, histograms, geo-aggregations — the analytical engine that powers Kibana / OpenSearch Dashboards.
• Distributed Sharding & Replication. Indexes split into shards, each replicated across nodes; automatic rebalancing.
• Vector Search. Both engines added native dense-vector ANN (HNSW) for semantic search and RAG use cases.
• Ingest Pipelines. Server-side document transformations (parsing, enrichment, geo-IP) before indexing.
• SQL. Limited SQL surface (SELECT ... FROM index WHERE ...) on top of the underlying query DSL.

Elasticsearch vs. OpenSearch:

• Elasticsearch. Ahead on commercial features (machine-learning anomaly detection, graph exploration, security tier). SSPL / Elastic License.
• OpenSearch. Apache 2.0 (truly open). AWS-backed; default on AWS managed search. Increasingly feature-comparable.
• For new self-hosted deployments without commercial features, OpenSearch is now the more conservative choice.

Use Cases:

• Application search. Product catalogs, documentation, knowledge bases.
• Log analytics. The ELK / OpenSearch stack — Logstash / Beats + Elasticsearch + Kibana — is the dominant open observability platform.
• Security analytics. SIEM workloads (high-volume log ingest with correlation rules).
• Hybrid search for RAG. BM25 + dense vector ANN combined and reranked — the production RAG retrieval pattern.
• Geospatial search. Location-based discovery for marketplaces and mapping apps.

Write and search path

Text fields are analyzed (tokenized, lower-cased, optionally stemmed) at write time, and the resulting terms go into the inverted index; keyword, numeric, and date fields go into column-oriented doc values used for sorting and aggregations. The translog makes acknowledged writes durable before segments are flushed to disk. Because segments are immutable, updates are implemented as delete plus reinsert, and deleted documents are only physically removed when segments merge.

Worked example: mapping, indexing, querying

# Create an index with an explicit mapping
curl -X PUT localhost:9200/articles -H 'Content-Type: application/json' -d '{
  "settings": { "number_of_shards": 1, "number_of_replicas": 1 },
  "mappings": { "properties": {
      "title":     { "type": "text" },
      "body":      { "type": "text", "analyzer": "english" },
      "tags":      { "type": "keyword" },
      "published": { "type": "date" }
  } }
}'

# Index a document
curl -X PUT localhost:9200/articles/_doc/1 -H 'Content-Type: application/json' -d '{
  "title": "Hybrid search for RAG", "body": "Combining BM25 with dense vectors...",
  "tags": ["rag", "search"], "published": "2025-06-01"
}'

# Full-text query with a filter and an aggregation
curl -X POST localhost:9200/articles/_search -H 'Content-Type: application/json' -d '{
  "query": { "bool": {
      "must":   { "multi_match": { "query": "bm25 vectors", "fields": ["title^2", "body"] } },
      "filter": { "range": { "published": { "gte": "2025-01-01" } } }
  } },
  "aggs": { "by_tag": { "terms": { "field": "tags" } } }
}'

The bool query separates scoring clauses (must) from non-scoring filters (filter), which are cached and cheaper. title^2 doubles the weight of title matches. The terms aggregation runs on the keyword field's doc values; running it on a text field fails by default.

When to use it vs alternatives

Pitfalls

• Dynamic mapping surprises. The first document decides field types; a string that looks like a date becomes a date. Declare mappings for important fields.
• Too many shards. Each shard costs memory and file handles; thousands of tiny shards destabilize clusters.
• Using it as the primary store. Keep the source of truth elsewhere and reindex from it.
• Deep pagination. from + size beyond 10,000 hits is rejected by default; use search_after.
• Security defaults. Never expose port 9200 publicly; enable authentication and TLS.

Related topics

• InfluxDB - time-series alternative
• BM25 - default relevance scoring
• Python Whoosh - embedded BM25 search
• Search and time-series databases

Featured Videos