--- name: elasticsearch-and-opensearch-tuning description: Production guidance and performance tuning strategies for Elasticsearch and OpenSearch clusters, covering indexing, query optimization, mapping design, memory management, and scaling. --- # Elasticsearch & OpenSearch Tuning Guide This skill provides comprehensive instructions, architectural pattern guidelines, and production-ready code examples for optimizing Elasticsearch and OpenSearch for high-throughput indexing, ultra-low latency search, and cost-effective cluster scaling. --- ## 1. Indexing & Mapping Strategies ### 1.1 Field Mapping Optimization - **Disable `_source` selectively** only when raw document retrieval is unnecessary and storage budget is tight; otherwise keep `_source` enabled for reindexing capabilities. - **Set `index: false`** on fields used only for storage/retrieval (e.g., metadata blobs, raw HTML strings) to prevent inverted index overhead. - **Explicit Mappings over Dynamic Mappings**: Never rely on dynamic mapping in production. Dynamic mapping introduces field explosion risks (`keyword` + `text` dual fields by default). - **Use `doc_values: false`** for high-cardinality text or keyword fields that will **never** be used for aggregations, sorting, or scripting. - **Optimize Text Analysis**: - Disable norm storage (`norms: false`) on text fields where relevance scoring is not required (e.g., exact matches, filter-only text). - Use `index_options: docs` for fields where positional or phrase queries are not needed (saves term frequency and position offset storage). #### Optimized Mapping Definition (DSL) ```json { "settings": { "index": { "number_of_shards": 3, "number_of_replicas": 1, "refresh_interval": "30s", "translog": { "durability": "async", "sync_interval": "5s", "flush_threshold_size": "1gb" }, "codec": "best_compression" } }, "mappings": { "dynamic": "strict", "properties": { "id": { "type": "keyword", "doc_values": true }, "title": { "type": "text", "analyzer": "standard", "fields": { "raw": { "type": "keyword", "ignore_above": 256 } } }, "body": { "type": "text", "analyzer": "standard", "index_options": "positions", "norms": true }, "category_id": { "type": "keyword", "doc_values": true, "norms": false }, "status": { "type": "keyword", "doc_values": true }, "created_at": { "type": "date", "format": "strict_date_optional_time||epoch_millis" }, "payload_metadata": { "type": "keyword", "index": false, "doc_values": false } } } } ``` --- ### 1.2 Bulk Ingestion & Shard Sizing - **Bulk Batch Size**: Ideal bulk request payload is typically between **5MB to 15MB** per request (or 2,000–5,000 documents depending on doc size). Tune based on CPU and throughput benchmarks rather than fixed doc counts. - **Shard Sizing Standard**: - **Search-heavy indices**: Shard size should be **10GB to 30GB**. - **Logging / Time-series indices**: Shard size should be **30GB to 50GB**. - Target total shards per GB of JVM heap: Keep under **20 shards per GB of heap**. - **Indexing Throughput Tuning**: - Increase `refresh_interval` from `1s` to `30s` or `60s` during heavy bulk ingestion. - Set `number_of_replicas: 0` during initial bulk loading, then restore to `1` or `2` after load completes. - Configure `translog.durability: async` for non-critical logging workloads to eliminate synchronous disk commits on every indexing request. --- ## 2. Query Tuning & Performance Optimization ### 2.1 Query Execution: Filter vs. Query Context - **Filter Context (`filter`, `must_not`)**: Does not compute relevance scores (`_score`). Results are cached in the **Node Query Cache** (LRU bitsets). Always place non-scoring exact-match conditions inside `filter`. - **Query Context (`must`, `should`)**: Computes relevance scores using BM25. Use exclusively for text relevance matching. #### Optimized Multi-Clause Search Request ```json { "query": { "bool": { "filter": [ { "term": { "status": "active" } }, { "range": { "created_at": { "gte": "now-7d/d" } } } ], "must": [ { "match": { "title": { "query": "distributed graph database", "operator": "and", "fuzziness": "AUTO:4,8" } } } ], "should": [ { "match_phrase": { "body": { "query": "distributed graph database", "slop": 2, "boost": 2.5 } } } ] } }, "timeout": "500ms", "terminate_after": 10000, "_source": ["id", "title", "created_at"] } ``` --- ### 2.2 Deep Pagination Mitigation - **Avoid `from` + `size`** for deep page iteration (`from + size > 10,000` causes Memory Overflow and high JVM CPU utilization). - **Use `search_after` with Point In Time (PIT)** for deterministic stateless pagination over large result sets: #### Python Production Example: `search_after` with PIT ```python import logging from typing import List, Dict, Any, Generator from opensearchpy import OpenSearch, NotFoundError logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) class DeepPaginationClient: def __init__(self, client: OpenSearch, index_name: str): self.client = client self.index_name = index_name def iterate_all_documents( self, query: Dict[str, Any], page_size: int = 1000 ) -> Generator[Dict[str, Any], None, None]: """ Executes deep pagination efficiently using Point In Time (PIT) / search_after. """ # Create Point In Time (PIT) context (OpenSearch / ES 7.10+) try: pit_resp = self.client.create_point_in_time( index=self.index_name, keep_alive="2m" ) pit_id = pit_resp["pit_id"] except Exception as e: logger.error("Failed to create Point In Time context: %s", str(e)) raise search_after_mark = None total_fetched = 0 try: while True: body: Dict[str, Any] = { "size": page_size, "query": query, "sort": [ {"created_at": {"order": "desc"}}, {"_shard_doc": {"order": "asc"}} # Tie-breaker ], "pit": { "id": pit_id, "keep_alive": "2m" } } if search_after_mark: body["search_after"] = search_after_mark response = self.client.search(body=body) hits = response["hits"]["hits"] if not hits: break for hit in hits: yield hit["_source"] total_fetched += 1 search_after_mark = hits[-1]["sort"] logger.debug("Fetched batch of %d docs. Total: %d", len(hits), total_fetched) finally: # Always release PIT resources try: self.client.delete_point_in_time(body={"pit_id": pit_id}) except Exception as e: logger.warning("Failed to delete PIT context: %s", str(e)) ``` --- ## 3. Memory & Infrastructure Management ### 3.1 JVM Heap Allocation - Set `-Xms` and `-Xmx` to the **exact same value** to prevent dynamic heap resizing runtime pauses. - **Never exceed 32GB heap**: Keep JVM heap under the Compressed OOPs threshold (typically **30GB–31GB** depending on JDK version) to ensure 32-bit pointer efficiency. - Allocate **50% of total system RAM to JVM Heap** and leave the remaining 50% for **OS Filesystem Page Cache** (crucial for Doc Values, Lucene segment caching, and bloom filters). ### 3.2 Circuit Breakers & Thread Pools - **Parent Circuit Breaker**: Set `indices.breaker.total.use_real_memory: true` with limit `70%` to prevent OutOfMemoryError (OOM). - **Fielddata Breaker**: Set `indices.breaker.fielddata.limit: 40%`. Avoid using `fielddata: true` on `text` fields; use `keyword` fields with `doc_values`. - **Search Thread Pool Queue**: `thread_pool.search.queue_size: 1000`. Do not set arbitrarily high; larger queues cause task latency accumulation and high garbage collection pressure. --- ## 4. Production Anti-Patterns & Solutions | Anti-Pattern | Operational Impact | Recommended Solution | | :--- | :--- | :--- | | **Dynamic Mapping Explosion** | Unbounded field creation leads to cluster state degradation and OOM. | Set `"dynamic": "strict"` in index mappings. | | **Using `wildcard` with leading asterisk (`*foo`)** | Forces full inverted index scanning; severe CPU consumption. | Use `wildcard` field type or n-gram tokenizer (`edge_ngram`). | | **Over-Sharding** | Thousands of small shards (<1GB) exhaust JVM heap on metadata. | Reindex/Shrink indices to maintain 10GB–50GB per shard. | | **Deep `from` + `size` Pagination** | Linear RAM and CPU scaling ($O(N)$ memory cost per request). | Use `search_after` with PIT or `scroll` API (for batch export). | | **Heavy `script_score` Without Caching** | High CPU overhead per doc evaluation during query phase. | Pre-compute values at ingestion or use `painless` optimized scripts. | --- ## 5. Diagnostic & Monitoring Commands ### Checking Cluster Health & Shard Distribution ```bash # Cluster health summary GET /_cluster/health?v # Shard count and size per index GET /_cat/shards?v&s=index,shard&h=index,shard,prirep,state,docs,store,node # Circuit breaker states and memory usage GET /_nodes/stats/breaker # Fielddata and Query Cache memory usage GET /_nodes/stats/indices/fielddata,query_cache,request_cache?human=true ```