Rejected requests
Stack ECE ECK ECH Self-Managed
When Elasticsearch rejects a request, it stops the operation and returns an error with a 429 response code. Rejected requests are commonly caused by:
- A depleted thread pool. A depleted
searchorwritethread pool returns aTOO_MANY_REQUESTSerror message. - A circuit breaker error.
- High indexing pressure that exceeds the
indexing_pressure.memory.limit.
If you're using Elastic Cloud Hosted, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, and real-time issue detection with resolution paths. For more information, refer to Monitor with AutoOps.
To check the number of rejected tasks for each thread pool, use the cat thread pool API. A high ratio of rejected to completed tasks, particularly in the search and write thread pools, means Elasticsearch regularly rejects requests.
GET /_cat/thread_pool?v=true&h=id,name,queue,active,rejected,completed
write thread pool rejections frequently appear in the erring API and correlating log as EsRejectedExecutionException with either QueueResizingEsThreadPoolExecutor or queue capacity.
These errors are often related to backlogged tasks.
See this video for a walkthrough of troubleshooting threadpool rejections.
To check the number of tripped circuit breakers, use the node stats API.
GET /_nodes/stats/breaker
These statistics are cumulative from node startup. For more information, see circuit breaker errors.
See this video for a walkthrough of diagnosing circuit breaker errors.
To check the number of indexing pressure rejections, use the node stats API.
GET _nodes/stats?human&filter_path=nodes.*.indexing_pressure
These stats are cumulative from node startup.
Indexing pressure rejections appear as an EsRejectedExecutionException, and indicate that they were rejected due to combined_coordinating_and_primary, coordinating, primary, or replica.
These errors are often related to backlogged tasks, bulk index sizing, or the ingest target's refresh_interval setting.
Stack
Serverless
Another cause of indexing pressure rejections might be the use of the semantic_text field type, which can cause rejections when indexing large batches of documents if the batch may otherwise incur an Out of Memory (OOM) error.
See this video for a walkthrough of diagnosing indexing pressure rejections.
If Elasticsearch regularly rejects requests and other tasks, your cluster likely has high CPU usage or high JVM memory pressure. For tips, see High CPU usage and High JVM memory pressure.
Stack Serverless
When bulk indexing documents with the semantic_text field type, you may encounter rejections due to high memory usage during inference processing.
These rejections will appear as an InferenceException in your cluster logs.
To resolve this issue:
- Reduce the batch size of documents in your indexing requests.
- If reducing batch size doesn't resolve the issue, then consider scaling up your machine resources.
-
Serverless
A last resort option is to adjust the
indexing_pressure.memory.coordinating.limitcluster setting. The default value is 10% of the heap. Increasing this limit allows more memory to be used for coordinating operations before rejections occur.
This adjustment should only be considered after exhausting other options, as setting this value too high may risk Out of Memory (OOM) errors in your cluster. A cluster restart is required for this change to take effect.