# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=467 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 468 --- ## [\_mget vs \_search for large amount of documents](https://discuss.elastic.co/t/mget-vs-search-for-large-amount-of-documents/339521)
**Author:** [@hattorihanzo](https://discuss.elastic.co/u/hattorihanzo)\ **Replies:** 4\ **Last updated:** [July 28, 2023, 12:40pm UTC](https://discuss.elastic.co/t/mget-vs-search-for-large-amount-of-documents/339521 "2023-07-28T12:40:39Z")
Hi there! I'm looking for some guidance around what's the most fit way to retrieve a large amount of documents (\>=1000) when you know their ID. I'd like it to be fast, yet efficient and not put unnecessary strain on ES s… --- ## [Need help dropping specific messages](https://discuss.elastic.co/t/need-help-dropping-specific-messages/339441)
**Author:** [@RJC](https://discuss.elastic.co/u/RJC)\ **Replies:** 4\ **Last updated:** [July 28, 2023, 12:31pm UTC](https://discuss.elastic.co/t/need-help-dropping-specific-messages/339441 "2023-07-28T12:31:15Z")
My logstash server generates the following messages every time it is restarted: {"syslog\_severity\_code":5,"syslog\_severity":"notice","syslog\_facility\_code":1,"message":"\\u0000\\u0016\\u0000\\u0014\\u0000\\u0017\\u0000\\u0018\\u… --- ## [Bulk indexing failed and after retrying 2 times. at Nest.BulkAllObservable\`1.\d\_\_20.MoveNext()](https://discuss.elastic.co/t/bulk-indexing-failed-and-after-retrying-2-times-at-nest-bulkallobservable-1-bulkasync-d-20-movenext/339546)
**Author:** [@stkollamp](https://discuss.elastic.co/u/stkollamp)\ **Replies:** 0\ **Last updated:** [July 28, 2023, 12:26pm UTC](https://discuss.elastic.co/t/bulk-indexing-failed-and-after-retrying-2-times-at-nest-bulkallobservable-1-bulkasync-d-20-movenext/339546 "2023-07-28T12:26:08Z")
ElastiSearch 7.x is working completely fine with 7.15.x Nest Client and then I have migrated to Elasticsearch 8.8.1 and its response has shown that "minimum\_wire\_compatibility\_version" : "7.17.0". As suggested upgrade… --- ## [Alerting on watermark threshold - Available API or field inside elasticsearch?](https://discuss.elastic.co/t/alerting-on-watermark-threshold-available-api-or-field-inside-elasticsearch/339451)
**Author:** [@chouben](https://discuss.elastic.co/u/chouben)\ **Replies:** 1\ **Last updated:** [July 28, 2023, 12:05pm UTC](https://discuss.elastic.co/t/alerting-on-watermark-threshold-available-api-or-field-inside-elasticsearch/339451 "2023-07-28T12:05:41Z")
Hi I'm looking into a way on how to achieve monitoring of the watermark thresholds. We would like to add alerting once the first threshold passes. I do not want to use fixed sizes inside the alerts, but would like to u… --- ## [Colors for line graph](https://discuss.elastic.co/t/colors-for-line-graph/339370)
**Author:** [@Rushi\_Bagul](https://discuss.elastic.co/u/Rushi_Bagul)\ **Replies:** 2\ **Last updated:** [July 28, 2023, 11:24am UTC](https://discuss.elastic.co/t/colors-for-line-graph/339370 "2023-07-28T11:24:36Z")
I want to create line chart or area with different background colors on specific areas as per following image. Can anyone tell me how it is possible? Suggest me to better approach --- ## [Ingest sharepoint on premises files to elk](https://discuss.elastic.co/t/ingest-sharepoint-on-premises-files-to-elk/339522)
**Author:** [@sunny2502](https://discuss.elastic.co/u/sunny2502)\ **Replies:** 1\ **Last updated:** [July 28, 2023, 9:54am UTC](https://discuss.elastic.co/t/ingest-sharepoint-on-premises-files-to-elk/339522 "2023-07-28T09:54:22Z")
Hi I want to ingest data from local sharepoint to elastic using python, can anyone please help me in same. --- ## [Limit on number of Clusters supported by Cross cluster search (CCS)](https://discuss.elastic.co/t/limit-on-number-of-clusters-supported-by-cross-cluster-search-ccs/339525)
**Author:** [@siddhartha\_c](https://discuss.elastic.co/u/siddhartha_c)\ **Replies:** 0\ **Last updated:** [July 28, 2023, 9:52am UTC](https://discuss.elastic.co/t/limit-on-number-of-clusters-supported-by-cross-cluster-search-ccs/339525 "2023-07-28T09:52:55Z")
Can I have a Architectural design where in we have around 100 clusters of Elastic. Where in each cluster is basically a small set of Master Node and Data Nodes . Using Cross Cluster Search if we send the search queries… --- ## [Logstash - GCP cloud storage Output plugin](https://discuss.elastic.co/t/logstash-gcp-cloud-storage-output-plugin/339524)
**Author:** [@Luko](https://discuss.elastic.co/u/Luko)\ **Replies:** 0\ **Last updated:** [July 28, 2023, 9:48am UTC](https://discuss.elastic.co/t/logstash-gcp-cloud-storage-output-plugin/339524 "2023-07-28T09:48:21Z")
Hello Does Losgatsh Output plugin - google\_cloud\_storage, support the action based on the content of the field. I want to do something like that: output { google\_cloud\_storage { bucket =\> "bucket\_name/%{… --- ## [No data Alert](https://discuss.elastic.co/t/no-data-alert/338315)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 18\ **Last updated:** [July 28, 2023, 9:27am UTC](https://discuss.elastic.co/t/no-data-alert/338315 "2023-07-28T09:27:40Z")
Hi Team, We want to create alert if no data is receiving is from last 15 minute. We have tried with the watcher it worked for us but we can't go as in watcher the alert status does not change like as in alert we have ac… --- ## [TSVB and interval issue](https://discuss.elastic.co/t/tsvb-and-interval-issue/339516)
**Author:** [@vincent2mots](https://discuss.elastic.co/u/vincent2mots)\ **Replies:** 0\ **Last updated:** [July 28, 2023, 9:14am UTC](https://discuss.elastic.co/t/tsvb-and-interval-issue/339516 "2023-07-28T09:14:20Z")
Hi there! I'm using a 7.12 version of Elasticsearch Stack and I'm struggling with TSVB. I'm using the following formula : (params.bytes \* 8) / (params.\_interval / 1000) as shown bellow : However, when I see my area… --- ## [Time difference between a field and current time](https://discuss.elastic.co/t/time-difference-between-a-field-and-current-time/339494)
**Author:** [@mathur7vidit](https://discuss.elastic.co/u/mathur7vidit)\ **Replies:** 2\ **Last updated:** [July 28, 2023, 8:37am UTC](https://discuss.elastic.co/t/time-difference-between-a-field-and-current-time/339494 "2023-07-28T08:37:49Z")
Hi Team, we have a field namely last\_modified\_date and we want to calculate aging time by noting the time difference between current time and last\_modified\_date. how can i achieve it via runtime field. --- ## [Rolling vs. Cluster Upgrades](https://discuss.elastic.co/t/rolling-vs-cluster-upgrades/339424)
**Author:** [@Tim\_Mobley](https://discuss.elastic.co/u/Tim_Mobley)\ **Replies:** 1\ **Last updated:** [July 28, 2023, 8:06am UTC](https://discuss.elastic.co/t/rolling-vs-cluster-upgrades/339424 "2023-07-28T08:06:30Z")
I'm wanting to better understand when to perform a full-cluster restart upgrade vs. a rolling upgrade of ES. I understand that a rolling upgrade has the benefit of minimizing impact to services during the upgrade (with t… --- ## [Memory consumption in io.netty.buffer.PoolThreadCache](https://discuss.elastic.co/t/memory-consumption-in-io-netty-buffer-poolthreadcache/339412)
**Author:** [@liguifa](https://discuss.elastic.co/u/liguifa)\ **Replies:** 10\ **Last updated:** [July 28, 2023, 8:02am UTC](https://discuss.elastic.co/t/memory-consumption-in-io-netty-buffer-poolthreadcache/339412 "2023-07-28T08:02:13Z")
Hi guys, I'm facing a large memory consumption in io.netty.buffer.PoolThreadCache. This portion of memory can reach up to 5GB. I think this is abnormal /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networ… --- ## ["stacktrace": \["org.apache.lucene.index.CorruptIndexException: compound sub-files must have a valid codec header and footer: file is too small (0 bytes) (resource=BufferedChecksumIndexInput)](https://discuss.elastic.co/t/stacktrace-org-apache-lucene-index-corruptindexexception-compound-sub-files-must-have-a-valid-codec-header-and-footer-file-is-too-small-0-bytes-resource-bufferedchecksumindexinput/338323)
**Author:** [@Aravindh\_M](https://discuss.elastic.co/u/Aravindh_M)\ **Replies:** 21\ **Last updated:** [July 28, 2023, 8:00am UTC](https://discuss.elastic.co/t/stacktrace-org-apache-lucene-index-corruptindexexception-compound-sub-files-must-have-a-valid-codec-header-and-footer-file-is-too-small-0-bytes-resource-bufferedchecksumindexinput/338323 "2023-07-28T08:00:43Z")
Recently, we have been encountering the "CorruptIndexException" frequently, accompanied by the following stacktrace: "org.apache.lucene.index.CorruptIndexException: compound sub-files must have a valid codec header and f… --- ## [Total count of a field](https://discuss.elastic.co/t/total-count-of-a-field/339289)
**Author:** [@Dana\_Pavaday](https://discuss.elastic.co/u/Dana_Pavaday)\ **Replies:** 6\ **Last updated:** [July 28, 2023, 7:04am UTC](https://discuss.elastic.co/t/total-count-of-a-field/339289 "2023-07-28T07:04:43Z")
Hello everyone, I need to get the total count of a field in Elasticsearch (in my case, the number of clients in client field). How do I write the query in Dev tools? --- ## [Metrics don't send after some time to Elasticsearch - Temp. bulk send fail](https://discuss.elastic.co/t/metrics-dont-send-after-some-time-to-elasticsearch-temp-bulk-send-fail/339500)
**Author:** [@Swathi12](https://discuss.elastic.co/u/Swathi12)\ **Replies:** 0\ **Last updated:** [July 28, 2023, 6:41am UTC](https://discuss.elastic.co/t/metrics-dont-send-after-some-time-to-elasticsearch-temp-bulk-send-fail/339500 "2023-07-28T06:41:58Z")
Hi, Few months ago I installed metricbeat in K8S cluster and everything was working fine. All of a sudden after 1-2 months I get error: Temporary bulk send failure - Drop batch INFO \[publisher\] pipeline/retry.go:223 … --- ## [I would like to know about the limit on the maximum number of documents per index](https://discuss.elastic.co/t/i-would-like-to-know-about-the-limit-on-the-maximum-number-of-documents-per-index/339497)
**Author:** [@bbasosuho](https://discuss.elastic.co/u/bbasosuho)\ **Replies:** 3\ **Last updated:** [July 28, 2023, 6:50am UTC](https://discuss.elastic.co/t/i-would-like-to-know-about-the-limit-on-the-maximum-number-of-documents-per-index/339497 "2023-07-28T06:50:17Z")
Hi We are going to use elasticsearch 8.7. I would like to know if there is a limit on the number of documents to be stored in one index. Is there a limit on the number of documents per index? Or is there a limit on th… --- ## [Can't see metricbeat logs](https://discuss.elastic.co/t/cant-see-metricbeat-logs/339407)
**Author:** [@Swathi12](https://discuss.elastic.co/u/Swathi12)\ **Replies:** 2\ **Last updated:** [July 28, 2023, 6:35am UTC](https://discuss.elastic.co/t/cant-see-metricbeat-logs/339407 "2023-07-28T06:35:02Z")
Hi together Do you know why i can't see since creating of metricbeat any logfiles here ? --- ## [Logstash- How to parse formatted JSON arrays in log files](https://discuss.elastic.co/t/logstash-how-to-parse-formatted-json-arrays-in-log-files/339498)
**Author:** [@fisher\_he](https://discuss.elastic.co/u/fisher_he)\ **Replies:** 0\ **Last updated:** [July 28, 2023, 6:15am UTC](https://discuss.elastic.co/t/logstash-how-to-parse-formatted-json-arrays-in-log-files/339498 "2023-07-28T06:15:33Z")
Logstash version: 7.17.10 Elasticsearch version:7.17.10 The logs are located in /var/logs directory and the format is as below: xxx.log \[ { "t": "SYS", "dt": "2023-04-17 19:46:40.147 GMT-04:00", "c": "M… --- ## [Elastic search not working](https://discuss.elastic.co/t/elastic-search-not-working/339423)
**Author:** [@gopikrish](https://discuss.elastic.co/u/gopikrish)\ **Replies:** 2\ **Last updated:** [July 28, 2023, 6:06am UTC](https://discuss.elastic.co/t/elastic-search-not-working/339423 "2023-07-28T06:06:52Z")
Hi Team, Suddenly, my Elasticsearch went down, and I'm not able to access it on remote systems or other systems with the same network band. I'm able to access it on my system only using my IP Address. For reference, I'… --- ## [Performance impact of upsert vs index with custom id](https://discuss.elastic.co/t/performance-impact-of-upsert-vs-index-with-custom-id/339492)
**Author:** [@sriapr98](https://discuss.elastic.co/u/sriapr98)\ **Replies:** 0\ **Last updated:** [July 28, 2023, 4:49am UTC](https://discuss.elastic.co/t/performance-impact-of-upsert-vs-index-with-custom-id/339492 "2023-07-28T04:49:47Z")
We have a huge traffic coming out of kafka which we are continuously ingesting to es using Index api(with custom doc id). Due to some edge cases we are thinking of moving to update api(upsert with doc\_as\_upsert with upd… --- ## [Data relation using logstash conf file](https://discuss.elastic.co/t/data-relation-using-logstash-conf-file/338925)
**Author:** [@Vinod\_Kumar2](https://discuss.elastic.co/u/Vinod_Kumar2)\ **Replies:** 6\ **Last updated:** [July 28, 2023, 4:20am UTC](https://discuss.elastic.co/t/data-relation-using-logstash-conf-file/338925 "2023-07-28T04:20:37Z")
Multiple csv files in data folder and one column in common to relate the data between files. created logstash conf file and running manually and logstash gets shutdown. Sample data: File1:sample\_orders.csv id,product… --- ## [Failed to flush the buffer ||Data too large, data for|| could not push logs to Elasticsearch cluster](https://discuss.elastic.co/t/failed-to-flush-the-buffer-data-too-large-data-for-could-not-push-logs-to-elasticsearch-cluster/339489)
**Author:** [@Vikas1633](https://discuss.elastic.co/u/Vikas1633)\ **Replies:** 0\ **Last updated:** [July 28, 2023, 4:08am UTC](https://discuss.elastic.co/t/failed-to-flush-the-buffer-data-too-large-data-for-could-not-push-logs-to-elasticsearch-cluster/339489 "2023-07-28T04:08:46Z")
Hi all i am facing this issue since long and not able to control the buffer file whenever large amount of data is ingested its get chocked any help would be apprecitated. 2023-05-24 17:34:11 +0300 \[warn\]: #0 failed to f… --- ## [Filebeat set add\_id: ~ does not take effect](https://discuss.elastic.co/t/filebeat-set-add-id-does-not-take-effect/336825)
**Author:** [@yt\_h](https://discuss.elastic.co/u/yt_h)\ **Replies:** 17\ **Last updated:** [July 28, 2023, 3:19am UTC](https://discuss.elastic.co/t/filebeat-set-add-id-does-not-take-effect/336825 "2023-07-28T03:19:49Z")
I have a filebeat 7.16.2 to extract messages in kafka 3.4. After setting add\_id: ~, restarting filebeat will repeatedly send data to elasticsearch. filebeat.yml: filebeat.yml: | filebeat.inputs: - type: kafka h… --- ## [Logstash HTTP client](https://discuss.elastic.co/t/logstash-http-client/339483)
**Author:** [@Arjun\_Nambiar](https://discuss.elastic.co/u/Arjun_Nambiar)\ **Replies:** 1\ **Last updated:** [July 27, 2023, 11:55pm UTC](https://discuss.elastic.co/t/logstash-http-client/339483 "2023-07-27T23:55:08Z")
I have an Elasticsearch cluster running on AWS which has an Elastic Load balancer(ELB) in front of it. I am scanning the ELB log file to find the clients writing to the Elasticsearch cluster. Logstash is also one of the … --- ## [Help with logstash output](https://discuss.elastic.co/t/help-with-logstash-output/339469)
**Author:** [@nach\_usal](https://discuss.elastic.co/u/nach_usal)\ **Replies:** 5\ **Last updated:** [July 27, 2023, 9:45pm UTC](https://discuss.elastic.co/t/help-with-logstash-output/339469 "2023-07-27T21:45:28Z")
Hi guys, I am trying to capture login and logout events in programs such as TeamViewer and AnyDesk, installed in a Windows virtual machine. Then I send them to my Logstash server via the same Filebeat node, here is the … --- ## [Controls showing incorrect document count](https://discuss.elastic.co/t/controls-showing-incorrect-document-count/339477)
**Author:** [@Matthew\_Salah](https://discuss.elastic.co/u/Matthew_Salah)\ **Replies:** 1\ **Last updated:** [July 27, 2023, 7:11pm UTC](https://discuss.elastic.co/t/controls-showing-incorrect-document-count/339477 "2023-07-27T19:11:59Z")
I have a map on a Kibana dashboard and a series of controls. The controls have a count of documents associated with the filter value. However, the counts there don't line up with my global count of documents (see upper … --- ## [If index does not exists does not show error, return empty](https://discuss.elastic.co/t/if-index-does-not-exists-does-not-show-error-return-empty/339343)
**Author:** [@Murilo\_Livorato](https://discuss.elastic.co/u/Murilo_Livorato)\ **Replies:** 2\ **Last updated:** [July 27, 2023, 6:54pm UTC](https://discuss.elastic.co/t/if-index-does-not-exists-does-not-show-error-return-empty/339343 "2023-07-27T18:54:22Z")
hello , I have a search with many indexes here - GET index-0001, index-0002, index-0003/\_search { "size": 30, "query": { "match\_all": {} } } index-0001 exists index-0002 , does not exists . index-0003 exist… --- ## [Split filter and add\_field encoding object to a JSON string](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590)
**Author:** [@MrOg](https://discuss.elastic.co/u/MrOg)\ **Replies:** 7\ **Last updated:** [July 27, 2023, 4:46pm UTC](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590 "2023-07-27T16:46:34Z")
Hi, I have such a set of filters filter { json { source =\> "\[sql\_data\]\[response\]" } split { field =\> "docs" add\_field =\> { "id" =\> "%{\[docs\]\[id\]}" "names" =\> … --- ## [How to use pagination with new Java Client](https://discuss.elastic.co/t/how-to-use-pagination-with-new-java-client/339460)
**Author:** [@Andre\_Schmer](https://discuss.elastic.co/u/Andre_Schmer)\ **Replies:** 0\ **Last updated:** [July 27, 2023, 4:42pm UTC](https://discuss.elastic.co/t/how-to-use-pagination-with-new-java-client/339460 "2023-07-27T16:42:24Z")
Trying to use a search\_after with the new Java API. That s what i did so far: SearchResponse\ sr = newClient.search(searchRequest, PeriodInventory.class); List\\> searchHits = sr.h… [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=466) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=468)