# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1978 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1979 --- ## [Elasticsearch cluster performance tuning](https://discuss.elastic.co/t/elasticsearch-cluster-performance-tuning/236812)
**Author:** [@AClerk](https://discuss.elastic.co/u/AClerk)\ **Replies:** 11\ **Last updated:** [June 24, 2020, 3:54am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-performance-tuning/236812 "2020-06-24T03:54:14Z")
Hi, Following an error I got in Kibana and the following posts How can I optimise and tune the elastic cluster? Elasticsearch/Kibana version are 7.6.2 I have 5 nodes in the cluster, all run on a CentOS 7 VMs 28G … --- ## [Shard allocation - primaries allocated on same data nodes](https://discuss.elastic.co/t/shard-allocation-primaries-allocated-on-same-data-nodes/238338)
**Author:** [@andrezenun](https://discuss.elastic.co/u/andrezenun)\ **Replies:** 4\ **Last updated:** [June 24, 2020, 3:42am UTC](https://discuss.elastic.co/t/shard-allocation-primaries-allocated-on-same-data-nodes/238338 "2020-06-24T03:42:23Z")
Hello everyone! I have a cluster, that I'm on the process of setup everything to go to production. I have six data nodes and 3 masters. I don't know why but for some reason that I cannot identify the shards are allocate… --- ## [Spark rdd save to elasticsearch with http signing](https://discuss.elastic.co/t/spark-rdd-save-to-elasticsearch-with-http-signing/238367)
**Author:** [@kdhccb](https://discuss.elastic.co/u/kdhccb)\ **Replies:** 0\ **Last updated:** [June 24, 2020, 3:12am UTC](https://discuss.elastic.co/t/spark-rdd-save-to-elasticsearch-with-http-signing/238367 "2020-06-24T03:12:26Z")
I would like to add a document to elasticsearch with the spark application. However, because http signature authentication cannot be added in the spark application, the document cannot be added due to permission issues. … --- ## [ES 7.8 ILM shrink action does not allocate primaries](https://discuss.elastic.co/t/es-7-8-ilm-shrink-action-does-not-allocate-primaries/238155)
**Author:** [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)\ **Replies:** 1\ **Last updated:** [June 24, 2020, 3:06am UTC](https://discuss.elastic.co/t/es-7-8-ilm-shrink-action-does-not-allocate-primaries/238155 "2020-06-24T03:06:30Z")
The ILM shrink action seems to be unreliable when trying to allocate primary shards to the same node because it stuck at this step check-shrink-allocation step with the message: Waiting for node \[XYZ\] to contain \[3\] shar… --- ## [Logstash-plain.log stop recording and logstash failed to start](https://discuss.elastic.co/t/logstash-plain-log-stop-recording-and-logstash-failed-to-start/238201)
**Author:** [@kase](https://discuss.elastic.co/u/kase)\ **Replies:** 1\ **Last updated:** [June 24, 2020, 3:06am UTC](https://discuss.elastic.co/t/logstash-plain-log-stop-recording-and-logstash-failed-to-start/238201 "2020-06-24T03:06:29Z")
Hi there, I'm using logstash to filter netflow data to save to elasticsearch, it works fine but I also want to filter data in elasticsearch to save to redis, so I wrote a file to do logstash pipeline. Here is my file i… --- ## [Logstash Google Cloud Storage Output Plugin Error Msg](https://discuss.elastic.co/t/logstash-google-cloud-storage-output-plugin-error-msg/238355)
**Author:** [@daniel\_a](https://discuss.elastic.co/u/daniel_a)\ **Replies:** 0\ **Last updated:** [June 24, 2020, 1:42am UTC](https://discuss.elastic.co/t/logstash-google-cloud-storage-output-plugin-error-msg/238355 "2020-06-24T01:42:51Z")
I'm trying to set up the logstash-output-google\_cloud\_storage plugin with a very simple configuration set. The input works just fine, I tested it writing to a local file. output { google\_cloud\_storage { bucket… --- ## [Have to export the log file ending with \_debug (log file name)](https://discuss.elastic.co/t/have-to-export-the-log-file-ending-with-debug-log-file-name/238300)
**Author:** [@nikhilesh](https://discuss.elastic.co/u/nikhilesh)\ **Replies:** 1\ **Last updated:** [June 24, 2020, 1:34am UTC](https://discuss.elastic.co/t/have-to-export-the-log-file-ending-with-debug-log-file-name/238300 "2020-06-24T01:34:49Z")
Hi Team, Can you please help on this requirement. I have a log files with naming "date\_action" and "date\_debug" , so in file beat input configuration we have to include "date\_debug" log file and to exclude "date\_acti… --- ## [Couldn't connect to any of the configured Elasticsearch hosts](https://discuss.elastic.co/t/couldnt-connect-to-any-of-the-configured-elasticsearch-hosts/238166)
**Author:** [@ibadawe](https://discuss.elastic.co/u/ibadawe)\ **Replies:** 5\ **Last updated:** [June 24, 2020, 1:06am UTC](https://discuss.elastic.co/t/couldnt-connect-to-any-of-the-configured-elasticsearch-hosts/238166 "2020-06-24T01:06:16Z")
Hello, i have configured all settings in configuration file at winlogbeat.yml, to send logs to Elastic. but unfortunately i couldn't receive any logs at Elastic and when i run the below command in Powershell ".\\winlogb… --- ## [\[WARN \]\[org.logstash.common.io.DeadLetterQueueWriter\] Event previously submitted to dead letter queue. Skipping](https://discuss.elastic.co/t/warn-org-logstash-common-io-deadletterqueuewriter-event-previously-submitted-to-dead-letter-queue-skipping/113250)
**Author:** [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Replies:** 2\ **Last updated:** [June 23, 2020, 11:07pm UTC](https://discuss.elastic.co/t/warn-org-logstash-common-io-deadletterqueuewriter-event-previously-submitted-to-dead-letter-queue-skipping/113250 "2020-06-23T23:07:03Z")
I just started working with the Dead Letter Queue, but after restarting my pipelines a few times I am now getting this message repeatedly in my logs: \[WARN \]\[org.logstash.common.io.DeadLetterQueueWriter\] Event previousl… --- ## [Grant privileges to specific documents, Kibana Role](https://discuss.elastic.co/t/grant-privileges-to-specific-documents-kibana-role/238000)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 1\ **Last updated:** [June 23, 2020, 11:01pm UTC](https://discuss.elastic.co/t/grant-privileges-to-specific-documents-kibana-role/238000 "2020-06-23T23:01:43Z")
Hi Im currently using this query for privileges to specifics documents { "terms":{ "host.keyword":\["meli","cmai","moja"\] } } But now I need to grant privileges to the documents that has a specific field "metrics" along… --- ## [How to create a Watcher for no ingested logs for X minutes](https://discuss.elastic.co/t/how-to-create-a-watcher-for-no-ingested-logs-for-x-minutes/237842)
**Author:** [@ckough](https://discuss.elastic.co/u/ckough)\ **Replies:** 1\ **Last updated:** [June 23, 2020, 11:00pm UTC](https://discuss.elastic.co/t/how-to-create-a-watcher-for-no-ingested-logs-for-x-minutes/237842 "2020-06-23T23:00:10Z")
Hi There, I'm using ELK 7.6. I have a question regarding watchers. Is it possible to create a watcher that sends an alert when it detects that there are no new documents or indexes being created for the past, say 30, m… --- ## [Flow Search Data MDM via Elasticsearch](https://discuss.elastic.co/t/flow-search-data-mdm-via-elasticsearch/238224)
**Author:** [@alf1an](https://discuss.elastic.co/u/alf1an)\ **Replies:** 3\ **Last updated:** [June 23, 2020, 10:57pm UTC](https://discuss.elastic.co/t/flow-search-data-mdm-via-elasticsearch/238224 "2020-06-23T22:57:26Z")
Hi All, I have a project to integrate Web Portal, Elasticsearch and IBM MDM. I already create the flow about it, but is it correct flow? Big appreciate for your thought and recommendation Thanks --- ## [How to Set up a cluster for high availability](https://discuss.elastic.co/t/how-to-set-up-a-cluster-for-high-availability/237978)
**Author:** [@emreorucoglu](https://discuss.elastic.co/u/emreorucoglu)\ **Replies:** 1\ **Last updated:** [June 23, 2020, 10:48pm UTC](https://discuss.elastic.co/t/how-to-set-up-a-cluster-for-high-availability/237978 "2020-06-23T22:48:13Z")
Hello there, I am trying to establish a remote cluster connection for high usability between two remote machines. However, although I can access the Elastic Search port, I cannot connect to the remote cluster. Can you h… --- ## [Resources usage](https://discuss.elastic.co/t/resources-usage/238333)
**Author:** [@Elastan](https://discuss.elastic.co/u/Elastan)\ **Replies:** 6\ **Last updated:** [June 23, 2020, 10:39pm UTC](https://discuss.elastic.co/t/resources-usage/238333 "2020-06-23T22:39:59Z")
Hello, I'm newbie to Elastic. Assuming index size will have 500GB, let's say 100 users each 5GB of index data. What hardware is needed for that? If querying/searching will be only performed always on about 1/100 of wh… --- ## [Failed to execute index. number of documents in the index cannot exceed 2147483519](https://discuss.elastic.co/t/failed-to-execute-index-number-of-documents-in-the-index-cannot-exceed-2147483519/238124)
**Author:** [@DucTran](https://discuss.elastic.co/u/DucTran)\ **Replies:** 1\ **Last updated:** [June 23, 2020, 10:43pm UTC](https://discuss.elastic.co/t/failed-to-execute-index-number-of-documents-in-the-index-cannot-exceed-2147483519/238124 "2020-06-23T22:43:31Z")
Hi Guys, I am having problems with the capacity of ElasticSearch as below. I'm understanding the number od docs per shard reached to maximum number - 2147483519. I deleted many docs but the error continues, Elasticsear… --- ## [Moving from tribe to cross cluster search](https://discuss.elastic.co/t/moving-from-tribe-to-cross-cluster-search/238238)
**Author:** [@dextermorgan](https://discuss.elastic.co/u/dextermorgan)\ **Replies:** 1\ **Last updated:** [June 23, 2020, 10:27pm UTC](https://discuss.elastic.co/t/moving-from-tribe-to-cross-cluster-search/238238 "2020-06-23T22:27:09Z")
Hi All, We have an infra where we have around 10 big clusters behind tribe nodes and all of them are on ES 5.4 and data is time sharded and ever growing. We need to move our infra to 7.8 (priority is to have live data c… --- ## [How to create patterns\_dir in elastic logstash helm chart?](https://discuss.elastic.co/t/how-to-create-patterns-dir-in-elastic-logstash-helm-chart/238346)
**Author:** [@Phillip\_Boushy](https://discuss.elastic.co/u/Phillip_Boushy)\ **Replies:** 0\ **Last updated:** [June 23, 2020, 9:53pm UTC](https://discuss.elastic.co/t/how-to-create-patterns-dir-in-elastic-logstash-helm-chart/238346 "2020-06-23T21:53:39Z")
I'm deploy the elastic.co logstash helm chart 7.7.1 to kubernetes. I'm using a ConfigMap to create an example file in /usr/share/logstash/pipeline/patterns/ But it seems like it's not being read. I keep getting an err… --- ## [Does utf8json used in elastic v7.x support user defined custom formatters?](https://discuss.elastic.co/t/does-utf8json-used-in-elastic-v7-x-support-user-defined-custom-formatters/237283)
**Author:** [@stoj](https://discuss.elastic.co/u/stoj)\ **Replies:** 5\ **Last updated:** [June 23, 2020, 9:43pm UTC](https://discuss.elastic.co/t/does-utf8json-used-in-elastic-v7-x-support-user-defined-custom-formatters/237283 "2020-06-23T21:43:29Z")
I've got an interesting situation whereby elastic v7 default utf8json has managed to serialize my object, but is unable to de-serialize it correctly. public class MyClass : FlagsSet { \[JsonIgnore\] … --- ## [Writing to Elasticsearch from Spark failing](https://discuss.elastic.co/t/writing-to-elasticsearch-from-spark-failing/238344)
**Author:** [@Biswajit.kundu](https://discuss.elastic.co/u/Biswajit.kundu)\ **Replies:** 0\ **Last updated:** [June 23, 2020, 9:33pm UTC](https://discuss.elastic.co/t/writing-to-elasticsearch-from-spark-failing/238344 "2020-06-23T21:33:36Z")
Hi, Am trying to write to elastic search from Spark running on my local. I have a data frame df and am doing this df.write.format("org.elasticsearch.spark.sql").option("es.nodes","abc-es.xyz.com").option("es.port","920… --- ## [Kibana cache the results of geo\_shape filter](https://discuss.elastic.co/t/kibana-cache-the-results-of-geo-shape-filter/237549)
**Author:** [@Scorpion21](https://discuss.elastic.co/u/Scorpion21)\ **Replies:** 2\ **Last updated:** [June 23, 2020, 9:16pm UTC](https://discuss.elastic.co/t/kibana-cache-the-results-of-geo-shape-filter/237549 "2020-06-23T21:16:43Z")
I have created a Kibana Dashboard with elastic maps showing thousands of geo\_shape (multi-polygons). It takes too long to query the Dashboard when applying filters. I wonder if there is a way to cache the geo\_shape fil… --- ## [Query Hit limit](https://discuss.elastic.co/t/query-hit-limit/238336)
**Author:** [@th3Hopper](https://discuss.elastic.co/u/th3Hopper)\ **Replies:** 1\ **Last updated:** [June 23, 2020, 9:05pm UTC](https://discuss.elastic.co/t/query-hit-limit/238336 "2020-06-23T21:05:37Z")
I have index 2M document. If I query GET indexName/\_search?q=email:info@yahoo.com It's giving me this result:- "took" : 12, "timed\_out" : false, "\_shards" : { "total" : 1, "successful" : 1, "skipped" … --- ## [Winlogbeat missing events](https://discuss.elastic.co/t/winlogbeat-missing-events/237961)
**Author:** [@craigothy](https://discuss.elastic.co/u/craigothy)\ **Replies:** 2\ **Last updated:** [June 23, 2020, 9:04pm UTC](https://discuss.elastic.co/t/winlogbeat-missing-events/237961 "2020-06-23T21:04:36Z")
Can someone tell me the best way to troubleshoot missing windows events from winlogbeat in elastic. We are using WEC to get events to a central collector. That collector is running winlogbeat to send to logstash and then… --- ## [Find exact result](https://discuss.elastic.co/t/find-exact-result/238339)
**Author:** [@th3Hopper](https://discuss.elastic.co/u/th3Hopper)\ **Replies:** 1\ **Last updated:** [June 23, 2020, 9:03pm UTC](https://discuss.elastic.co/t/find-exact-result/238339 "2020-06-23T21:03:15Z")
My goal is to find the exact match result from all of my documents. How can I achieve this? I have found 3 ways to achieve it but I'm confused about which one will be more effective. ### Exact search 1 GET email/\_searc… --- ## [Filebeat's default index mapping causes Kibana to display many empty date fields](https://discuss.elastic.co/t/filebeats-default-index-mapping-causes-kibana-to-display-many-empty-date-fields/238324)
**Author:** [@temp4746](https://discuss.elastic.co/u/temp4746)\ **Replies:** 2\ **Last updated:** [June 23, 2020, 8:42pm UTC](https://discuss.elastic.co/t/filebeats-default-index-mapping-causes-kibana-to-display-many-empty-date-fields/238324 "2020-06-23T20:42:32Z")
I tried a default dev setup of the ELK stack with Filebeat reading from some Docker container outputting Elastic Common Schema JSON. It looks like the default index mapping that Filebeat sets up causes Kibana to show man… --- ## [How to see Long running apache requests from apache\_access logs in KIbana?](https://discuss.elastic.co/t/how-to-see-long-running-apache-requests-from-apache-access-logs-in-kibana/235082)
**Author:** [@marco2005](https://discuss.elastic.co/u/marco2005)\ **Replies:** 1\ **Last updated:** [June 23, 2020, 8:36pm UTC](https://discuss.elastic.co/t/how-to-see-long-running-apache-requests-from-apache-access-logs-in-kibana/235082 "2020-06-23T20:36:31Z")
Is there any visualization with which I can achieve this scenario of finding long running Apache requests ? I have parsed the Apache access log successfully via Logstash to Kibana. Now, I was looking in Kibana how to ac… --- ## [ELK - SO Security correlations](https://discuss.elastic.co/t/elk-so-security-correlations/237313)
**Author:** [@YASSER\_MEMO](https://discuss.elastic.co/u/YASSER_MEMO)\ **Replies:** 1\ **Last updated:** [June 23, 2020, 8:34pm UTC](https://discuss.elastic.co/t/elk-so-security-correlations/237313 "2020-06-23T20:34:57Z")
Hey guys, I installed Security onion with ELK to monitor enterprise network. I want to send Cisco Firewall logs , IPS, F5 and antivirus logs to logstash. I've already sent them however, I'm facing two issues. 1- non of… --- ## [Kibana doesn't show graphics for netflow index](https://discuss.elastic.co/t/kibana-doesnt-show-graphics-for-netflow-index/235010)
**Author:** [@Adrian\_Garrido](https://discuss.elastic.co/u/Adrian_Garrido)\ **Replies:** 1\ **Last updated:** [June 23, 2020, 8:33pm UTC](https://discuss.elastic.co/t/kibana-doesnt-show-graphics-for-netflow-index/235010 "2020-06-23T20:33:24Z")
Hi. I ingested the traffic from a pcap with softflowd when logstash was listening that port, and a index was created in elasticsearch, the problem is kibanas doesn't show any dashboard or chart but it detects the index … --- ## [How to get max of 10minute Moving Average](https://discuss.elastic.co/t/how-to-get-max-of-10minute-moving-average/234886)
**Author:** [@Bhavya\_Bansal](https://discuss.elastic.co/u/Bhavya_Bansal)\ **Replies:** 1\ **Last updated:** [June 23, 2020, 8:31pm UTC](https://discuss.elastic.co/t/how-to-get-max-of-10minute-moving-average/234886 "2020-06-23T20:31:45Z")
Hi, I have a data table in which I want to show the max of 10min and 1hr moving average of the CPU in a day. But in order to calculate 10m moving average, I have to set the minimum time interval of my data histogram buc… --- ## [Grok filter extracting fields from message](https://discuss.elastic.co/t/grok-filter-extracting-fields-from-message/238298)
**Author:** [@heinrich](https://discuss.elastic.co/u/heinrich)\ **Replies:** 8\ **Last updated:** [June 23, 2020, 8:21pm UTC](https://discuss.elastic.co/t/grok-filter-extracting-fields-from-message/238298 "2020-06-23T20:21:41Z")
Hi guys, I've never used grok patterns before, after hours of looking into this i'm making very slow progress. Here's a sample of what i have. 2020-06-23 15:30:02.568 \[https-jsse-nio-0.0.0.0-8443-exec-58\] INFO c.c.d.… --- ## ["condition" part example](https://discuss.elastic.co/t/condition-part-example/234613)
**Author:** [@elify](https://discuss.elastic.co/u/elify)\ **Replies:** 1\ **Last updated:** [June 23, 2020, 7:33pm UTC](https://discuss.elastic.co/t/condition-part-example/234613 "2020-06-23T19:33:29Z")
Hello, I am pretty new using Watcher. I saw some examples but I couldn't find what I wanted. I have query like: layers.arp.arp\_arp\_src\_hw\_mac:"00:b6:44:11:3a:61" or \_index I want to generate an alarm every time this … [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1977) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1979)