# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1709 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1710 --- ## [Elasticsearch memory is not completely utilized](https://discuss.elastic.co/t/elasticsearch-memory-is-not-completely-utilized/253380)
**Author:** [@Varun\_S](https://discuss.elastic.co/u/Varun_S)\ **Replies:** 5\ **Last updated:** [November 9, 2020, 8:42am UTC](https://discuss.elastic.co/t/elasticsearch-memory-is-not-completely-utilized/253380 "2020-11-09T08:42:24Z")
Hi Team, I have deployed elasticsearch in k8s environment with below configuration. Es-client:- replicas: 1 resources: limits: cpu: 2 memory: "4Gi" requests: cpu: "500m" memory: "2Gi" es\_java\_opts: "-Xms2g -X… --- ## [Index filebeat-7.8.1 gets created automatically](https://discuss.elastic.co/t/index-filebeat-7-8-1-gets-created-automatically/254227)
**Author:** [@Anirbaan\_Chowdhury](https://discuss.elastic.co/u/Anirbaan_Chowdhury)\ **Replies:** 4\ **Last updated:** [November 9, 2020, 8:34am UTC](https://discuss.elastic.co/t/index-filebeat-7-8-1-gets-created-automatically/254227 "2020-11-09T08:34:34Z")
Hi Experts, I am sending logs from various instances to my ES cluster using filebeat. I can see that everytime I start sending data, a new index "filebeat-7.8.1" gets created which does not comply with the index templa… --- ## [Filebeat failed to insert too much log](https://discuss.elastic.co/t/filebeat-failed-to-insert-too-much-log/254607)
**Author:** [@Mohammad\_Mousavi](https://discuss.elastic.co/u/Mohammad_Mousavi)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 8:28am UTC](https://discuss.elastic.co/t/filebeat-failed-to-insert-too-much-log/254607 "2020-11-09T08:28:04Z")
Hi, It's look like my filebeats agents are failed to insert some logs into elasticsearch. I have multiple filebeats and 3 logstash and 3 clustered elasticsearch (1 shard , 2 replica). I see some data exists in nginx lo… --- ## [In logstash, How can we check if a data exists another elastic search index?](https://discuss.elastic.co/t/in-logstash-how-can-we-check-if-a-data-exists-another-elastic-search-index/254709)
**Author:** [@meghavinam](https://discuss.elastic.co/u/meghavinam)\ **Replies:** 4\ **Last updated:** [November 9, 2020, 8:26am UTC](https://discuss.elastic.co/t/in-logstash-how-can-we-check-if-a-data-exists-another-elastic-search-index/254709 "2020-11-09T08:26:47Z")
I want to sync set of data from mysql to elastic search. That time i need to remove if same data exists in another index --- ## [Json file from logtash using fast API](https://discuss.elastic.co/t/json-file-from-logtash-using-fast-api/254716)
**Author:** [@abhishek1310](https://discuss.elastic.co/u/abhishek1310)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 8:18am UTC](https://discuss.elastic.co/t/json-file-from-logtash-using-fast-api/254716 "2020-11-09T08:18:42Z")
Can any one suggest me the approach to receive a json file through fast API from logtash --- ## [Logstash config error Expected one of \[ \\\\t\\\\r\\\\n\], \\"#\\", \\"{\\", \\",\\", \\"\]\\" at line 10, column 83](https://discuss.elastic.co/t/logstash-config-error-expected-one-of-t-r-n-at-line-10-column-83/254619)
**Author:** [@Tejas\_Bhosale](https://discuss.elastic.co/u/Tejas_Bhosale)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 7:39am UTC](https://discuss.elastic.co/t/logstash-config-error-expected-one-of-t-r-n-at-line-10-column-83/254619 "2020-11-09T07:39:52Z")
Hello, I am using Logstash to ingest nginx into ElasticSearch. I am using a Linux Machine but i am getting error when try to start logstatsh service elk logstash\[1233\]: \[2020-11-07T20:25:03,741\]\[ERROR\]\[logstash.agent … --- ## [How to display Due, Over Due and Not Due based on a date field in Saved Search](https://discuss.elastic.co/t/how-to-display-due-over-due-and-not-due-based-on-a-date-field-in-saved-search/254648)
**Author:** [@bkmagnetron](https://discuss.elastic.co/u/bkmagnetron)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 7:37am UTC](https://discuss.elastic.co/t/how-to-display-due-over-due-and-not-due-based-on-a-date-field-in-saved-search/254648 "2020-11-09T07:37:12Z")
For example I have a date field delivery\_datetime in Index, I have to show the user for current day whether a particular parcel is Due today or Over Due or Not Due I can't create a separate field and do reindex because … --- ## [Errors io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException: newPosition \> limit](https://discuss.elastic.co/t/errors-io-netty-handler-codec-decoderexception-java-lang-illegalargumentexception-newposition-limit/254711)
**Author:** [@sreekanth](https://discuss.elastic.co/u/sreekanth)\ **Replies:** 3\ **Last updated:** [November 9, 2020, 7:33am UTC](https://discuss.elastic.co/t/errors-io-netty-handler-codec-decoderexception-java-lang-illegalargumentexception-newposition-limit/254711 "2020-11-09T07:33:45Z")
ES Version: 6.8.6 Lucene Version: 7.7.2 We have a 3 nodes cluster where all 3 nodes are marked as "mdi". When we loop on /\_cat/nodes, we saw one of the node randomly go out of cluster and entire cluster is in unstable… --- ## [Node does not cluster setting \[cluster.routing.allocation.include\] filters](https://discuss.elastic.co/t/node-does-not-cluster-setting-cluster-routing-allocation-include-filters/254671)
**Author:** [@ludaca](https://discuss.elastic.co/u/ludaca)\ **Replies:** 3\ **Last updated:** [November 9, 2020, 7:31am UTC](https://discuss.elastic.co/t/node-does-not-cluster-setting-cluster-routing-allocation-include-filters/254671 "2020-11-09T07:31:42Z")
Hello, ES version 5.6.10 16 data nodes 3 master nodes Can't perform any shard's relocation: Error while starting relocation { "error": { "root\_cause": \[ { "type": "remote\_transport\_exception", … --- ## [How do I process an array with enrich processors](https://discuss.elastic.co/t/how-do-i-process-an-array-with-enrich-processors/254713)
**Author:** [@yhseo](https://discuss.elastic.co/u/yhseo)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 7:03am UTC](https://discuss.elastic.co/t/how-do-i-process-an-array-with-enrich-processors/254713 "2020-11-09T07:03:50Z")
The current work is to look at the tags of the original text whenever the original text comes in, and when a matching word comes in, we are trying to apply the classification system of the above source index. I am trying… --- ## ["type":"parse\_exception","reason":"request body is required"},"status":400}](https://discuss.elastic.co/t/type-parse-exception-reason-request-body-is-required-status-400/254664)
**Author:** [@PADMAVATHI\_GUNTURU](https://discuss.elastic.co/u/PADMAVATHI_GUNTURU)\ **Replies:** 4\ **Last updated:** [November 9, 2020, 7:01am UTC](https://discuss.elastic.co/t/type-parse-exception-reason-request-body-is-required-status-400/254664 "2020-11-09T07:01:18Z")
When using the \_bulk API, curl command, I'm seeing this error. I have given 3 JSON files. 2 files were worked, one file is thrown this error. I don't understand the reason here. The 3rd file input data is like the below … --- ## [How to make a automaticaly renew token authorization in file logstash configuration](https://discuss.elastic.co/t/how-to-make-a-automaticaly-renew-token-authorization-in-file-logstash-configuration/254712)
**Author:** [@salma\_widiarti](https://discuss.elastic.co/u/salma_widiarti)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 6:46am UTC](https://discuss.elastic.co/t/how-to-make-a-automaticaly-renew-token-authorization-in-file-logstash-configuration/254712 "2020-11-09T06:46:38Z")
Hi all, Do you know how to make a automaticaly renew token authorization in file logstash configuration? Thanks.. --- ## [When ECSLayout is default for Elasticsearch and Kibana log configuration?](https://discuss.elastic.co/t/when-ecslayout-is-default-for-elasticsearch-and-kibana-log-configuration/254710)
**Author:** [@Bingu\_Shim](https://discuss.elastic.co/u/Bingu_Shim)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 6:40am UTC](https://discuss.elastic.co/t/when-ecslayout-is-default-for-elasticsearch-and-kibana-log-configuration/254710 "2020-11-09T06:40:05Z")
Hi, I'm trying to change the log format of Elasticsearch and Kibana to ECS format The Current Document guided using ESJsonLayout, which is different fro ECS format. As far as I know ECS is the standard format for Elas… --- ## [Creating Single URL for multiple Kibana URL](https://discuss.elastic.co/t/creating-single-url-for-multiple-kibana-url/254524)
**Author:** [@ksaha](https://discuss.elastic.co/u/ksaha)\ **Replies:** 4\ **Last updated:** [November 9, 2020, 5:57am UTC](https://discuss.elastic.co/t/creating-single-url-for-multiple-kibana-url/254524 "2020-11-09T05:57:26Z")
Hi, I have a ELK stack with 2 Logstash server, 3 node Elasticsearch cluster and 2 Kibana server (considering High Availability). Due to two Kibana instance we have two Kibana URL. Both the Kibana URL is working. Can I h… --- ## [Kibana Visualisation Doubts-Conditional](https://discuss.elastic.co/t/kibana-visualisation-doubts-conditional/254438)
**Author:** [@Shubhamrai7](https://discuss.elastic.co/u/Shubhamrai7)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 5:30am UTC](https://discuss.elastic.co/t/kibana-visualisation-doubts-conditional/254438 "2020-11-09T05:30:32Z")
Need to Find Availability of APIs- I am new to Kibana . I am stuck with one visualisation. What I have to do is ,first find the failure rate of APIs( Count Http code\[500 to 599\] divided by /All the count of APIs) for p… --- ## [Filebeat and Data stream](https://discuss.elastic.co/t/filebeat-and-data-stream/253841)
**Author:** [@HelpComputer](https://discuss.elastic.co/u/HelpComputer)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 5:10am UTC](https://discuss.elastic.co/t/filebeat-and-data-stream/253841 "2020-11-09T05:10:13Z")
Hello, What would be the best way to get the Filebeat legacy template converted into a data stream template or some other ILM? It seems like data streams are recommended based on this guide - https://www.elastic.co/guid… --- ## [Filebeat using Fortinet module TZ issue](https://discuss.elastic.co/t/filebeat-using-fortinet-module-tz-issue/251518)
**Author:** [@Derick\_Jansen](https://discuss.elastic.co/u/Derick_Jansen)\ **Replies:** 11\ **Last updated:** [November 9, 2020, 4:54am UTC](https://discuss.elastic.co/t/filebeat-using-fortinet-module-tz-issue/251518 "2020-11-09T04:54:51Z")
Hi, We are using Filebeat with the Fortinet module to ingest our FW logs via Syslog. Our Fortigates do not send timezone information in log entries. The logs all have this format \<189\>date=2020-10-09 time=14:06:56 de… --- ## [Scripted Field Unique Count of Sum Value](https://discuss.elastic.co/t/scripted-field-unique-count-of-sum-value/254215)
**Author:** [@111207](https://discuss.elastic.co/u/111207)\ **Replies:** 8\ **Last updated:** [November 9, 2020, 3:19am UTC](https://discuss.elastic.co/t/scripted-field-unique-count-of-sum-value/254215 "2020-11-09T03:19:48Z")
I need to create visualization metric for count of ID of value is 0 I have imported data to elasticsearch And the data in each ID has multiple lines, I would like to sum value by ID. Then Unique Count is a number whose … --- ## [\_default\_ mapping of template is not compatible with index sorting](https://discuss.elastic.co/t/default-mapping-of-template-is-not-compatible-with-index-sorting/251735)
**Author:** [@hackerwin7](https://discuss.elastic.co/u/hackerwin7)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 3:11am UTC](https://discuss.elastic.co/t/default-mapping-of-template-is-not-compatible-with-index-sorting/251735 "2020-11-09T03:11:31Z")
version: 6.8.0 we create a \_default\_ mapping of index template with the index sorting settings. when we put data to create a new index of this template, then it will throw "unknown index sort field:\[${index sort settin… --- ## [A new support profile search for trace API](https://discuss.elastic.co/t/a-new-support-profile-search-for-trace-api/252467)
**Author:** [@hackerwin7](https://discuss.elastic.co/u/hackerwin7)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 3:10am UTC](https://discuss.elastic.co/t/a-new-support-profile-search-for-trace-api/252467 "2020-11-09T03:10:29Z")
Hi，teams Can we create a new API for trace for search profile. as we all know the profile API can list the shard request execution time and phase. With the only help of profile API, some cases we can not know the true … --- ## [Snapshot stored in GCS bucket will be encrypted](https://discuss.elastic.co/t/snapshot-stored-in-gcs-bucket-will-be-encrypted/254700)
**Author:** [@knagasri](https://discuss.elastic.co/u/knagasri)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 2:49am UTC](https://discuss.elastic.co/t/snapshot-stored-in-gcs-bucket-will-be-encrypted/254700 "2020-11-09T02:49:58Z")
If the elasticsearch data is backed up to gcs bucket. Is data will be in encrypted form. If not how to achieve this? --- ## [Relationship in elasticsearch](https://discuss.elastic.co/t/relationship-in-elasticsearch/254358)
**Author:** [@Syed.Ubaid](https://discuss.elastic.co/u/Syed.Ubaid)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 2:04am UTC](https://discuss.elastic.co/t/relationship-in-elasticsearch/254358 "2020-11-09T02:04:01Z")
I have two tables in mysql and i want to join both tables and extract data in elk stack. I have join both tables and create view and import in elk stack but the thing is that data is duplicate . Is that possible through… --- ## [Zeek with elasticsearch](https://discuss.elastic.co/t/zeek-with-elasticsearch/254231)
**Author:** [@shabi037](https://discuss.elastic.co/u/shabi037)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 1:45am UTC](https://discuss.elastic.co/t/zeek-with-elasticsearch/254231 "2020-11-09T01:45:49Z")
Hi all, I am running Zeek (Bro) in an offline mode to analyze the pcap file. I get the logs in the "/opt/zeek/logs/current" directory. To further analyze the received logs I am passing those to Filebeat to be visualized… --- ## [Elasticsearch upgrade from 6.8.0 to 7.9.2](https://discuss.elastic.co/t/elasticsearch-upgrade-from-6-8-0-to-7-9-2/254223)
**Author:** [@maadhav](https://discuss.elastic.co/u/maadhav)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 1:43am UTC](https://discuss.elastic.co/t/elasticsearch-upgrade-from-6-8-0-to-7-9-2/254223 "2020-11-09T01:43:28Z")
Hi Team We have around 60 nodes docker based production cluster running with version 6.8. We need to upgrade this to version 7.9.2. We have used discovery settings such as discovery.zen.ping.unicast.hosts , discovery… --- ## [Rollup experimental](https://discuss.elastic.co/t/rollup-experimental/254364)
**Author:** [@schophil](https://discuss.elastic.co/u/schophil)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 1:36am UTC](https://discuss.elastic.co/t/rollup-experimental/254364 "2020-11-09T01:36:21Z")
Hi, I am looking into using the rollup feature on my project. It seems very interesting and the early tests I performed are positive. However I am conflicted about the "experimental feature". I woud like to use this in… --- ## [Range query in doc lvl security](https://discuss.elastic.co/t/range-query-in-doc-lvl-security/254431)
**Author:** [@pbabik](https://discuss.elastic.co/u/pbabik)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 1:33am UTC](https://discuss.elastic.co/t/range-query-in-doc-lvl-security/254431 "2020-11-09T01:33:14Z")
Hi, I am trying to set policy/rule for a user role in elasticsearch, to allow them to search 2 weeks old data only. One solution would be rotating aliases, and set permissions for those, but it isn't ideal. Better solut… --- ## [Beginner question: Skipping the filtering of id into \[@metadata\]\[\_id\]](https://discuss.elastic.co/t/beginner-question-skipping-the-filtering-of-id-into-metadata-id/254508)
**Author:** [@pyrovoice](https://discuss.elastic.co/u/pyrovoice)\ **Replies:** 0\ **Last updated:** [November 6, 2020, 9:16am UTC](https://discuss.elastic.co/t/beginner-question-skipping-the-filtering-of-id-into-metadata-id/254508 "2020-11-06T09:16:53Z")
I followed the following tutorial: https://www.elastic.co/blog/how-to-keep-elasticsearch-synchronized-with-a-relational-database-using-logstash The configuration file for the pipeline contains the following: input … --- ## [Does watcher action webhook involve retrying?](https://discuss.elastic.co/t/does-watcher-action-webhook-involve-retrying/254686)
**Author:** [@goodmirek5849](https://discuss.elastic.co/u/goodmirek5849)\ **Replies:** 0\ **Last updated:** [November 8, 2020, 4:05pm UTC](https://discuss.elastic.co/t/does-watcher-action-webhook-involve-retrying/254686 "2020-11-08T16:05:05Z")
We use Elasticsearch in ElasticCloud and use Watcher. When the Watcher raises an alert, it invokes a configured webhook action. It works most of the time. I need to know whether the Watcher retries the webhook action, … --- ## [Getting all documents in Elastic Search Shard](https://discuss.elastic.co/t/getting-all-documents-in-elastic-search-shard/254690)
**Author:** [@asagarwal10](https://discuss.elastic.co/u/asagarwal10)\ **Replies:** 0\ **Last updated:** [November 8, 2020, 7:41pm UTC](https://discuss.elastic.co/t/getting-all-documents-in-elastic-search-shard/254690 "2020-11-08T19:41:31Z")
I am trying to get all documents in a shard. When creating the index, I used the query: PUT mortgages-stage { "mappings": { "\_routing": { "required": true } } } When adding documents to the index, I … --- ## [How do the "query" and "suggest" blocks relate in a suggestion request?](https://discuss.elastic.co/t/how-do-the-query-and-suggest-blocks-relate-in-a-suggestion-request/254685)
**Author:** [@Timothy\_Wall](https://discuss.elastic.co/u/Timothy_Wall)\ **Replies:** 2\ **Last updated:** [November 8, 2020, 7:09pm UTC](https://discuss.elastic.co/t/how-do-the-query-and-suggest-blocks-relate-in-a-suggestion-request/254685 "2020-11-08T19:09:50Z")
If I'm doing a suggestion request like below, how do the "query" and "suggest" blocks relate to one another? POST /\_search { "query": {...}, "suggest": {...}, } [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1708) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1710)