# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1208 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1209 --- ## [Huge monitoring-es indexes](https://discuss.elastic.co/t/huge-monitoring-es-indexes/285102)
**Author:** [@Daniele\_Renda](https://discuss.elastic.co/u/Daniele_Renda)\ **Replies:** 10\ **Last updated:** [October 10, 2021, 11:06pm UTC](https://discuss.elastic.co/t/huge-monitoring-es-indexes/285102 "2021-10-10T23:06:44Z")
Today my ES cluster on Elastic cloud frozen due to high CPU consumption. There was not too much pressure with requests so I took a look at indexes to check if there is some problem. I found a lot of old indexes from apm… --- ## [Queries about indexing data into elastic search](https://discuss.elastic.co/t/queries-about-indexing-data-into-elastic-search/285542)
**Author:** [@Magesh\_02](https://discuss.elastic.co/u/Magesh_02)\ **Replies:** 3\ **Last updated:** [October 10, 2021, 11:06pm UTC](https://discuss.elastic.co/t/queries-about-indexing-data-into-elastic-search/285542 "2021-10-10T23:06:24Z")
I have some questions regarding indexing the data. In my case, I needed to push 100 million JSON records into a single index on a specific date, and the date had to be included in the JSON field. I imported this data in… --- ## [Functionbeat cannot process more than 1 message from pub/sub](https://discuss.elastic.co/t/functionbeat-cannot-process-more-than-1-message-from-pub-sub/285903)
**Author:** [@Yero\_Me](https://discuss.elastic.co/u/Yero_Me)\ **Replies:** 0\ **Last updated:** [October 5, 2021, 10:30am UTC](https://discuss.elastic.co/t/functionbeat-cannot-process-more-than-1-message-from-pub-sub/285903 "2021-10-05T10:30:57Z")
Hello, Since 7.11 up to 7.15, I cannot configure functionbeat to process more than 1 message. I deploy functionbeat on GCP cloud function. Every first message send to cloud function will be properly traited and sent t… --- ## [Conditional colouring Kibana canvas](https://discuss.elastic.co/t/conditional-colouring-kibana-canvas/285951)
**Author:** [@Mira\_9](https://discuss.elastic.co/u/Mira_9)\ **Replies:** 1\ **Last updated:** [October 10, 2021, 10:52pm UTC](https://discuss.elastic.co/t/conditional-colouring-kibana-canvas/285951 "2021-10-10T22:52:03Z")
Hi, I'm trying to change the color of the bars here depending on the value of "Primary channel " . i may have 12 values in "Primary channel" . the graph is displayed in real time so the color changing must updated th… --- ## [Elastic Agent Enroll](https://discuss.elastic.co/t/elastic-agent-enroll/286026)
**Author:** [@ravenmx](https://discuss.elastic.co/u/ravenmx)\ **Replies:** 0\ **Last updated:** [October 6, 2021, 12:41pm UTC](https://discuss.elastic.co/t/elastic-agent-enroll/286026 "2021-10-06T12:41:08Z")
Good Day. Do not tell me by chance whether it is possible to add any custom field to the elastic agent, where I can leave a record when installing the agent and work with it when searching (it seems that this is impossi… --- ## [Nested condition with double action if returns false or true](https://discuss.elastic.co/t/nested-condition-with-double-action-if-returns-false-or-true/286055)
**Author:** [@Juan\_David\_Jaramillo](https://discuss.elastic.co/u/Juan_David_Jaramillo)\ **Replies:** 0\ **Last updated:** [October 6, 2021, 4:30pm UTC](https://discuss.elastic.co/t/nested-condition-with-double-action-if-returns-false-or-true/286055 "2021-10-06T16:30:27Z")
Hello, I have a question, I do not know if you can have a nested condition in an alert in elastisearch, ie I have a condition that if met returns true and send an email, but I also want if it is not met and returns fals… --- ## [How to create an array in logstash with already defined fields](https://discuss.elastic.co/t/how-to-create-an-array-in-logstash-with-already-defined-fields/286086)
**Author:** [@Juan\_David\_Jaramillo](https://discuss.elastic.co/u/Juan_David_Jaramillo)\ **Replies:** 0\ **Last updated:** [October 7, 2021, 2:35am UTC](https://discuss.elastic.co/t/how-to-create-an-array-in-logstash-with-already-defined-fields/286086 "2021-10-07T02:35:28Z")
Good, I have a doubt and is that I have some specific oids already named and I would like to know how I can do to concatenate them, that is to say to put them in an array or table, so that it contains me all the oids tha… --- ## [Measure volume of events we are consuming(Size of all input events arrived)](https://discuss.elastic.co/t/measure-volume-of-events-we-are-consuming-size-of-all-input-events-arrived/286113)
**Author:** [@Dhananjay\_vispute](https://discuss.elastic.co/u/Dhananjay_vispute)\ **Replies:** 0\ **Last updated:** [October 7, 2021, 9:46am UTC](https://discuss.elastic.co/t/measure-volume-of-events-we-are-consuming-size-of-all-input-events-arrived/286113 "2021-10-07T09:46:55Z")
I have added metric to count number events as input, dropped and arrived at output. I also want to measure size of all input events bytes. Is there any way we achieve this through metrics ? --- ## [Creating alerts for logs](https://discuss.elastic.co/t/creating-alerts-for-logs/286205)
**Author:** [@Kapil\_K](https://discuss.elastic.co/u/Kapil_K)\ **Replies:** 0\ **Last updated:** [October 8, 2021, 8:17am UTC](https://discuss.elastic.co/t/creating-alerts-for-logs/286205 "2021-10-08T08:17:55Z")
Hi all. I need to add alerts to my dashboards. I have deployed Elasticsearch helm chart to deploy in my K8s cluster. I find only stateful set has been deployed. How can I work with Elasticsearch.yml? --- ## [Thread pool queue size suddenly spikes to thousands](https://discuss.elastic.co/t/thread-pool-queue-size-suddenly-spikes-to-thousands/286176)
**Author:** [@vibgy](https://discuss.elastic.co/u/vibgy)\ **Replies:** 4\ **Last updated:** [October 10, 2021, 10:16pm UTC](https://discuss.elastic.co/t/thread-pool-queue-size-suddenly-spikes-to-thousands/286176 "2021-10-10T22:16:57Z")
Every once in a while, thread pool queue size suddenly spike to thousands When this happens, users start to notice that kibana becomes pretty much unresponsive, for minutes. In the above case, it became unresponsive … --- ## [Send email alerts if log level == ERROR in more than one index](https://discuss.elastic.co/t/send-email-alerts-if-log-level-error-in-more-than-one-index/285750)
**Author:** [@Chandrakant\_Naik](https://discuss.elastic.co/u/Chandrakant_Naik)\ **Replies:** 5\ **Last updated:** [October 10, 2021, 10:04pm UTC](https://discuss.elastic.co/t/send-email-alerts-if-log-level-error-in-more-than-one-index/285750 "2021-10-10T22:04:52Z")
i have ingested python logs in Elasticsearch. Sample of my logs given below \[2021-10-01 23:54:39,752\] \[INFO\] \[run\]\[MainThread\] \[xxxxxxxxx\] \[function\_name:line no\] : updating user request mapping for the request xxxxxxx … --- ## [How to parse two timestamp fields from one single log line message](https://discuss.elastic.co/t/how-to-parse-two-timestamp-fields-from-one-single-log-line-message/286282)
**Author:** [@Patr123](https://discuss.elastic.co/u/Patr123)\ **Replies:** 6\ **Last updated:** [October 10, 2021, 3:33pm UTC](https://discuss.elastic.co/t/how-to-parse-two-timestamp-fields-from-one-single-log-line-message/286282 "2021-10-10T15:33:30Z")
Hello All, I have log lines coming from K8's to logstash and they have two timestamps back to back. How can I parse those fields in let's say @timestamp field and timestamp2 field? My log lines look like: 2021-10-08 1… --- ## [Memory, storage and CPU requirements depending on node roles](https://discuss.elastic.co/t/memory-storage-and-cpu-requirements-depending-on-node-roles/286294)
**Author:** [@flavienbwk](https://discuss.elastic.co/u/flavienbwk)\ **Replies:** 3\ **Last updated:** [October 10, 2021, 2:58pm UTC](https://discuss.elastic.co/t/memory-storage-and-cpu-requirements-depending-on-node-roles/286294 "2021-10-10T14:58:38Z")
Hi, Has anyone any intuition about what are the memory, storage and CPU requirements depending on node roles ? Is the following intuition right ? Coordinating nodes : high CPU, low memory Master-only nodes : high CPU… --- ## [Some thoughts on adding new Analyzer to existing Index without reindex or close](https://discuss.elastic.co/t/some-thoughts-on-adding-new-analyzer-to-existing-index-without-reindex-or-close/285242)
**Author:** [@garyzjq](https://discuss.elastic.co/u/garyzjq)\ **Replies:** 5\ **Last updated:** [October 10, 2021, 12:19pm UTC](https://discuss.elastic.co/t/some-thoughts-on-adding-new-analyzer-to-existing-index-without-reindex-or-close/285242 "2021-10-10T12:19:57Z")
Hi team, (Elasticsearch 7.5) My scenario is to add a new custom Analyzer to an existing Index. Analyzer definition may like below: { "analyzer": { "ngram\_analyzer": { "type": "custom", … --- ## [Add a node to elastic running on docker from another server](https://discuss.elastic.co/t/add-a-node-to-elastic-running-on-docker-from-another-server/285156)
**Author:** [@kakigadol](https://discuss.elastic.co/u/kakigadol)\ **Replies:** 9\ **Last updated:** [October 10, 2021, 11:38am UTC](https://discuss.elastic.co/t/add-a-node-to-elastic-running-on-docker-from-another-server/285156 "2021-10-10T11:38:48Z")
I have an Elasticsearch 6.6 cluster with two nodes running on the first server using docker-compose I want to add another node running on a separate server to get more physical resources. I configured everything the sa… --- ## [Get a count of buckets from a terms agg, but not the buckets themselves](https://discuss.elastic.co/t/get-a-count-of-buckets-from-a-terms-agg-but-not-the-buckets-themselves/286259)
**Author:** [@annmarie-switzer](https://discuss.elastic.co/u/annmarie-switzer)\ **Replies:** 1\ **Last updated:** [October 10, 2021, 7:58am UTC](https://discuss.elastic.co/t/get-a-count-of-buckets-from-a-terms-agg-but-not-the-buckets-themselves/286259 "2021-10-10T07:58:40Z")
Is there any way to get the bucket count from a terms agg, but just the count, not the buckets themselves? I have a scenario where I want to display a table (with pagination) of all the non-unique values (buckets) for … --- ## [Error: Unknown key for a VALUE\_STRING in \[analyzer\]](https://discuss.elastic.co/t/error-unknown-key-for-a-value-string-in-analyzer/286105)
**Author:** [@Drsaud](https://discuss.elastic.co/u/Drsaud)\ **Replies:** 10\ **Last updated:** [October 10, 2021, 7:37am UTC](https://discuss.elastic.co/t/error-unknown-key-for-a-value-string-in-analyzer/286105 "2021-10-10T07:37:23Z")
Hi, I'm having this error in postman Unknown key for a VALUE\_STRING in \[analyzer\]. when I search like this: (note I get this after I add the analyzer) { "query": { "terms": { "username": … --- ## [Parsing nested json with logstash](https://discuss.elastic.co/t/parsing-nested-json-with-logstash/286287)
**Author:** [@diaztech](https://discuss.elastic.co/u/diaztech)\ **Replies:** 3\ **Last updated:** [October 10, 2021, 12:44am UTC](https://discuss.elastic.co/t/parsing-nested-json-with-logstash/286287 "2021-10-10T00:44:57Z")
I'm trying to import a collection from mongo with the following input input { mongodb { uri =\> "uripath" placeholder\_db\_dir =\> "../opt/logstash-mongodb/" placeholder\_db\_name =\> "logstash\_sqli… --- ## [Can Painless script to execute search](https://discuss.elastic.co/t/can-painless-script-to-execute-search/286305)
**Author:** [@Param](https://discuss.elastic.co/u/Param)\ **Replies:** 0\ **Last updated:** [October 9, 2021, 7:29pm UTC](https://discuss.elastic.co/t/can-painless-script-to-execute-search/286305 "2021-10-09T19:29:30Z")
Hi I am using scripted grouping of a index . if any bucket's document count is greater than 0, i want to select description filed from index with all of grouping field using as filter for that query. "query": { … --- ## [Import text file to logstash](https://discuss.elastic.co/t/import-text-file-to-logstash/286303)
**Author:** [@quyennguyen](https://discuss.elastic.co/u/quyennguyen)\ **Replies:** 1\ **Last updated:** [October 9, 2021, 6:45pm UTC](https://discuss.elastic.co/t/import-text-file-to-logstash/286303 "2021-10-09T18:45:24Z")
I had some text files with formats like this. 10/09-22:09:34.989323 \[\*\*\] \[1:100000015:3\] "OR SQL Injection Detected" \[\*\*\] \[Priority: 0\] {TCP} 192.168.186.25:44748 -\> 192.168.186.162:80 10/09-22:09:34.989323 \[\*\*\] \[1:1000… --- ## [Metricbeat from source builds x-pack modules but doesnt contain directories in image](https://discuss.elastic.co/t/metricbeat-from-source-builds-x-pack-modules-but-doesnt-contain-directories-in-image/285752)
**Author:** [@ss\_22](https://discuss.elastic.co/u/ss_22)\ **Replies:** 2\ **Last updated:** [October 9, 2021, 3:46pm UTC](https://discuss.elastic.co/t/metricbeat-from-source-builds-x-pack-modules-but-doesnt-contain-directories-in-image/285752 "2021-10-09T15:46:27Z")
Hello, I have recently built Metricbeat from source code after following the developer guides. I noticed it builds all of the x-pack modules, but at the end of the day the image doesnt contain these module directories. … --- ## [How to remove ipv6 from host.ip](https://discuss.elastic.co/t/how-to-remove-ipv6-from-host-ip/284341)
**Author:** [@xizhimen](https://discuss.elastic.co/u/xizhimen)\ **Replies:** 4\ **Last updated:** [October 9, 2021, 6:40am UTC](https://discuss.elastic.co/t/how-to-remove-ipv6-from-host-ip/284341 "2021-10-09T06:40:59Z")
background: winlogbeat7.3.2 windows2016 winlogbeat.yml: processors: - add\_host\_metadata: netinfo.enabled: true requirement: in kibana: I want to remove this ipv6 info ,How? THANKS! I try to remove ipv… --- ## [Custom Plugin add filter to search query by external access policies](https://discuss.elastic.co/t/custom-plugin-add-filter-to-search-query-by-external-access-policies/285723)
**Author:** [@David\_Sheard](https://discuss.elastic.co/u/David_Sheard)\ **Replies:** 5\ **Last updated:** [October 9, 2021, 12:24am UTC](https://discuss.elastic.co/t/custom-plugin-add-filter-to-search-query-by-external-access-policies/285723 "2021-10-09T00:24:46Z")
Hi, I'm wanting to create a custom plugin. The purpose is to make a rest call to an external service to apply access control polices to search requests. Is there any sample code or an article that would assist? Appreci… --- ## [Searching through every document in ES?](https://discuss.elastic.co/t/searching-through-every-document-in-es/286281)
**Author:** [@bensalerno](https://discuss.elastic.co/u/bensalerno)\ **Replies:** 7\ **Last updated:** [October 8, 2021, 11:13pm UTC](https://discuss.elastic.co/t/searching-through-every-document-in-es/286281 "2021-10-08T23:13:56Z")
My cluster has roughly 300,000 documents in it. Right now, I'm only able to query 10,000 at a time. Is there a way to query through all documents? Or is this bad practice? Many of search queries are returning 0 results … --- ## [Frozen Indices Deprecation](https://discuss.elastic.co/t/frozen-indices-deprecation/286288)
**Author:** [@vthacker](https://discuss.elastic.co/u/vthacker)\ **Replies:** 1\ **Last updated:** [October 9, 2021, 12:06am UTC](https://discuss.elastic.co/t/frozen-indices-deprecation/286288 "2021-10-09T00:06:50Z")
Watching this recent talk frozen indices sounds like a feature recommended to users However when I look at the doc here the feature is deprecated. I'm not sure which one to read into - Is the feature going to exist goi… --- ## [Issue with FSCrawler (fscrawler-es7-2.7) . Refusing to upload new files . Warning: set filename as stored field!](https://discuss.elastic.co/t/issue-with-fscrawler-fscrawler-es7-2-7-refusing-to-upload-new-files-warning-set-filename-as-stored-field/286272)
**Author:** [@Francois\_Saab](https://discuss.elastic.co/u/Francois_Saab)\ **Replies:** 1\ **Last updated:** [October 8, 2021, 11:15pm UTC](https://discuss.elastic.co/t/issue-with-fscrawler-fscrawler-es7-2-7-refusing-to-upload-new-files-warning-set-filename-as-stored-field/286272 "2021-10-08T23:15:06Z")
We are facing an issue with FSCrawler (fscrawler-es7-2.7). FSCrawler is refusing to upload new files and raises the below error: 08:37:04,730 WARN \[f.p.e.c.f.FsParserAbstract\] Error while crawling \\SearchEngineFiles\\D… --- ## [Issue with escaping pipe "|" in Ingest Pipeline](https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279)
**Author:** [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\ **Replies:** 7\ **Last updated:** [October 8, 2021, 9:20pm UTC](https://discuss.elastic.co/t/issue-with-escaping-pipe-in-ingest-pipeline/286279 "2021-10-08T21:20:37Z")
Hi Guys, I have a log patern that look like this (windows event log) Dummy request has been approved.|Node=blah-blah.example.net\_iam|BatchSig=jhsdjahsdgsjahdg|Requester=john.test|Recipient=ex0000123456 below grok pat… --- ## [New filebeat module - how to move to production? Template not loading](https://discuss.elastic.co/t/new-filebeat-module-how-to-move-to-production-template-not-loading/285775)
**Author:** [@kelvins](https://discuss.elastic.co/u/kelvins)\ **Replies:** 7\ **Last updated:** [October 8, 2021, 8:18pm UTC](https://discuss.elastic.co/t/new-filebeat-module-how-to-move-to-production-template-not-loading/285775 "2021-10-08T20:18:37Z")
I have created a new module to Filebeat. To be honest I have been struggling with this for a long time, perhaps a blog post somewhere (creating a new filebeat start to finish) I could find could help me immensely. Anyw… --- ## [Warn users when they do unindexed searches](https://discuss.elastic.co/t/warn-users-when-they-do-unindexed-searches/286164)
**Author:** [@vibgy](https://discuss.elastic.co/u/vibgy)\ **Replies:** 5\ **Last updated:** [October 8, 2021, 6:23pm UTC](https://discuss.elastic.co/t/warn-users-when-they-do-unindexed-searches/286164 "2021-10-08T18:23:58Z")
We have observed in our stack that a lot of the users are making queries without specifying a single indexed field. And we would like to soft-warn our users when they do so. What would be the best way to implement somet… --- ## [Missing delete phase](https://discuss.elastic.co/t/missing-delete-phase/286261)
**Author:** [@ggab31](https://discuss.elastic.co/u/ggab31)\ **Replies:** 2\ **Last updated:** [October 8, 2021, 5:56pm UTC](https://discuss.elastic.co/t/missing-delete-phase/286261 "2021-10-08T17:56:34Z")
Hi, I have cluster with a single node. Today, I upgraded Elasticsearch and Kibana from 7.5 to 7.15. Maybe it's not related to the upgrade but now, the delete phase is missing when I want to create a new Index Lifecycle… [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1207) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1209)