# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1707 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1708 --- ## [Start from scratch every time you read a file for window logstash-input-file](https://discuss.elastic.co/t/start-from-scratch-every-time-you-read-a-file-for-window-logstash-input-file/254512)
**Author:** [@xiaoping1993](https://discuss.elastic.co/u/xiaoping1993)\ **Replies:** 3\ **Last updated:** [November 10, 2020, 3:55am UTC](https://discuss.elastic.co/t/start-from-scratch-every-time-you-read-a-file-for-window-logstash-input-file/254512 "2020-11-10T03:55:31Z")
I know if linux just set sincedb\_path =\> "/dev/null" but my os is window,and I find a method set sincedb\_path =\> "NUL",and it's not useful so if I want to realized this effect ,how I can do --- ## [DNS filter is not working](https://discuss.elastic.co/t/dns-filter-is-not-working/254781)
**Author:** [@jhansibalu](https://discuss.elastic.co/u/jhansibalu)\ **Replies:** 1\ **Last updated:** [November 10, 2020, 2:20am UTC](https://discuss.elastic.co/t/dns-filter-is-not-working/254781 "2020-11-10T02:20:53Z")
Hi here I am trying to resolve the hostname to host ipaddress Here is my dns configuration in filter block mutate { rename =\> {"host" =\> "\[host\]\[hostname\]"} add\_field =\> {"\[host\]\[ipaddress\]" =\> "%{\[host\]\[hostname\]}"… --- ## [Grok Extractor Help](https://discuss.elastic.co/t/grok-extractor-help/254798)
**Author:** [@Micah\_Bailey](https://discuss.elastic.co/u/Micah_Bailey)\ **Replies:** 1\ **Last updated:** [November 10, 2020, 2:20am UTC](https://discuss.elastic.co/t/grok-extractor-help/254798 "2020-11-10T02:20:02Z")
I am new to Grok Extractors and I am trying to extract some fields from Logstash. Example Message: Nov 9 08:53:00 192.168.131.6 rsgpchkd\[867\]: Portforwarding for rsgcadc123215 on /dev/pts/0, destination 10.208.203.30:… --- ## [Data Volume](https://discuss.elastic.co/t/data-volume/254808)
**Author:** [@lks](https://discuss.elastic.co/u/lks)\ **Replies:** 4\ **Last updated:** [November 10, 2020, 2:10am UTC](https://discuss.elastic.co/t/data-volume/254808 "2020-11-10T02:10:11Z")
on the open source version, How much data does Elastic stack handle? Can I import a pipe delimited data in Kibana? What is the max volume size of data that I can import in Kibana? --- ## [Heartbeat monitoring system resources utilization](https://discuss.elastic.co/t/heartbeat-monitoring-system-resources-utilization/254745)
**Author:** [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Replies:** 4\ **Last updated:** [November 10, 2020, 1:13am UTC](https://discuss.elastic.co/t/heartbeat-monitoring-system-resources-utilization/254745 "2020-11-10T01:13:52Z")
Hello, I wanted to ask what is a better approach in setting up for example icmp monitor for 1 minute ping: 1.) Schedule a monitor to start every 24h with wait 1m value? 2.) Schedule monitor every 1 minute with 1 minut… --- ## [Unique count and bucket aggregation with top hits](https://discuss.elastic.co/t/unique-count-and-bucket-aggregation-with-top-hits/254698)
**Author:** [@smruthip](https://discuss.elastic.co/u/smruthip)\ **Replies:** 3\ **Last updated:** [November 9, 2020, 5:28pm UTC](https://discuss.elastic.co/t/unique-count-and-bucket-aggregation-with-top-hits/254698 "2020-11-09T17:28:48Z")
Hi all, I want to display unique count of container\_id's for each state. I want to do a bucket aggregation to get the latest state of the container and then do a unique count of the containers for each state to get the… --- ## [Implement collocation in ElasticSearch](https://discuss.elastic.co/t/implement-collocation-in-elasticsearch/254118)
**Author:** [@a22xel](https://discuss.elastic.co/u/a22xel)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 10:16pm UTC](https://discuss.elastic.co/t/implement-collocation-in-elasticsearch/254118 "2020-11-09T22:16:16Z")
I am new in Elastic Search Usage I am trying to do an analyser or an ingest pipeline that would create collocations of words ( unigram, bigrams, trigrams with a step up to 2 ) . I am aware it's feasible in python but I a… --- ## [Memory & CPU stats for Xpack Monitoring plugin](https://discuss.elastic.co/t/memory-cpu-stats-for-xpack-monitoring-plugin/254810)
**Author:** [@valleram](https://discuss.elastic.co/u/valleram)\ **Replies:** 3\ **Last updated:** [November 9, 2020, 10:15pm UTC](https://discuss.elastic.co/t/memory-cpu-stats-for-xpack-monitoring-plugin/254810 "2020-11-09T22:15:13Z")
Hi All, Is there a way to know how much system resources(MEM and CPU) is xpack.monitoring consuming? Any help would be much appreciated Thank You --- ## [ILM policy behaviour with disk-based shard allocation settings](https://discuss.elastic.co/t/ilm-policy-behaviour-with-disk-based-shard-allocation-settings/254766)
**Author:** [@Tarasovych](https://discuss.elastic.co/u/Tarasovych)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 10:14pm UTC](https://discuss.elastic.co/t/ilm-policy-behaviour-with-disk-based-shard-allocation-settings/254766 "2020-11-09T22:14:26Z")
Hi! I have a cluster with two host data nodes, one warm data node and one cold data node. I have a policy which moves indices from hot to warm, when index grows up to 1gb. I'd like to know, what will happen if warm data… --- ## [How to "recreate" events from an index by just changing time?](https://discuss.elastic.co/t/how-to-recreate-events-from-an-index-by-just-changing-time/254761)
**Author:** [@kelk](https://discuss.elastic.co/u/kelk)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 10:06pm UTC](https://discuss.elastic.co/t/how-to-recreate-events-from-an-index-by-just-changing-time/254761 "2020-11-09T22:06:21Z")
hi We have some "enriched" events in an index from last month for a customer product checkout. The enrichment has happened in "elastic beats" && "logstash" and not entirely in my control. So I cannot simulate the \_raw e… --- ## [Kibana](https://discuss.elastic.co/t/kibana/254730)
**Author:** [@Syed.Ubaid](https://discuss.elastic.co/u/Syed.Ubaid)\ **Replies:** 4\ **Last updated:** [November 9, 2020, 9:59pm UTC](https://discuss.elastic.co/t/kibana/254730 "2020-11-09T21:59:35Z")
I have create an index and insert 5 records in kibana problem is that name field is showing in discover but when i make visualisation name field is not in showing in the visualisation field. please help --- ## [Grok debugger works but logstash fails multiple lines](https://discuss.elastic.co/t/grok-debugger-works-but-logstash-fails-multiple-lines/254824)
**Author:** [@Carlos\_Fernando\_Palm](https://discuss.elastic.co/u/Carlos_Fernando_Palm)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 9:58pm UTC](https://discuss.elastic.co/t/grok-debugger-works-but-logstash-fails-multiple-lines/254824 "2020-11-09T21:58:25Z")
Hello. I am trying to use the following regex to capture some info from a log entry: (?m)(?\<=responses=)(.\*)(?=extraData) Now it works in the grok debugger: But it fails in logstash: Any idea why that might be t… --- ## [XPack encryption mechanism/Suite](https://discuss.elastic.co/t/xpack-encryption-mechanism-suite/254773)
**Author:** [@Pooja](https://discuss.elastic.co/u/Pooja)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 9:57pm UTC](https://discuss.elastic.co/t/xpack-encryption-mechanism-suite/254773 "2020-11-09T21:57:14Z")
What kind of encryption methodology does Xpack use to encrypt data? Also, is the present version of ELK stack provide Xpack for free? --- ## [Kibana Alerting : Need to include the hit count](https://discuss.elastic.co/t/kibana-alerting-need-to-include-the-hit-count/254135)
**Author:** [@naveen\_wilson](https://discuss.elastic.co/u/naveen_wilson)\ **Replies:** 10\ **Last updated:** [November 4, 2020, 3:09pm UTC](https://discuss.elastic.co/t/kibana-alerting-need-to-include-the-hit-count/254135 "2020-11-04T15:09:43Z")
Hi All, I am configuring a Alert in Kibana. So in the alert, i need to add one more field for the count of errors. PFB the current Kibana message configured. I need to add the count of errors also in this. When i tried … --- ## [High load (\>3.0.0), 60% CPU and 30% heap usage](https://discuss.elastic.co/t/high-load-3-0-0-60-cpu-and-30-heap-usage/254278)
**Author:** [@amackenzie](https://discuss.elastic.co/u/amackenzie)\ **Replies:** 3\ **Last updated:** [November 9, 2020, 8:47pm UTC](https://discuss.elastic.co/t/high-load-3-0-0-60-cpu-and-30-heap-usage/254278 "2020-11-09T20:47:35Z")
Hi there, I'm wondering if someone could help me analyse this hot thread output. We have a number of data nodes. All of these data nodes also ingest and index too. We're using ES 5.6 Thank you, Ally --- ## [Elasticsearch Filter Syntax](https://discuss.elastic.co/t/elasticsearch-filter-syntax/254793)
**Author:** [@ksremo](https://discuss.elastic.co/u/ksremo)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 8:43pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-syntax/254793 "2020-11-09T20:43:13Z")
filter { elasticsearch { hosts =\> \["https://10. 0.0.20:9200","https://10. 0.0.21:9200","https://10. 0.0.22:9200","https://10. 0.0.23:9200","https://10. 0.0.24:9200","https://10. 0.0.25:9200","https://10. 0.0.26:9200"\] ca… --- ## [How to periodically harvest/read Status-File that changes in content, but not in size?](https://discuss.elastic.co/t/how-to-periodically-harvest-read-status-file-that-changes-in-content-but-not-in-size/254371)
**Author:** [@anriko](https://discuss.elastic.co/u/anriko)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 8:05pm UTC](https://discuss.elastic.co/t/how-to-periodically-harvest-read-status-file-that-changes-in-content-but-not-in-size/254371 "2020-11-09T20:05:37Z")
Hi, I am having a Status-Logfile, that changes periodically (every minute). The problem is, that mostly only content of the file changes (timestamps, some numbers) and the size of the file stays the same. I am having tr… --- ## [Anybody help me about Regex for parsing stack trace in my Log pattern?](https://discuss.elastic.co/t/anybody-help-me-about-regex-for-parsing-stack-trace-in-my-log-pattern/254790)
**Author:** [@quangdv.6466](https://discuss.elastic.co/u/quangdv.6466)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 6:47pm UTC](https://discuss.elastic.co/t/anybody-help-me-about-regex-for-parsing-stack-trace-in-my-log-pattern/254790 "2020-11-09T18:47:23Z")
This is my log pattern: 2020-10-27 00:02:24.021 \[http-nio-8080-exec-42\] \[|78CDD621831532CBD69F2F3DEF1859DC|-\] DEBUG c.q.u.checkpoint.CheckpointFilter CheckpointFilter.afterRequestProcessing(...) (CheckpointFilter.java:1… --- ## [Need sum Of Value](https://discuss.elastic.co/t/need-sum-of-value/254058)
**Author:** [@errupeshmca](https://discuss.elastic.co/u/errupeshmca)\ **Replies:** 3\ **Last updated:** [November 9, 2020, 6:33pm UTC](https://discuss.elastic.co/t/need-sum-of-value/254058 "2020-11-09T18:33:03Z")
working on calculating Total sales per day, I have two column in table to calculate total sales per day based on total shipped that day. if Total Shipped = 1 and date = today's date. I need sum of total dollar value … --- ## [Foreach for bucket aggregation in watcher not working](https://discuss.elastic.co/t/foreach-for-bucket-aggregation-in-watcher-not-working/254523)
**Author:** [@priti](https://discuss.elastic.co/u/priti)\ **Replies:** 6\ **Last updated:** [November 9, 2020, 6:10pm UTC](https://discuss.elastic.co/t/foreach-for-bucket-aggregation-in-watcher-not-working/254523 "2020-11-09T18:10:58Z")
Hi , I am trying to logging according to user using foreach but it is giving exception "type" : "exception", "reason" : "specified foreach object was null: \[ctx.payload.aggregations.user.buckets.keys\]" } Foreach C… --- ## [Vega-lite invalid spec error](https://discuss.elastic.co/t/vega-lite-invalid-spec-error/254706)
**Author:** [@smruthip](https://discuss.elastic.co/u/smruthip)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 5:26pm UTC](https://discuss.elastic.co/t/vega-lite-invalid-spec-error/254706 "2020-11-09T17:26:59Z")
I am just trying to do a simple count of all container\_id 's for every state in vega-lite and it thorws an error as invalid spec. What am I doing wrong here? { "$schema": "https://vega.github.io/schema/vega-lite/v2.js… --- ## [Grok multiple pattern definitions](https://discuss.elastic.co/t/grok-multiple-pattern-definitions/254797)
**Author:** [@Carlos\_Fernando\_Palm](https://discuss.elastic.co/u/Carlos_Fernando_Palm)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 5:24pm UTC](https://discuss.elastic.co/t/grok-multiple-pattern-definitions/254797 "2020-11-09T17:24:57Z")
Hello, I am trying to define several patterns to use on logstash.conf. So far I have tried this: grok{ tag\_on\_failure =\> \[\] pattern\_definitions =\> {"text1" =\> "((?\<=responses=)\[a-zA-z \]+(?=,))"} patt… --- ## [Pipeline aborted due to undefined method \`mutex'](https://discuss.elastic.co/t/pipeline-aborted-due-to-undefined-method-mutex/254777)
**Author:** [@lens](https://discuss.elastic.co/u/lens)\ **Replies:** 4\ **Last updated:** [November 9, 2020, 5:16pm UTC](https://discuss.elastic.co/t/pipeline-aborted-due-to-undefined-method-mutex/254777 "2020-11-09T17:16:22Z")
I've recently been running into an error when starting logstash. We have several pipelines configured and all will start up without a problem, except for one (not always the same one and there is not a problem every time… --- ## [Metricbeat ILM issue](https://discuss.elastic.co/t/metricbeat-ilm-issue/254541)
**Author:** [@ManuelF](https://discuss.elastic.co/u/ManuelF)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 4:39pm UTC](https://discuss.elastic.co/t/metricbeat-ilm-issue/254541 "2020-11-09T16:39:04Z")
Hi, \*Note: Running ELK 7.9.2. Beats version matches ELK version. I recently deployed Metricbeat to monitor my MSSQL server. Setup was successfully completed according to the PS console output. Metricbeat index, index t… --- ## [ILM not executing DELETE phase](https://discuss.elastic.co/t/ilm-not-executing-delete-phase/254776)
**Author:** [@radw](https://discuss.elastic.co/u/radw)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 4:38pm UTC](https://discuss.elastic.co/t/ilm-not-executing-delete-phase/254776 "2020-11-09T16:38:36Z")
Hi all! I'm using ELK stack v7.6.2 and v7.4.0. The same behaviour is happening on both versions. I've created an ILM policy together with timeseries indices and a template for an index. My ILM is rolling over my index… --- ## [Unrecognized VM option 'UseConcMarkSweepGC](https://discuss.elastic.co/t/unrecognized-vm-option-useconcmarksweepgc/254534)
**Author:** [@seePyou](https://discuss.elastic.co/u/seePyou)\ **Replies:** 4\ **Last updated:** [November 9, 2020, 4:37pm UTC](https://discuss.elastic.co/t/unrecognized-vm-option-useconcmarksweepgc/254534 "2020-11-09T16:37:47Z")
Hello I'm trying to use logstash in Windows 10 but I am stuck at this error: PS C:\\logstash-7.9.3\\bin\> .\\logstash -e 'input {stdin{}} output {stdout{}}' Unrecognized VM option 'UseConcMarkSweepGC' Error: Could not crea… --- ## [Cat api nodes returning stale data?](https://discuss.elastic.co/t/cat-api-nodes-returning-stale-data/254656)
**Author:** [@falkenbt](https://discuss.elastic.co/u/falkenbt)\ **Replies:** 9\ **Last updated:** [November 9, 2020, 4:00pm UTC](https://discuss.elastic.co/t/cat-api-nodes-returning-stale-data/254656 "2020-11-09T16:00:46Z")
ES version: 7.7.1 In our startup/shutdown scripts we call \_cat/nodes in order to get the number of running nodes to decide if we need to change the shard allocation (only the first stop should change it). The script the… --- ## [High CPU usage when log format not matching with filter rule](https://discuss.elastic.co/t/high-cpu-usage-when-log-format-not-matching-with-filter-rule/254728)
**Author:** [@4orty](https://discuss.elastic.co/u/4orty)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 3:57pm UTC](https://discuss.elastic.co/t/high-cpu-usage-when-log-format-not-matching-with-filter-rule/254728 "2020-11-09T15:57:00Z")
CPU utilization increases when data comes in that it does not meet the set rules.(grok pattern) Isn't there a defense logic to this phenomenon, such as preventing data from entering in case of failure of parsing? --- ## [Generate png report automatically and download](https://discuss.elastic.co/t/generate-png-report-automatically-and-download/254621)
**Author:** [@Jordi](https://discuss.elastic.co/u/Jordi)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 3:50pm UTC](https://discuss.elastic.co/t/generate-png-report-automatically-and-download/254621 "2020-11-09T15:50:08Z")
Hello, we are developing a project for industry, and we are working with the ELK stack. We have a problem / doubt, we are creating some reports dynamically to pdf, where we need to be able to embed in that pdf the "visu… --- ## [Is there a mecanism to check that the data coming from the source (filebeat) is the same that we have in elastic](https://discuss.elastic.co/t/is-there-a-mecanism-to-check-that-the-data-coming-from-the-source-filebeat-is-the-same-that-we-have-in-elastic/253674)
**Author:** [@soufian.eldouqe](https://discuss.elastic.co/u/soufian.eldouqe)\ **Replies:** 6\ **Last updated:** [November 9, 2020, 3:46pm UTC](https://discuss.elastic.co/t/is-there-a-mecanism-to-check-that-the-data-coming-from-the-source-filebeat-is-the-same-that-we-have-in-elastic/253674 "2020-11-09T15:46:04Z")
Hello everyone, We are using logstash to parse our application log and index to elastic search. Following is the high level architecture file beat -----\> Logstash ----\> Elastic search ------\> kibana. How can we verify… [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1706) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1708)