# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=283 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 284 --- ## [AWS Elasticsearch](https://discuss.elastic.co/t/aws-elasticsearch/254544)
**Author:** [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\ **Replies:** 6\ **Last updated:** [November 11, 2020, 8:33am UTC](https://discuss.elastic.co/t/aws-elasticsearch/254544 "2020-11-11T08:33:40Z")
Hi all. is it possible with the ELK opendistro to send logs and metrics directly to Elasticsearch without Xpack? I implemented the above but filebeat of metricbeat complains about xpack so I am using the OSS version. … --- ## [Log stack freezing After starting API end point](https://discuss.elastic.co/t/log-stack-freezing-after-starting-api-end-point/253618)
**Author:** [@Srini12](https://discuss.elastic.co/u/Srini12)\ **Replies:** 23\ **Last updated:** [November 10, 2020, 8:24pm UTC](https://discuss.elastic.co/t/log-stack-freezing-after-starting-api-end-point/253618 "2020-11-10T20:24:44Z")
Hi Guys I am new to Logstash. I am trying to write small json file to Logstash. It is reading my config etc and freezing after Starting API End Point. No errors showing. Below is my config file; input { file { path… --- ## [Logstash and RabbitMQ input duplicate messages](https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957)
**Author:** [@tomsozolins](https://discuss.elastic.co/u/tomsozolins)\ **Replies:** 7\ **Last updated:** [November 10, 2020, 7:50pm UTC](https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957 "2020-11-10T19:50:07Z")
Hello! I have annoying problem which i am trying to solve for like a week now and i have no idea where to look anymore... ElasticStack 7.9.3 on 3 node cluster (CentOS 7, 3.10.0-1127.19.1.el7.x86\_64). RabbitMQ 3.8.9 E… --- ## [Discover in Kibana show differrent data in Table vs. JSON](https://discuss.elastic.co/t/discover-in-kibana-show-differrent-data-in-table-vs-json/254892)
**Author:** [@Michaela\_Krkosova](https://discuss.elastic.co/u/Michaela_Krkosova)\ **Replies:** 4\ **Last updated:** [November 10, 2020, 7:13pm UTC](https://discuss.elastic.co/t/discover-in-kibana-show-differrent-data-in-table-vs-json/254892 "2020-11-10T19:13:37Z")
Hi there I use ELK Stack 7.2. and when I import data, I Discover in Kibana I see proper data, but in JSON, there is missing letter (Š). I see that Kibana can read it properly. I push data to Elastistack via Logstash (IS… --- ## [Logstash/grok patterns with ECS](https://discuss.elastic.co/t/logstash-grok-patterns-with-ecs/197871)
**Author:** [@prophoto](https://discuss.elastic.co/u/prophoto)\ **Replies:** 11\ **Last updated:** [November 10, 2020, 6:33pm UTC](https://discuss.elastic.co/t/logstash-grok-patterns-with-ecs/197871 "2020-11-10T18:33:10Z")
For a couple weeks I've been attempting to migrate my logs to ECS. I have a running ELK 6.x cluster and it works fine, but my new cluster I want to see if I can get ECS running. I am starting from scratch so old logs don… --- ## [Logstash--jackson-databind vulnerabilities for logstash-7.9.0](https://discuss.elastic.co/t/logstash-jackson-databind-vulnerabilities-for-logstash-7-9-0/254962)
**Author:** [@irfan5189](https://discuss.elastic.co/u/irfan5189)\ **Replies:** 0\ **Last updated:** [November 10, 2020, 6:15pm UTC](https://discuss.elastic.co/t/logstash-jackson-databind-vulnerabilities-for-logstash-7-9-0/254962 "2020-11-10T18:15:57Z")
We found vulnerabilities with logstash version 7.9.0 for jackson-databind. below are the few CVE's for the versions: CVE-2020-24750 CVE-2017-18640 CVE-2020-24616 CVE-2020-14195 CVE-2020-14060 CVE-2020-14062 CVE-2… --- ## [Date\_time\_parse\_exception Warning message in Logstash while fetching data from CSV file](https://discuss.elastic.co/t/date-time-parse-exception-warning-message-in-logstash-while-fetching-data-from-csv-file/254922)
**Author:** [@Pavitra\_Poojary](https://discuss.elastic.co/u/Pavitra_Poojary)\ **Replies:** 5\ **Last updated:** [November 10, 2020, 5:54pm UTC](https://discuss.elastic.co/t/date-time-parse-exception-warning-message-in-logstash-while-fetching-data-from-csv-file/254922 "2020-11-10T17:54:11Z")
Hi, I am trying to fetch data from few csv file input { file { path =\> "/root/API\*" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" type =\> "API" } } filter { csv { separator =\> ",… --- ## [How to handle nonexistent index in Logstash Elasticsearch filter](https://discuss.elastic.co/t/how-to-handle-nonexistent-index-in-logstash-elasticsearch-filter/254938)
**Author:** [@egray](https://discuss.elastic.co/u/egray)\ **Replies:** 0\ **Last updated:** [November 10, 2020, 3:46pm UTC](https://discuss.elastic.co/t/how-to-handle-nonexistent-index-in-logstash-elasticsearch-filter/254938 "2020-11-10T15:46:10Z")
Hi, I'm getting the exception Failed to query elasticsearch for previous event in Logstash when I use the Elasticsearch filter plugin, because index is dynamic and I can't guarantee that it exists. I could do this with … --- ## [Mutate documentation error](https://discuss.elastic.co/t/mutate-documentation-error/254917)
**Author:** [@gyterpena](https://discuss.elastic.co/u/gyterpena)\ **Replies:** 1\ **Last updated:** [November 10, 2020, 3:34pm UTC](https://discuss.elastic.co/t/mutate-documentation-error/254917 "2020-11-10T15:34:41Z")
Mistake in documentation In above image mutate rename example differs with --- ## [\_grokparsingfailure with Cisco Catalyst switch syslogs](https://discuss.elastic.co/t/grokparsingfailure-with-cisco-catalyst-switch-syslogs/254932)
**Author:** [@Christer\_Palmen](https://discuss.elastic.co/u/Christer_Palmen)\ **Replies:** 0\ **Last updated:** [November 10, 2020, 3:21pm UTC](https://discuss.elastic.co/t/grokparsingfailure-with-cisco-catalyst-switch-syslogs/254932 "2020-11-10T15:21:32Z")
'Hello everybody!' 'I am setting up an Elastic PoC logserver and I am trying to setup Logstash for parsing syslogs from a Cisco switch. Everything is going forward but now I need some help with the filter part of the c… --- ## [Number of grok filters per logstash.conf file](https://discuss.elastic.co/t/number-of-grok-filters-per-logstash-conf-file/254914)
**Author:** [@Carlos\_Fernando\_Palm](https://discuss.elastic.co/u/Carlos_Fernando_Palm)\ **Replies:** 7\ **Last updated:** [November 10, 2020, 2:28pm UTC](https://discuss.elastic.co/t/number-of-grok-filters-per-logstash-conf-file/254914 "2020-11-10T14:28:23Z")
Hello everybody. I am trying to parse logs using multiple grok filters. I have noticed that I can use 8 filters so far, each one with its own pattern, like this: grok{ tag\_on\_failure =\> \[\] pattern\_definitions =\> {"… --- ## [Grok regex with or](https://discuss.elastic.co/t/grok-regex-with-or/254815)
**Author:** [@Carlos\_Fernando\_Palm](https://discuss.elastic.co/u/Carlos_Fernando_Palm)\ **Replies:** 4\ **Last updated:** [November 10, 2020, 2:14pm UTC](https://discuss.elastic.co/t/grok-regex-with-or/254815 "2020-11-10T14:14:34Z")
Hello. I am trying to get some info from a message field using regular expressions. Concretely I want to get what's after certain keywords. This is the regex: (?\<="title" :)(.+?)(?=,)|((?\<="description" :)(.+?)(?=,)) N… --- ## [Bug Metricbeat -\> Logstash](https://discuss.elastic.co/t/bug-metricbeat-logstash/254916)
**Author:** [@Vlad\_Piratov](https://discuss.elastic.co/u/Vlad_Piratov)\ **Replies:** 0\ **Last updated:** [November 10, 2020, 1:26pm UTC](https://discuss.elastic.co/t/bug-metricbeat-logstash/254916 "2020-11-10T13:26:25Z")
Version stack = 7.6.2 Errors \[2020-11-10T16:26:23,295\]\[WARN \]\[logstash.outputs.elasticsearch\]\[main\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_index=\>"metricbeat-7.6.2-202… --- ## [Logstash conf for parsing ignored](https://discuss.elastic.co/t/logstash-conf-for-parsing-ignored/254879)
**Author:** [@Andex](https://discuss.elastic.co/u/Andex)\ **Replies:** 0\ **Last updated:** [November 10, 2020, 9:08am UTC](https://discuss.elastic.co/t/logstash-conf-for-parsing-ignored/254879 "2020-11-10T09:08:52Z")
Hi , i have a problem with logstash. I receive filebeat jboss log but these don't go trough the conf (with grok filter) but they go on elasticsearch without parsing. I don't know why my conf is ignored. I try to chang… --- ## [Why logstash generator input generated elasticsearch index size remains constant?](https://discuss.elastic.co/t/why-logstash-generator-input-generated-elasticsearch-index-size-remains-constant/254870)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 0\ **Last updated:** [November 10, 2020, 8:35am UTC](https://discuss.elastic.co/t/why-logstash-generator-input-generated-elasticsearch-index-size-remains-constant/254870 "2020-11-10T08:35:47Z")
Hi, I have the latest elastic stack and I am using Logstash's generator input plugin to generate dummy data and I am pushing it to an elasticsearch index testindex. I have ILM policy and template configured for testinde… --- ## [Logstash Elapsed filter](https://discuss.elastic.co/t/logstash-elapsed-filter/254858)
**Author:** [@melorium](https://discuss.elastic.co/u/melorium)\ **Replies:** 0\ **Last updated:** [November 10, 2020, 7:26am UTC](https://discuss.elastic.co/t/logstash-elapsed-filter/254858 "2020-11-10T07:26:42Z")
Hi. I have problem with elapsed filter il tried everything but it random miss events. It matches around 75% of all events only and different everytime I sent the log in. Starting Logstash with one worker also. Can I … --- ## [Can i use notification for logstash's error or warn logs?](https://discuss.elastic.co/t/can-i-use-notification-for-logstashs-error-or-warn-logs/254742)
**Author:** [@talbehat](https://discuss.elastic.co/u/talbehat)\ **Replies:** 5\ **Last updated:** [November 10, 2020, 7:14am UTC](https://discuss.elastic.co/t/can-i-use-notification-for-logstashs-error-or-warn-logs/254742 "2020-11-10T07:14:40Z")
for example : \[2020-11-09T16:30:46,039\]\[WARN \]\[org.logstash.instrument.metrics.gauge.LazyDelegatingGauge\]\[main\] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: clust… --- ## [Logstash Integration with Azure](https://discuss.elastic.co/t/logstash-integration-with-azure/254851)
**Author:** [@manish9](https://discuss.elastic.co/u/manish9)\ **Replies:** 0\ **Last updated:** [November 10, 2020, 6:21am UTC](https://discuss.elastic.co/t/logstash-integration-with-azure/254851 "2020-11-10T06:21:59Z")
Can you please help me in finding a way to integrate the logs from different services in Azure to Logstash? --- ## [Logstash-input-file read all txt data](https://discuss.elastic.co/t/logstash-input-file-read-all-txt-data/254845)
**Author:** [@xiaoping1993](https://discuss.elastic.co/u/xiaoping1993)\ **Replies:** 0\ **Last updated:** [November 10, 2020, 3:59am UTC](https://discuss.elastic.co/t/logstash-input-file-read-all-txt-data/254845 "2020-11-10T03:59:10Z")
I want to read all txt data,so I set configuration: codec =\> multiline { pattern =\> "EOF" negate =\> true what =\> "previous" auto\_flush\_interval =\> 1 charset=\>"GBK" } but I can get all txt data except the Last lin… --- ## [Start from scratch every time you read a file for window logstash-input-file](https://discuss.elastic.co/t/start-from-scratch-every-time-you-read-a-file-for-window-logstash-input-file/254512)
**Author:** [@xiaoping1993](https://discuss.elastic.co/u/xiaoping1993)\ **Replies:** 3\ **Last updated:** [November 10, 2020, 3:55am UTC](https://discuss.elastic.co/t/start-from-scratch-every-time-you-read-a-file-for-window-logstash-input-file/254512 "2020-11-10T03:55:31Z")
I know if linux just set sincedb\_path =\> "/dev/null" but my os is window,and I find a method set sincedb\_path =\> "NUL",and it's not useful so if I want to realized this effect ,how I can do --- ## [DNS filter is not working](https://discuss.elastic.co/t/dns-filter-is-not-working/254781)
**Author:** [@jhansibalu](https://discuss.elastic.co/u/jhansibalu)\ **Replies:** 1\ **Last updated:** [November 10, 2020, 2:20am UTC](https://discuss.elastic.co/t/dns-filter-is-not-working/254781 "2020-11-10T02:20:53Z")
Hi here I am trying to resolve the hostname to host ipaddress Here is my dns configuration in filter block mutate { rename =\> {"host" =\> "\[host\]\[hostname\]"} add\_field =\> {"\[host\]\[ipaddress\]" =\> "%{\[host\]\[hostname\]}"… --- ## [Grok Extractor Help](https://discuss.elastic.co/t/grok-extractor-help/254798)
**Author:** [@Micah\_Bailey](https://discuss.elastic.co/u/Micah_Bailey)\ **Replies:** 1\ **Last updated:** [November 10, 2020, 2:20am UTC](https://discuss.elastic.co/t/grok-extractor-help/254798 "2020-11-10T02:20:02Z")
I am new to Grok Extractors and I am trying to extract some fields from Logstash. Example Message: Nov 9 08:53:00 192.168.131.6 rsgpchkd\[867\]: Portforwarding for rsgcadc123215 on /dev/pts/0, destination 10.208.203.30:… --- ## [Grok debugger works but logstash fails multiple lines](https://discuss.elastic.co/t/grok-debugger-works-but-logstash-fails-multiple-lines/254824)
**Author:** [@Carlos\_Fernando\_Palm](https://discuss.elastic.co/u/Carlos_Fernando_Palm)\ **Replies:** 0\ **Last updated:** [November 9, 2020, 9:58pm UTC](https://discuss.elastic.co/t/grok-debugger-works-but-logstash-fails-multiple-lines/254824 "2020-11-09T21:58:25Z")
Hello. I am trying to use the following regex to capture some info from a log entry: (?m)(?\<=responses=)(.\*)(?=extraData) Now it works in the grok debugger: But it fails in logstash: Any idea why that might be t… --- ## [Elasticsearch Filter Syntax](https://discuss.elastic.co/t/elasticsearch-filter-syntax/254793)
**Author:** [@ksremo](https://discuss.elastic.co/u/ksremo)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 8:43pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-syntax/254793 "2020-11-09T20:43:13Z")
filter { elasticsearch { hosts =\> \["https://10. 0.0.20:9200","https://10. 0.0.21:9200","https://10. 0.0.22:9200","https://10. 0.0.23:9200","https://10. 0.0.24:9200","https://10. 0.0.25:9200","https://10. 0.0.26:9200"\] ca… --- ## [Anybody help me about Regex for parsing stack trace in my Log pattern?](https://discuss.elastic.co/t/anybody-help-me-about-regex-for-parsing-stack-trace-in-my-log-pattern/254790)
**Author:** [@quangdv.6466](https://discuss.elastic.co/u/quangdv.6466)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 6:47pm UTC](https://discuss.elastic.co/t/anybody-help-me-about-regex-for-parsing-stack-trace-in-my-log-pattern/254790 "2020-11-09T18:47:23Z")
This is my log pattern: 2020-10-27 00:02:24.021 \[http-nio-8080-exec-42\] \[|78CDD621831532CBD69F2F3DEF1859DC|-\] DEBUG c.q.u.checkpoint.CheckpointFilter CheckpointFilter.afterRequestProcessing(...) (CheckpointFilter.java:1… --- ## [Need sum Of Value](https://discuss.elastic.co/t/need-sum-of-value/254058)
**Author:** [@errupeshmca](https://discuss.elastic.co/u/errupeshmca)\ **Replies:** 3\ **Last updated:** [November 9, 2020, 6:33pm UTC](https://discuss.elastic.co/t/need-sum-of-value/254058 "2020-11-09T18:33:03Z")
working on calculating Total sales per day, I have two column in table to calculate total sales per day based on total shipped that day. if Total Shipped = 1 and date = today's date. I need sum of total dollar value … --- ## [Grok multiple pattern definitions](https://discuss.elastic.co/t/grok-multiple-pattern-definitions/254797)
**Author:** [@Carlos\_Fernando\_Palm](https://discuss.elastic.co/u/Carlos_Fernando_Palm)\ **Replies:** 2\ **Last updated:** [November 9, 2020, 5:24pm UTC](https://discuss.elastic.co/t/grok-multiple-pattern-definitions/254797 "2020-11-09T17:24:57Z")
Hello, I am trying to define several patterns to use on logstash.conf. So far I have tried this: grok{ tag\_on\_failure =\> \[\] pattern\_definitions =\> {"text1" =\> "((?\<=responses=)\[a-zA-z \]+(?=,))"} patt… --- ## [Pipeline aborted due to undefined method \`mutex'](https://discuss.elastic.co/t/pipeline-aborted-due-to-undefined-method-mutex/254777)
**Author:** [@lens](https://discuss.elastic.co/u/lens)\ **Replies:** 4\ **Last updated:** [November 9, 2020, 5:16pm UTC](https://discuss.elastic.co/t/pipeline-aborted-due-to-undefined-method-mutex/254777 "2020-11-09T17:16:22Z")
I've recently been running into an error when starting logstash. We have several pipelines configured and all will start up without a problem, except for one (not always the same one and there is not a problem every time… --- ## [Unrecognized VM option 'UseConcMarkSweepGC](https://discuss.elastic.co/t/unrecognized-vm-option-useconcmarksweepgc/254534)
**Author:** [@seePyou](https://discuss.elastic.co/u/seePyou)\ **Replies:** 4\ **Last updated:** [November 9, 2020, 4:37pm UTC](https://discuss.elastic.co/t/unrecognized-vm-option-useconcmarksweepgc/254534 "2020-11-09T16:37:47Z")
Hello I'm trying to use logstash in Windows 10 but I am stuck at this error: PS C:\\logstash-7.9.3\\bin\> .\\logstash -e 'input {stdin{}} output {stdout{}}' Unrecognized VM option 'UseConcMarkSweepGC' Error: Could not crea… --- ## [High CPU usage when log format not matching with filter rule](https://discuss.elastic.co/t/high-cpu-usage-when-log-format-not-matching-with-filter-rule/254728)
**Author:** [@4orty](https://discuss.elastic.co/u/4orty)\ **Replies:** 1\ **Last updated:** [November 9, 2020, 3:57pm UTC](https://discuss.elastic.co/t/high-cpu-usage-when-log-format-not-matching-with-filter-rule/254728 "2020-11-09T15:57:00Z")
CPU utilization increases when data comes in that it does not meet the set rules.(grok pattern) Isn't there a defense logic to this phenomenon, such as preventing data from entering in case of failure of parsing? [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=282) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=284)