# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=671 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 672 --- ## [Logstash Hangs due to overload](https://discuss.elastic.co/t/logstash-hangs-due-to-overload/151215)
**Author:** [@Vignesh\_Kalaimani](https://discuss.elastic.co/u/Vignesh_Kalaimani)\ **Replies:** 6\ **Last updated:** [October 10, 2018, 10:48am UTC](https://discuss.elastic.co/t/logstash-hangs-due-to-overload/151215 "2018-10-10T10:48:31Z")
I have a Filebeat -\> Logstash -\> Elasticsearch setup where filebeat reads local log from VM and sends it to Logstash which in-turn indexes the log to Elasticsearch. The above setup worked well for a small environment. B… --- ## [How to filter numbers with comma and decimal as one](https://discuss.elastic.co/t/how-to-filter-numbers-with-comma-and-decimal-as-one/151798)
**Author:** [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\ **Replies:** 0\ **Last updated:** [October 10, 2018, 9:17am UTC](https://discuss.elastic.co/t/how-to-filter-numbers-with-comma-and-decimal-as-one/151798 "2018-10-10T09:17:20Z")
Hi Guys, Can you help me to solve my problem, I want to get this value of 1,000.00 as one. But, grook gave me this 1,%{JAVACLASS} The problem is, the value before the comma (,) is dynamic so basically it will change f… --- ## [Need help with grok and zabbix](https://discuss.elastic.co/t/need-help-with-grok-and-zabbix/151788)
**Author:** [@Franz](https://discuss.elastic.co/u/Franz)\ **Replies:** 0\ **Last updated:** [October 10, 2018, 8:18am UTC](https://discuss.elastic.co/t/need-help-with-grok-and-zabbix/151788 "2018-10-10T08:18:45Z")
Hello there, im trying to send some logs to zabbix with grok filter, i'm rly trying to understand what im doing but no luck at the moment :confused: This is the log im trying to filter: 2898880;2018-03-30;13:00:10;1;\[0… --- ## [Unable to create index from logstash to elastic search (beginner)](https://discuss.elastic.co/t/unable-to-create-index-from-logstash-to-elastic-search-beginner/151774)
**Author:** [@ajb](https://discuss.elastic.co/u/ajb)\ **Replies:** 0\ **Last updated:** [October 10, 2018, 7:31am UTC](https://discuss.elastic.co/t/unable-to-create-index-from-logstash-to-elastic-search-beginner/151774 "2018-10-10T07:31:21Z")
Hi team, Im unable to create index in the elastic search, im using 6.4.2 for the whole stack, and below is my config file, i believe im doing wrong somwhere in output section output to elasticsearch, as this is my firs… --- ## [Multiple logstash instances](https://discuss.elastic.co/t/multiple-logstash-instances/151776)
**Author:** [@venki522](https://discuss.elastic.co/u/venki522)\ **Replies:** 0\ **Last updated:** [October 10, 2018, 7:36am UTC](https://discuss.elastic.co/t/multiple-logstash-instances/151776 "2018-10-10T07:36:52Z")
i have created two logstash instances on same machine i was able to start the instances with assigning different ports for two instances. iam trying to achieve to give the ip address to both the instances but when i a… --- ## [Grok Pattern to extract brackets content](https://discuss.elastic.co/t/grok-pattern-to-extract-brackets-content/149802)
**Author:** [@gsiap5](https://discuss.elastic.co/u/gsiap5)\ **Replies:** 4\ **Last updated:** [October 10, 2018, 7:19am UTC](https://discuss.elastic.co/t/grok-pattern-to-extract-brackets-content/149802 "2018-10-10T07:19:56Z")
Hi , I am attempting to create a grok pattern to parse the following log file. 14:10:49:158017|5860-00088:JMIX: ChainSubscription {INFO} Action {Update} Chain {MOVE.AA\_MOVE\_TOP.ABC\_SWITCH\_COMP\_GO.SHOP\_ABC} Snapshot {fa… --- ## [Xpath output brings arrays instead of json](https://discuss.elastic.co/t/xpath-output-brings-arrays-instead-of-json/151768)
**Author:** [@balumurari1](https://discuss.elastic.co/u/balumurari1)\ **Replies:** 0\ **Last updated:** [October 10, 2018, 7:09am UTC](https://discuss.elastic.co/t/xpath-output-brings-arrays-instead-of-json/151768 "2018-10-10T07:09:49Z")
filter { xml { source =\> "message" target =\> "parsed" store\_xml =\> "false" xpath =\> \[ "Report/@Name", "cell\_value1", /Report/sr08/Report\[@Name='sr08'\]/matrix1\[@commodity\_header1='World'\] , "cell\_value2", .... \] … --- ## [How to create a new field by extracting data from already generated field?](https://discuss.elastic.co/t/how-to-create-a-new-field-by-extracting-data-from-already-generated-field/151386)
**Author:** [@pramodchoudhari](https://discuss.elastic.co/u/pramodchoudhari)\ **Replies:** 4\ **Last updated:** [October 10, 2018, 4:11am UTC](https://discuss.elastic.co/t/how-to-create-a-new-field-by-extracting-data-from-already-generated-field/151386 "2018-10-10T04:11:35Z")
This is how my source field looks like, the filename is in the format YYYYMMDD.HHMM.min so the above file was created in the year:- 2045 , month:- 02 ,date:- 18 , and time :- 09:54 what i want to do is from the ab… --- ## [Logstash query for IPs and GeoData](https://discuss.elastic.co/t/logstash-query-for-ips-and-geodata/151689)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 1\ **Last updated:** [October 10, 2018, 3:44am UTC](https://discuss.elastic.co/t/logstash-query-for-ips-and-geodata/151689 "2018-10-10T03:44:35Z")
Hi Team, Here is my logstash config file and would like to know that how do I put a filter for IP addresses? \*.out can contain domains, URLs, IP addresses or hashes. So as soon as logstash sees IP addresses geo data ha… --- ## [Grok Custom filter](https://discuss.elastic.co/t/grok-custom-filter/151523)
**Author:** [@zolthar-z](https://discuss.elastic.co/u/zolthar-z)\ **Replies:** 3\ **Last updated:** [October 10, 2018, 3:18am UTC](https://discuss.elastic.co/t/grok-custom-filter/151523 "2018-10-10T03:18:33Z")
Hi I'm new in ELK and I've installed the app for log management. I have a legacy application and want to make some tranformation to the logs, right now a I recevied the logs in the following format: 2018-10-08 13:06:28… --- ## [Enrich individual records with data from external REST calls](https://discuss.elastic.co/t/enrich-individual-records-with-data-from-external-rest-calls/151710)
**Author:** [@Sudhakar\_Madeti](https://discuss.elastic.co/u/Sudhakar_Madeti)\ **Replies:** 0\ **Last updated:** [October 9, 2018, 9:02pm UTC](https://discuss.elastic.co/t/enrich-individual-records-with-data-from-external-rest-calls/151710 "2018-10-09T21:02:39Z")
Hello, I am using Logstash to process some files using the File input plugin. The data consists of some some Key Value pairs and I am exporting that into the ElasticSearch. All works fine so far. Now my requirement is … --- ## [Send json to My URL from logstash](https://discuss.elastic.co/t/send-json-to-my-url-from-logstash/151305)
**Author:** [@robindeshwal](https://discuss.elastic.co/u/robindeshwal)\ **Replies:** 1\ **Last updated:** [October 9, 2018, 8:22pm UTC](https://discuss.elastic.co/t/send-json-to-my-url-from-logstash/151305 "2018-10-09T20:22:26Z")
Hi, I need to send the logs from nagios core to my API URL, I am using HTTP plugin for send the data. I am tring to send the json formatted data, but logstgash not send the data in json it send the data in the string. … --- ## [Logstash error to execute action](https://discuss.elastic.co/t/logstash-error-to-execute-action/151324)
**Author:** [@balumurari1](https://discuss.elastic.co/u/balumurari1)\ **Replies:** 1\ **Last updated:** [October 9, 2018, 8:20pm UTC](https://discuss.elastic.co/t/logstash-error-to-execute-action/151324 "2018-10-09T20:20:05Z")
when i am running the logstash getting the following error, Ignoring the 'pipelines.yml' file because modules or command line options are specified. Starting Logstash {"logstash.version"=\>"6.4.2"} Unknown setting 'hos… --- ## [Coverting timezone to corect format](https://discuss.elastic.co/t/coverting-timezone-to-corect-format/151304)
**Author:** [@punit\_j](https://discuss.elastic.co/u/punit_j)\ **Replies:** 2\ **Last updated:** [October 9, 2018, 8:18pm UTC](https://discuss.elastic.co/t/coverting-timezone-to-corect-format/151304 "2018-10-09T20:18:35Z")
Hi, I have date from logs in below format: 2018-10-06 04:25:43 EDT how can I convert it using date filter. I tried date { match =\> { "Date", "yyyy-MM-dd HH:mm:ss ZZZ" } target =\> \["@timestamp"\] } however it produ… --- ## [Parse this date format? (Oct 3 11:32:48)](https://discuss.elastic.co/t/parse-this-date-format-oct-3-1148/151275)
**Author:** [@jbrowe](https://discuss.elastic.co/u/jbrowe)\ **Replies:** 1\ **Last updated:** [October 9, 2018, 8:14pm UTC](https://discuss.elastic.co/t/parse-this-date-format-oct-3-1148/151275 "2018-10-09T20:14:14Z")
I can parse log file dates like: "Oct 3 11:32:48" with "MMM d HH:mm:ss" "Oct 03 11:32:48" with "MMM dd HH:mm:ss" Now i have: "Oct 3 11:32:48" (notice there are two spaces before the single digit month-day.) I… --- ## [How to use environment variable in pipelines.yml? (Windows)](https://discuss.elastic.co/t/how-to-use-environment-variable-in-pipelines-yml-windows/151704)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 0\ **Last updated:** [October 9, 2018, 8:07pm UTC](https://discuss.elastic.co/t/how-to-use-environment-variable-in-pipelines-yml-windows/151704 "2018-10-09T20:07:37Z")
Hi, as dev environment for the elastic stack I am using my laptop which runs windows. Since I have different paths and different operating systems on my laptop and on the live systems, I would like to use environment v… --- ## [Performance implication of multiple grok patterns for a single log file](https://discuss.elastic.co/t/performance-implication-of-multiple-grok-patterns-for-a-single-log-file/150088)
**Author:** [@asgs](https://discuss.elastic.co/u/asgs)\ **Replies:** 4\ **Last updated:** [October 9, 2018, 7:27pm UTC](https://discuss.elastic.co/t/performance-implication-of-multiple-grok-patterns-for-a-single-log-file/150088 "2018-10-09T19:27:13Z")
hello! we're discussing in our team whether it's OK to log two set of log entries to a single file and then having logstash config specify the two patterns (OR) it makes sense to direct each set of entries to its own … --- ## [How to Ignore some messages from Kafka topic when consuming using Logstash](https://discuss.elastic.co/t/how-to-ignore-some-messages-from-kafka-topic-when-consuming-using-logstash/151505)
**Author:** [@deeps](https://discuss.elastic.co/u/deeps)\ **Replies:** 7\ **Last updated:** [October 9, 2018, 6:48pm UTC](https://discuss.elastic.co/t/how-to-ignore-some-messages-from-kafka-topic-when-consuming-using-logstash/151505 "2018-10-09T18:48:59Z")
I have ten's of different messages in a kafka topic. I need to just consume messages that has only "America" in it. I have tried some suggestions available on the blog but no luck, could anyone help? Flow is App -\> Kafk… --- ## [Drop logs on the basis \_jsonparsefailure](https://discuss.elastic.co/t/drop-logs-on-the-basis-jsonparsefailure/151029)
**Author:** [@Bharat\_Gupta](https://discuss.elastic.co/u/Bharat_Gupta)\ **Replies:** 3\ **Last updated:** [October 9, 2018, 6:41pm UTC](https://discuss.elastic.co/t/drop-logs-on-the-basis-jsonparsefailure/151029 "2018-10-09T18:41:40Z")
"tags": \[ "beats\_input\_codec\_json\_applied", "\_jsonparsefailure" \] i want drop the log detail if get \_jsonparsefailure i have applied this inside filter :- if "\_jsonparsefailure" in \[tags\]{ drop { } } but still n… --- ## [Logstash error on start](https://discuss.elastic.co/t/logstash-error-on-start/151695)
**Author:** [@cmahan](https://discuss.elastic.co/u/cmahan)\ **Replies:** 0\ **Last updated:** [October 9, 2018, 5:54pm UTC](https://discuss.elastic.co/t/logstash-error-on-start/151695 "2018-10-09T17:54:12Z")
\[root@ip-172-16-10-12 logstash\]# bin/logstash -f first-pipeline.conf WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. C… --- ## [Flow replication with logstash-codec-\*flow?](https://discuss.elastic.co/t/flow-replication-with-logstash-codec-flow/151687)
**Author:** [@jlixfeld](https://discuss.elastic.co/u/jlixfeld)\ **Replies:** 0\ **Last updated:** [October 9, 2018, 4:57pm UTC](https://discuss.elastic.co/t/flow-replication-with-logstash-codec-flow/151687 "2018-10-09T16:57:10Z")
Hi! Wondering if there's a way for the \*flow-codecs to replicate (and forward) raw, incoming flows to another host? Thanks! --- ## [How to write quotes and double quotes in the same String](https://discuss.elastic.co/t/how-to-write-quotes-and-double-quotes-in-the-same-string/151465)
**Author:** [@Pierre2](https://discuss.elastic.co/u/Pierre2)\ **Replies:** 5\ **Last updated:** [October 9, 2018, 3:00pm UTC](https://discuss.elastic.co/t/how-to-write-quotes-and-double-quotes-in-the-same-string/151465 "2018-10-09T15:00:56Z")
Hello, I want to use the file output. The problem is that my String that I want to write in the output file contains quotes and double quote. I want to write this line : {"script": {"source": "ctx.\_source\['Categorie SV… --- ## [Handle filter exception in Logstash](https://discuss.elastic.co/t/handle-filter-exception-in-logstash/151623)
**Author:** [@adrianolimit](https://discuss.elastic.co/u/adrianolimit)\ **Replies:** 3\ **Last updated:** [October 9, 2018, 2:59pm UTC](https://discuss.elastic.co/t/handle-filter-exception-in-logstash/151623 "2018-10-09T14:59:39Z")
Hello elasticusers, I have a question about exceptions in the filter part of Logstash. This is my config file (I don't describe the "intput" and "output" parts) : input { file{} } filter { csv { ...} grok… --- ## [ELK stack issue with CDN data](https://discuss.elastic.co/t/elk-stack-issue-with-cdn-data/151640)
**Author:** [@satish\_aherkar](https://discuss.elastic.co/u/satish_aherkar)\ **Replies:** 0\ **Last updated:** [October 9, 2018, 1:38pm UTC](https://discuss.elastic.co/t/elk-stack-issue-with-cdn-data/151640 "2018-10-09T13:38:36Z")
Hi guys, I am having issue with ELK stack, specially with CDN data. The problem is event count is, let me explore. Basically, I've ELK setup on prod which has 6.3 version, when I'm starting the filebeat agent on CDN h… --- ## [Logstash has started but no elastic search index created in kibana](https://discuss.elastic.co/t/logstash-has-started-but-no-elastic-search-index-created-in-kibana/151497)
**Author:** [@Rakeshcse590](https://discuss.elastic.co/u/Rakeshcse590)\ **Replies:** 2\ **Last updated:** [October 9, 2018, 1:33pm UTC](https://discuss.elastic.co/t/logstash-has-started-but-no-elastic-search-index-created-in-kibana/151497 "2018-10-09T13:33:34Z")
Hi , I am trying to push csv data into elastic search db from log stash, I am able to see the message successfully started logstash, But elastic search index was not created. below is my logstash config file input { … --- ## [Disk Space gets full when file descriptors reaches to its limit](https://discuss.elastic.co/t/disk-space-gets-full-when-file-descriptors-reaches-to-its-limit/151636)
**Author:** [@meetdave2611997](https://discuss.elastic.co/u/meetdave2611997)\ **Replies:** 0\ **Last updated:** [October 9, 2018, 1:15pm UTC](https://discuss.elastic.co/t/disk-space-gets-full-when-file-descriptors-reaches-to-its-limit/151636 "2018-10-09T13:15:32Z")
Hi We are currently using logstash 6.2.4. Beats, TCP and HTTP inputs are being used. Logstash has default limit of max file descriptors around 16K. When open file descriptors reach to its max limit logstash stops workin… --- ## [Unable to push data to elastic search and Kibana through Logstash](https://discuss.elastic.co/t/unable-to-push-data-to-elastic-search-and-kibana-through-logstash/147544)
**Author:** [@shaheelkhan59](https://discuss.elastic.co/u/shaheelkhan59)\ **Replies:** 2\ **Last updated:** [October 9, 2018, 12:01pm UTC](https://discuss.elastic.co/t/unable-to-push-data-to-elastic-search-and-kibana-through-logstash/147544 "2018-10-09T12:01:58Z")
Hi all, I'm new to ELK. I need to use ELK for log analysis. But for getting a hands on experiencing I've started with a small .csv file. I'm using Elasticsearch 6.4, Kibana 6.4 and Logstash 6.4. I'm using Windows 8. I'… --- ## [Reading XML Attribute using XPATH in logstash](https://discuss.elastic.co/t/reading-xml-attribute-using-xpath-in-logstash/148833)
**Author:** [@Vivek\_Sharma](https://discuss.elastic.co/u/Vivek_Sharma)\ **Replies:** 3\ **Last updated:** [October 9, 2018, 12:00pm UTC](https://discuss.elastic.co/t/reading-xml-attribute-using-xpath-in-logstash/148833 "2018-10-09T12:00:27Z")
Hi I am trying to parse below XML file in logstash. \ \ORD12345\ \V11111\ \ I was able to successfully parse OrderNumber … --- ## [Unable to log CSV](https://discuss.elastic.co/t/unable-to-log-csv/148991)
**Author:** [@AdamD25](https://discuss.elastic.co/u/AdamD25)\ **Replies:** 2\ **Last updated:** [October 9, 2018, 11:54am UTC](https://discuss.elastic.co/t/unable-to-log-csv/148991 "2018-10-09T11:54:44Z")
Hi, I'm new to logstash so forgive any obvious mistakes! I've recently installed the ELK stack and have been trying to setup a pipeline to elasticsearch but have hit a wall. My config file (see below) seems to run witho… --- ## [S3 input assume role bug](https://discuss.elastic.co/t/s3-input-assume-role-bug/151606)
**Author:** [@shed7](https://discuss.elastic.co/u/shed7)\ **Replies:** 1\ **Last updated:** [October 9, 2018, 11:28am UTC](https://discuss.elastic.co/t/s3-input-assume-role-bug/151606 "2018-10-09T11:28:19Z")
Hi! I have an ec2 instance in eu-west-1a, an IAM role attached and I can see the temporary creds if I do: curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ I have a small Python program to test if… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=670) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=672)