# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=823 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 824 --- ## [Convert a hash map to array in order to split into multiple events](https://discuss.elastic.co/t/convert-a-hash-map-to-array-in-order-to-split-into-multiple-events/119529)
**Author:** [@antonis.mandilas](https://discuss.elastic.co/u/antonis.mandilas)\ **Replies:** 0\ **Last updated:** [February 12, 2018, 5:59pm UTC](https://discuss.elastic.co/t/convert-a-hash-map-to-array-in-order-to-split-into-multiple-events/119529 "2018-02-12T17:59:10Z")
Hello all, In my use case I have an XML file that contains elements that I have to convert to array of objects. Example: \ \ \ using the xml filter … --- ## [Ruby exception occurred: undefined method \`include?' for nil:NilClass](https://discuss.elastic.co/t/ruby-exception-occurred-undefined-method-include-for-nil-nilclass/119507)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 0\ **Last updated:** [February 12, 2018, 3:59pm UTC](https://discuss.elastic.co/t/ruby-exception-occurred-undefined-method-include-for-nil-nilclass/119507 "2018-02-12T15:59:47Z")
Hi, What is wrong with my code? Why am I getting this error? My Ruby code looks like that: message\_array = event.get("message").split("\\n\\n\[", -1) counter=0 message\_array.each do | index | counter+=1 if mes… --- ## [Write multiply csv files in logstash output](https://discuss.elastic.co/t/write-multiply-csv-files-in-logstash-output/119482)
**Author:** [@yosi\_ela](https://discuss.elastic.co/u/yosi_ela)\ **Replies:** 3\ **Last updated:** [February 12, 2018, 3:02pm UTC](https://discuss.elastic.co/t/write-multiply-csv-files-in-logstash-output/119482 "2018-02-12T15:02:53Z")
hii , i have an input in logstash of beats , and i am writing to csv file in the output . i want to make a new file of csv in every hour or in every day ? how i can to define logstash that the current file finish to … --- ## [Groking of cisco switch logs (non asa)](https://discuss.elastic.co/t/groking-of-cisco-switch-logs-non-asa/119485)
**Author:** [@ssi](https://discuss.elastic.co/u/ssi)\ **Replies:** 0\ **Last updated:** [February 12, 2018, 2:44pm UTC](https://discuss.elastic.co/t/groking-of-cisco-switch-logs-non-asa/119485 "2018-02-12T14:44:06Z")
Hi just wanted to give something back to the community here is the grok and pattern file that i got things working with. if there are a shorter/better way to do this please feel free to amend the post. I thought this wa… --- ## [Grok filter for application logs](https://discuss.elastic.co/t/grok-filter-for-application-logs/119479)
**Author:** [@Sridhar](https://discuss.elastic.co/u/Sridhar)\ **Replies:** 0\ **Last updated:** [February 12, 2018, 2:10pm UTC](https://discuss.elastic.co/t/grok-filter-for-application-logs/119479 "2018-02-12T14:10:04Z")
Hi all, I have a problem in reading the below logs log file 1 router#001(26322) 2018/02/09 00:00:17 -FKSPT- {2:2} Read from MQMNGR1P2:TSS\_MTHIGH\_WMQ\_IN\[wmqin(srid)\] \ unknown, id =\> (2)"-1", userref… --- ## [Find length of 'tag' attribute](https://discuss.elastic.co/t/find-length-of-tag-attribute/119415)
**Author:** [@dsakpal](https://discuss.elastic.co/u/dsakpal)\ **Replies:** 4\ **Last updated:** [February 12, 2018, 1:31pm UTC](https://discuss.elastic.co/t/find-length-of-tag-attribute/119415 "2018-02-12T13:31:29Z")
Hi, How to find length/size of 'tag' attribute in logstash(output section)? My scenario : All my errors are automatically puts into the 'tag' attribute by logstash & thats why I wanted to know the size of 'tag' attribu… --- ## [Upgrading to logstash 6.2](https://discuss.elastic.co/t/upgrading-to-logstash-6-2/119000)
**Author:** [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\ **Replies:** 2\ **Last updated:** [February 12, 2018, 5:18am UTC](https://discuss.elastic.co/t/upgrading-to-logstash-6-2/119000 "2018-02-12T05:18:02Z")
Hello I am trying to upgrade to logstash 6.2 but i am getting this error C:\\Users\\gagarwal3\\Downloads\\logstash-6.2.0\\logstash-6.2.0\\bin\>logstash "could not find jruby in C:\\Users\\gagarwal3\\Downloads\\logstash-6.2.0\\log… --- ## [How to pull data using redis and rabbit-mq also what are the pros and cons with either of them](https://discuss.elastic.co/t/how-to-pull-data-using-redis-and-rabbit-mq-also-what-are-the-pros-and-cons-with-either-of-them/119460)
**Author:** [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\ **Replies:** 0\ **Last updated:** [February 12, 2018, 11:38am UTC](https://discuss.elastic.co/t/how-to-pull-data-using-redis-and-rabbit-mq-also-what-are-the-pros-and-cons-with-either-of-them/119460 "2018-02-12T11:38:15Z")
We are defining an architecture to collect log information by Logstash shippers which are installed in various machines and index the data in one elasticsearch server centrally and use Kibana as the graphical layer. We n… --- ## [Problems with winlogbeat](https://discuss.elastic.co/t/problems-with-winlogbeat/119436)
**Author:** [@gaudi71240](https://discuss.elastic.co/u/gaudi71240)\ **Replies:** 10\ **Last updated:** [February 12, 2018, 10:46am UTC](https://discuss.elastic.co/t/problems-with-winlogbeat/119436 "2018-02-12T10:46:10Z")
Hello i have install ELK Stack in version 6.2 with centos when i want to send the logs of my windows server it goes this way at logstash service logs : 2018-02-09T10:20:40+01:00 ERR Failed to publish events: client is… --- ## [\_dateparsefailure apex logging with a '@' between date and time](https://discuss.elastic.co/t/dateparsefailure-apex-logging-with-a-between-date-and-time/118672)
**Author:** [@DannyM](https://discuss.elastic.co/u/DannyM)\ **Replies:** 4\ **Last updated:** [February 12, 2018, 10:00am UTC](https://discuss.elastic.co/t/dateparsefailure-apex-logging-with-a-between-date-and-time/118672 "2018-02-12T10:00:17Z")
Hello, I'm working on a logstash config but can't get a working date from the log line to @timestamp the paas\_apex\_access lines are successfully using a date line but the paas\_apex\_oblog won't. also when mutate the @ … --- ## [Update single fileds in a given document already stored in ES?](https://discuss.elastic.co/t/update-single-fileds-in-a-given-document-already-stored-in-es/119445)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 0\ **Last updated:** [February 12, 2018, 9:58am UTC](https://discuss.elastic.co/t/update-single-fileds-in-a-given-document-already-stored-in-es/119445 "2018-02-12T09:58:32Z")
Hi all, I would like to track timestamp and status of long running benchmarks in Elasticsearch. Start and ending of the run can be identified by logstash by it's input. Is it possible that logstash updates a single fie… --- ## [Help me pls Year from 2018 righ now 1475 what wrong?](https://discuss.elastic.co/t/help-me-pls-year-from-2018-righ-now-1475-what-wrong/119409)
**Author:** [@athlona64](https://discuss.elastic.co/u/athlona64)\ **Replies:** 2\ **Last updated:** [February 12, 2018, 8:26am UTC](https://discuss.elastic.co/t/help-me-pls-year-from-2018-righ-now-1475-what-wrong/119409 "2018-02-12T08:26:30Z")
i used logstash copy data from sybase to elasticsearch but not year output from 2018 become 1475 what is wrong ? Help me pls. --- ## [Can't parse netflow by logstash codec](https://discuss.elastic.co/t/cant-parse-netflow-by-logstash-codec/119429)
**Author:** [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Replies:** 0\ **Last updated:** [February 12, 2018, 8:17am UTC](https://discuss.elastic.co/t/cant-parse-netflow-by-logstash-codec/119429 "2018-02-12T08:17:32Z")
Hi everyone, I send the netflow to elasticsearch through logstash, but I found there are some fields not parsed. this is my logstash config: input{ udp{ host =\> "163.19.1X3.XX" p… --- ## [Customised Grok pattern](https://discuss.elastic.co/t/customised-grok-pattern/118748)
**Author:** [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\ **Replies:** 13\ **Last updated:** [February 12, 2018, 5:20am UTC](https://discuss.elastic.co/t/customised-grok-pattern/118748 "2018-02-12T05:20:15Z")
Hello I m new to ELK. Can someone help me how to create a customized grok pattern for my logs. Also is there any filter which I can use for (yyyy/mm/dd hh:mm:ss) format which I can use as my logs have this type of time… --- ## [Parse and ingest email files using logstack -- help needed](https://discuss.elastic.co/t/parse-and-ingest-email-files-using-logstack-help-needed/119141)
**Author:** [@abildgaard](https://discuss.elastic.co/u/abildgaard)\ **Replies:** 16\ **Last updated:** [February 9, 2018, 9:44pm UTC](https://discuss.elastic.co/t/parse-and-ingest-email-files-using-logstack-help-needed/119141 "2018-02-09T21:44:20Z")
I have multiple files each containing a single email in the following format: Message-ID: \<18435268.1075855378308.JavaMail.evans@thyme\> Date: Mon, 7 May 2001 12:28:00 -0700 (PDT) From: phillip@xxx.com To: matthew@xxx.co… --- ## [HAProxy / NGINX SYSLOG keep originating IP?](https://discuss.elastic.co/t/haproxy-nginx-syslog-keep-originating-ip/118894)
**Author:** [@devinacosta](https://discuss.elastic.co/u/devinacosta)\ **Replies:** 1\ **Last updated:** [February 11, 2018, 9:32pm UTC](https://discuss.elastic.co/t/haproxy-nginx-syslog-keep-originating-ip/118894 "2018-02-11T21:32:35Z")
We are looking to put either NGinx or HAProxy in front to provide some level of High Availability in regards to our syslog ingest. What we have noticed is to get syslog to work we use protocol TCP however all the message… --- ## [RabbtiMQ cluster with logstash cluster](https://discuss.elastic.co/t/rabbtimq-cluster-with-logstash-cluster/119392)
**Author:** [@anshul\_pandey](https://discuss.elastic.co/u/anshul_pandey)\ **Replies:** 1\ **Last updated:** [February 11, 2018, 9:26pm UTC](https://discuss.elastic.co/t/rabbtimq-cluster-with-logstash-cluster/119392 "2018-02-11T21:26:30Z")
Hi all, I am quite sure that this a solved problem, so trying to figure out the optimal way... My ELK stack is pretty basic, clients pushing message to RabbitMQ & Logstash pulls data (uses input plugin for RabbitMQ)… --- ## [How to pull data from Hadoop directly using logstash](https://discuss.elastic.co/t/how-to-pull-data-from-hadoop-directly-using-logstash/119386)
**Author:** [@yash\_mangla](https://discuss.elastic.co/u/yash_mangla)\ **Replies:** 0\ **Last updated:** [February 11, 2018, 7:04pm UTC](https://discuss.elastic.co/t/how-to-pull-data-from-hadoop-directly-using-logstash/119386 "2018-02-11T19:04:02Z")
Hello Folks, Anyone has an sample configuration to pull data from hadoop HDFS using logstash ? --- ## [Bin/logstash --modules netflow --setup not working](https://discuss.elastic.co/t/bin-logstash-modules-netflow-setup-not-working/119359)
**Author:** [@Junaid\_Khan](https://discuss.elastic.co/u/Junaid_Khan)\ **Replies:** 0\ **Last updated:** [February 11, 2018, 2:34am UTC](https://discuss.elastic.co/t/bin-logstash-modules-netflow-setup-not-working/119359 "2018-02-11T02:34:04Z")
Enabled netflow module for logstash and imported the netflow\* index but kibana visualisations and dashboard are not importing using this command from the instructions: bin/logstash --modules netflow --setup Receive bu… --- ## [Offline plugin installation](https://discuss.elastic.co/t/offline-plugin-installation/118667)
**Author:** [@bossi6of9](https://discuss.elastic.co/u/bossi6of9)\ **Replies:** 17\ **Last updated:** [February 10, 2018, 8:43am UTC](https://discuss.elastic.co/t/offline-plugin-installation/118667 "2018-02-10T08:43:52Z")
Hi, My test server does not have access to the internet, and when I try to install plugin locally, they fail. I see a fair amount of issues with this, but nothing showing a solution. Is there a way to do this? i try … --- ## [Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml](https://discuss.elastic.co/t/failed-to-read-pipelines-yaml-file-location-usr-share-logstash-config-pipelines-yml/119329)
**Author:** [@Vivek\_Nigam](https://discuss.elastic.co/u/Vivek_Nigam)\ **Replies:** 0\ **Last updated:** [February 10, 2018, 5:17am UTC](https://discuss.elastic.co/t/failed-to-read-pipelines-yaml-file-location-usr-share-logstash-config-pipelines-yml/119329 "2018-02-10T05:17:28Z")
unable to find pipelines.yml previously my logstash install on location /opt/logstash new version installed on /usr/share/logstash in new version i m unable to find pipelines.yml file. ERROR: Failed to read pipelines… --- ## [Logstash.yml not found](https://discuss.elastic.co/t/logstash-yml-not-found/119328)
**Author:** [@Vivek\_Nigam](https://discuss.elastic.co/u/Vivek_Nigam)\ **Replies:** 0\ **Last updated:** [February 10, 2018, 4:53am UTC](https://discuss.elastic.co/t/logstash-yml-not-found/119328 "2018-02-10T04:53:09Z")
while run ./logstash command getting following error: WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using… --- ## [Assistance with grok](https://discuss.elastic.co/t/assistance-with-grok/119319)
**Author:** [@Blake\_2112](https://discuss.elastic.co/u/Blake_2112)\ **Replies:** 1\ **Last updated:** [February 9, 2018, 11:31pm UTC](https://discuss.elastic.co/t/assistance-with-grok/119319 "2018-02-09T23:31:48Z")
I'm a bit stuck when determining a pattern to capture what I need from my logfile. To start, the logfile columns are variable. Thus, the line in the log could have 5 pieces of information, the next line 10. 2017/01/18 … --- ## [Trend Micro: Control Manager, Deep Security, Deep Discovery Inspector](https://discuss.elastic.co/t/trend-micro-control-manager-deep-security-deep-discovery-inspector/116940)
**Author:** [@GlennH](https://discuss.elastic.co/u/GlennH)\ **Replies:** 1\ **Last updated:** [February 9, 2018, 9:42pm UTC](https://discuss.elastic.co/t/trend-micro-control-manager-deep-security-deep-discovery-inspector/116940 "2018-02-09T21:42:34Z")
Im a bit new to elastic and spend too much time trying to figure out how to configure these so figured I would share so someone can either show me a better way or use this and save some time. FYI this thing still needs w… --- ## [Logstash 6.2.1 - unable to install unbundled filter plugin](https://discuss.elastic.co/t/logstash-6-2-1-unable-to-install-unbundled-filter-plugin/119292)
**Author:** [@ld\_pvl](https://discuss.elastic.co/u/ld_pvl)\ **Replies:** 3\ **Last updated:** [February 9, 2018, 9:10pm UTC](https://discuss.elastic.co/t/logstash-6-2-1-unable-to-install-unbundled-filter-plugin/119292 "2018-02-09T21:10:18Z")
Hi Team, I am trying to install an unbundled logstash plugin, namely logstash-filter-age. I've exported both http\_proxy and https\_proxy and verified that I can connect to the internet and ran this command to try to ins… --- ## [Pipelines.yml overwritten by upgrade](https://discuss.elastic.co/t/pipelines-yml-overwritten-by-upgrade/118903)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 1\ **Last updated:** [February 9, 2018, 9:05pm UTC](https://discuss.elastic.co/t/pipelines-yml-overwritten-by-upgrade/118903 "2018-02-09T21:05:03Z")
I just upgraded from 6.1.1 to 6.2. It is rather irritating that pipelines.yml gets overwritten. It would be nicer if the package created a .rpmnew file instead. --- ## [Does the multiline codec plugin in Logstash include newline characters in the output string?](https://discuss.elastic.co/t/does-the-multiline-codec-plugin-in-logstash-include-newline-characters-in-the-output-string/119278)
**Author:** [@daved](https://discuss.elastic.co/u/daved)\ **Replies:** 2\ **Last updated:** [February 9, 2018, 9:02pm UTC](https://discuss.elastic.co/t/does-the-multiline-codec-plugin-in-logstash-include-newline-characters-in-the-output-string/119278 "2018-02-09T21:02:35Z")
I'm wondering if the multiline codec plugin in Logstash includes newlines in the output? I want to use regex on the output and this is important to understand in order to design my regex and have it perform well. Lookin… --- ## [Send logs from secondary ip](https://discuss.elastic.co/t/send-logs-from-secondary-ip/119142)
**Author:** [@Geezer](https://discuss.elastic.co/u/Geezer)\ **Replies:** 1\ **Last updated:** [February 9, 2018, 7:34pm UTC](https://discuss.elastic.co/t/send-logs-from-secondary-ip/119142 "2018-02-09T19:34:16Z")
Hi, I need to send syslogs to a collector from Logstash but have it come from the secondary IP of the Logstash host. How can I specify the source IP that the Logstash will use to send the logs? I have tried using sourc… --- ## [Can't connect to ElasticSearch after X-pack install](https://discuss.elastic.co/t/cant-connect-to-elasticsearch-after-x-pack-install/119062)
**Author:** [@dyl](https://discuss.elastic.co/u/dyl)\ **Replies:** 8\ **Last updated:** [February 9, 2018, 2:57pm UTC](https://discuss.elastic.co/t/cant-connect-to-elasticsearch-after-x-pack-install/119062 "2018-02-09T14:57:53Z")
Hello. I just follow steps to configure x-pack with Logstash (\[Installing X-pack In Logsatsh\])(Installing X-Pack in Logstash | Logstash Reference \[6.2\] | Elastic) But after this, my logstash pipeline doesn't work anymor… --- ## [How to configure SSL connection between Logstash and Filebeat](https://discuss.elastic.co/t/how-to-configure-ssl-connection-between-logstash-and-filebeat/119252)
**Author:** [@itspecialistsimon](https://discuss.elastic.co/u/itspecialistsimon)\ **Replies:** 2\ **Last updated:** [February 9, 2018, 2:46pm UTC](https://discuss.elastic.co/t/how-to-configure-ssl-connection-between-logstash-and-filebeat/119252 "2018-02-09T14:46:57Z")
Hi everyone, I'm trying to configure the SSL connection between Logstash and Filebeat. Filebeat is working fine, but Logstash is not starting. Logstash.conf input { beats { port =\> 5044 ssl =\> true ssl\_… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=822) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=824)