# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=975 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 976 --- ## [Data are cut into logstash](https://discuss.elastic.co/t/data-are-cut-into-logstash/87493)
**Author:** [@SunDog](https://discuss.elastic.co/u/SunDog)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 11:02pm UTC](https://discuss.elastic.co/t/data-are-cut-into-logstash/87493 "2017-05-29T23:02:17Z")
Hello, I collect logs from 40 devices. Logs are read by filebeat from a text file, filebeat also adds a field with the device name (dev\_id). The contents of the file looks something like this: 23:53:15 SOME TEXT \*\*… --- ## [Logstash and multi filebeat index (different)](https://discuss.elastic.co/t/logstash-and-multi-filebeat-index-different/86413)
**Author:** [@Globule](https://discuss.elastic.co/u/Globule)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 8:17pm UTC](https://discuss.elastic.co/t/logstash-and-multi-filebeat-index-different/86413 "2017-05-29T20:17:52Z")
HI , I have set a wazuh ids server and a elk server a part... No problem with logstash and filebeat, no problem with kibana dashboard, but I have in logstash a config for parse filebeat wazuh log... and now I want to a… --- ## [Disallow duplicate key to be added on kv](https://discuss.elastic.co/t/disallow-duplicate-key-to-be-added-on-kv/86323)
**Author:** [@Justin\_Cross](https://discuss.elastic.co/u/Justin_Cross)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 8:15pm UTC](https://discuss.elastic.co/t/disallow-duplicate-key-to-be-added-on-kv/86323 "2017-05-29T20:15:52Z")
I need to stop logstash from adding to ReceiverInfo more than once. msg\_attributes = ReceiverInfo=111111111,ReceiverInfo=22222222,ReceiverInfo=3333333333 It should add the first key/value and then no more. What I en… --- ## [Loading records with date\_range field from Logstash](https://discuss.elastic.co/t/loading-records-with-date-range-field-from-logstash/86299)
**Author:** [@chrisbeck](https://discuss.elastic.co/u/chrisbeck)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 8:12pm UTC](https://discuss.elastic.co/t/loading-records-with-date-range-field-from-logstash/86299 "2017-05-29T20:12:29Z")
I have a set of data with a start date/time and an end date/time. I would like to create a date\_range field via a filter clause in logstash as part of inserting the data. has anyone been able to do this? Thanks, Chr… --- ## [How to store array in elasticsearch from logstash](https://discuss.elastic.co/t/how-to-store-array-in-elasticsearch-from-logstash/87445)
**Author:** [@abhi\_chinchole](https://discuss.elastic.co/u/abhi_chinchole)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 7:48pm UTC](https://discuss.elastic.co/t/how-to-store-array-in-elasticsearch-from-logstash/87445 "2017-05-29T19:48:09Z")
I am trying to store data in array field of elasticsearch from logstash Raw JSON event : {"user-metadata":{"tags":\["BCT-LTP-G109","Elasticsearch","logstash"\]},"data":"abc def xyz"} I want to take "tags" field out as… --- ## [How do I re-index when I'm using logstash to apply an index template](https://discuss.elastic.co/t/how-do-i-re-index-when-im-using-logstash-to-apply-an-index-template/87294)
**Author:** [@red888](https://discuss.elastic.co/u/red888)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 7:02pm UTC](https://discuss.elastic.co/t/how-do-i-re-index-when-im-using-logstash-to-apply-an-index-template/87294 "2017-05-29T19:02:41Z")
This is the output section of my logstash config: output { if \[type\] == "syslog" { elasticsearch { hosts =\> myserver index =\> "logstash-syslog-%{+YYYY-MM-dd}" template\_name =\> "MyTemplateIWantTo… --- ## [Conditional Output Not Working](https://discuss.elastic.co/t/conditional-output-not-working/87477)
**Author:** [@bjmaynard01](https://discuss.elastic.co/u/bjmaynard01)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 7:00pm UTC](https://discuss.elastic.co/t/conditional-output-not-working/87477 "2017-05-29T19:00:38Z")
I have come across an interesting problem that is causing me to bash my head against the wall. I'm trying to parse nginx access logs and eventually dump them into ES. However, for testing purposes, I am using stdout. in … --- ## [Parse a logfile](https://discuss.elastic.co/t/parse-a-logfile/87284)
**Author:** [@aqsyounas](https://discuss.elastic.co/u/aqsyounas)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 6:49pm UTC](https://discuss.elastic.co/t/parse-a-logfile/87284 "2017-05-29T18:49:15Z")
Hello Community, Pardon me for my naive question i am new to elasticsearch and this is my first time posting to this mailing list. i am try to parse a log file which contains data in format given below. connection… --- ## [Grok Debug (Workflow) with Logstash 5.4.0](https://discuss.elastic.co/t/grok-debug-workflow-with-logstash-5-4-0/87175)
**Author:** [@felixhummel](https://discuss.elastic.co/u/felixhummel)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 6:45pm UTC](https://discuss.elastic.co/t/grok-debug-workflow-with-logstash-5-4-0/87175 "2017-05-29T18:45:55Z")
Hi! What is the best workflow when developing Grok filters? The docs advise to use http://grokdebug.herokuapp.com/. Unfortunately the patterns on grokdebug seem outdated. They do not match logstash-patterns 4.0.2 t… --- ## [How to parse dd MMM HH:mm:ss IST date format?](https://discuss.elastic.co/t/how-to-parse-dd-mmm-hh-mm-ss-ist-date-format/87172)
**Author:** [@vikas\_gopal](https://discuss.elastic.co/u/vikas_gopal)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 6:42pm UTC](https://discuss.elastic.co/t/how-to-parse-dd-mmm-hh-mm-ss-ist-date-format/87172 "2017-05-29T18:42:10Z")
Hi Experts, I have date value like 15 May 2017 11:52:59 IST Here I am not sure how to handle IST Please suggest in Logstash what date format I can apply on this value . I used date {match =\> \["endTime","dd MMM YYYY H… --- ## [Logstash 5.4 Plugins - Unable to install plugins - Errors with httparty](https://discuss.elastic.co/t/logstash-5-4-plugins-unable-to-install-plugins-errors-with-httparty/87013)
**Author:** [@techpanga](https://discuss.elastic.co/u/techpanga)\ **Replies:** 3\ **Last updated:** [May 29, 2017, 6:14pm UTC](https://discuss.elastic.co/t/logstash-5-4-plugins-unable-to-install-plugins-errors-with-httparty/87013 "2017-05-29T18:14:20Z")
Logstash - 5.4, Ruby 1.9 (default comes with logstash) I do see below errors for DEBUG=1 bin/logstash-plugin update. Due to these errors (httparty), I am unable to install any plugins like email, hipchat etc. Gem::… --- ## [Grok for parsing java Log](https://discuss.elastic.co/t/grok-for-parsing-java-log/43718)
**Author:** [@sankallada](https://discuss.elastic.co/u/sankallada)\ **Replies:** 8\ **Last updated:** [May 29, 2017, 5:48pm UTC](https://discuss.elastic.co/t/grok-for-parsing-java-log/43718 "2017-05-29T17:48:35Z")
Hi, I'm learning ELK for one of my upcoming implementation for the past few days. I have created a logstash conf which sends the data to my elasticsearch. With my logstash conf, I'm expecting a few fields to be added i… --- ## [Need help how to parse such log from Kubernetes nginx pod](https://discuss.elastic.co/t/need-help-how-to-parse-such-log-from-kubernetes-nginx-pod/86942)
**Author:** [@Igor\_Gerasimow](https://discuss.elastic.co/u/Igor_Gerasimow)\ **Replies:** 3\ **Last updated:** [May 29, 2017, 3:56pm UTC](https://discuss.elastic.co/t/need-help-how-to-parse-such-log-from-kubernetes-nginx-pod/86942 "2017-05-29T15:56:20Z")
Hello - desperately asking for help with logs from nginx on kubernetes. {"log":"10.128.0.21 - - \[24/May/2017:10:12:32 +0000\] \\"GET /pp/apiv2/personprofile/check\_summ/na/A%20Bird%20of%20Prey%20 HTTP/1.1\\" 200 259 \\"-… --- ## [JSon Array parsing - Field name](https://discuss.elastic.co/t/json-array-parsing-field-name/32870)
**Author:** [@apoc](https://discuss.elastic.co/u/apoc)\ **Replies:** 3\ **Last updated:** [May 29, 2017, 1:41pm UTC](https://discuss.elastic.co/t/json-array-parsing-field-name/32870 "2017-05-29T13:41:34Z")
Hi, I manage to parse JSON array with logstash (Returns/Row) My configuration is ... filter { split { field =\> "Returns\[Row\]" add\_field =\> \[ "name" , "%{Returns.Row.name}"\] } } output { stdout { codec =\> "rub… --- ## [A n00b question about sections in configuration files](https://discuss.elastic.co/t/a-n00b-question-about-sections-in-configuration-files/87286)
**Author:** [@LinuxDan](https://discuss.elastic.co/u/LinuxDan)\ **Replies:** 4\ **Last updated:** [May 29, 2017, 1:38pm UTC](https://discuss.elastic.co/t/a-n00b-question-about-sections-in-configuration-files/87286 "2017-05-29T13:38:25Z")
I have seen examples with input, filter, and output sections in separate files and others with them grouped into fewer files. Does the behavior change when you group config sections into one file ? ... when you sepa… --- ## [Logstash config for using filebeat auditd module](https://discuss.elastic.co/t/logstash-config-for-using-filebeat-auditd-module/87033)
**Author:** [@jclose](https://discuss.elastic.co/u/jclose)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 1:33pm UTC](https://discuss.elastic.co/t/logstash-config-for-using-filebeat-auditd-module/87033 "2017-05-29T13:33:52Z")
We would like to use filebeat to collect audit logs across several systems. We put our Elastic cluster behind a security VLAN, and don't want to open up ports to the public. So the best solution here is to simply pu… --- ## [\[netflow\] new sources are not picked up immediately](https://discuss.elastic.co/t/netflow-new-sources-are-not-picked-up-immediately/87442)
**Author:** [@massimo](https://discuss.elastic.co/u/massimo)\ **Replies:** 0\ **Last updated:** [May 29, 2017, 11:47am UTC](https://discuss.elastic.co/t/netflow-new-sources-are-not-picked-up-immediately/87442 "2017-05-29T11:47:50Z")
Hello, there had already been several devices configured to send netflow information to Logstash. Today we added a few more sources but they were only picked up/recognised after reloading logstash's configuration file a… --- ## [Merging information from two log files](https://discuss.elastic.co/t/merging-information-from-two-log-files/28914)
**Author:** [@Navneet\_Mathpal](https://discuss.elastic.co/u/Navneet_Mathpal)\ **Replies:** 4\ **Last updated:** [May 29, 2017, 11:34am UTC](https://discuss.elastic.co/t/merging-information-from-two-log-files/28914 "2017-05-29T11:34:35Z")
Hi , I need to merge the information from two log file ex: From the first log file I am getting some log information and second log file I am getting ID , I want to append that ID with each logs coming from first file.… --- ## [Duplicate rows/events according to number of config files](https://discuss.elastic.co/t/duplicate-rows-events-according-to-number-of-config-files/87403)
**Author:** [@tmoeller](https://discuss.elastic.co/u/tmoeller)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 10:29am UTC](https://discuss.elastic.co/t/duplicate-rows-events-according-to-number-of-config-files/87403 "2017-05-29T10:29:05Z")
Hi, i know this hats been discussed more than once, but no solutions seems to apply to my problem. I have a very basic installation with logstash 5.4 and redis 3.2.3 on a test VM (CentOS 7). It is using the jdbc plug… --- ## [Protobuf plugin installation on Logstash](https://discuss.elastic.co/t/protobuf-plugin-installation-on-logstash/86996)
**Author:** [@sgarap](https://discuss.elastic.co/u/sgarap)\ **Replies:** 0\ **Last updated:** [May 24, 2017, 3:04pm UTC](https://discuss.elastic.co/t/protobuf-plugin-installation-on-logstash/86996 "2017-05-24T15:04:25Z")
Hi, We have logstash 5.2 installed on a server which doesn't have internet access. We want to install the logstash-codec-protobuf plugin. We don't even have a staging server with internet access. So using 'Offline Plug… --- ## [Logstash adds field to output although not specified](https://discuss.elastic.co/t/logstash-adds-field-to-output-although-not-specified/86927)
**Author:** [@massimo](https://discuss.elastic.co/u/massimo)\ **Replies:** 2\ **Last updated:** [May 29, 2017, 9:18am UTC](https://discuss.elastic.co/t/logstash-adds-field-to-output-although-not-specified/86927 "2017-05-29T09:18:30Z")
Hello, in a test environment we collect, parse/grok firewall log messages and put them into elasticsearch - using logstash. The logs are received using tcp input. In the processed logs I noticed another field which was… --- ## [Determine when to stop capturing event and start another one](https://discuss.elastic.co/t/determine-when-to-stop-capturing-event-and-start-another-one/86078)
**Author:** [@sahar\_q](https://discuss.elastic.co/u/sahar_q)\ **Replies:** 4\ **Last updated:** [May 29, 2017, 8:07am UTC](https://discuss.elastic.co/t/determine-when-to-stop-capturing-event-and-start-another-one/86078 "2017-05-29T08:07:09Z")
Hi guys, say im sending an data to log stahs through http, the data is separated in commas and each line represent and single event (that i want it to be independent as a document) Data for example: "192.168.0.1","25… --- ## [Setting a timestamp from a JSON parsed object](https://discuss.elastic.co/t/setting-a-timestamp-from-a-json-parsed-object/86910)
**Author:** [@Vedran\_Maricevic](https://discuss.elastic.co/u/Vedran_Maricevic)\ **Replies:** 2\ **Last updated:** [May 29, 2017, 7:21am UTC](https://discuss.elastic.co/t/setting-a-timestamp-from-a-json-parsed-object/86910 "2017-05-29T07:21:17Z")
I am having an issue with setting a timestamp from a JSON parse. I have this string (valid JSON): \[{"orderNumber":"423523-4325-3212-4235-463a72e76fe8","externalOrderNumber":"reactivate\_22d6ff0d8f55eb821be14df9d35505a6… --- ## [Sending unique filename pattern to unique Index?](https://discuss.elastic.co/t/sending-unique-filename-pattern-to-unique-index/85683)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 3\ **Last updated:** [May 29, 2017, 6:30am UTC](https://discuss.elastic.co/t/sending-unique-filename-pattern-to-unique-index/85683 "2017-05-29T06:30:09Z")
Hi, How can I send different log files to different indexs? For example: I read with the "filebeat" a file called: abc.full.log and a file called xyz.cdr and send both to LogStash. How can I make that abc.full.log w… --- ## [How to handle logstash with multiple config file?](https://discuss.elastic.co/t/how-to-handle-logstash-with-multiple-config-file/87162)
**Author:** [@Dheerajg](https://discuss.elastic.co/u/Dheerajg)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 5:26am UTC](https://discuss.elastic.co/t/how-to-handle-logstash-with-multiple-config-file/87162 "2017-05-29T05:26:32Z")
I have almost 20 config files from different servers, each config file is creating to a separate index. How can I handle the config files here, I have placed all the config files in the /etc/logstash/conf.d/ folder and s… --- ## [\_grokparsefailure tag not removed](https://discuss.elastic.co/t/-grokparsefailure-tag-not-removed/87292)
**Author:** [@mikygee](https://discuss.elastic.co/u/mikygee)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 5:21am UTC](https://discuss.elastic.co/t/-grokparsefailure-tag-not-removed/87292 "2017-05-29T05:21:17Z")
Hello, I'm strugling to get rid of these \_grokparsefailure tags Case 1: Message 1.1.1.1 - - \[26/May/2017:21:24:04 +0200\] "GET /plugins/jqueryui/themes/classic/jquery-ui-1.10.4.custom.css?s=1450862292 HTTP/1.1" 200 62… --- ## [How to change all dates to a supported format](https://discuss.elastic.co/t/how-to-change-all-dates-to-a-supported-format/87381)
**Author:** [@ideal](https://discuss.elastic.co/u/ideal)\ **Replies:** 1\ **Last updated:** [May 29, 2017, 2:56am UTC](https://discuss.elastic.co/t/how-to-change-all-dates-to-a-supported-format/87381 "2017-05-29T02:56:19Z")
I'm sending JSON to logstash and it has about a dozen fields with dates in this format: yyyy-MM-dd HH:mm:ss That date format generates an error because it is missing a "T" at the start of the time. The error logstash g… --- ## [Grok patterns for syslog configuration in logstash](https://discuss.elastic.co/t/grok-patterns-for-syslog-configuration-in-logstash/86938)
**Author:** [@elbesraoui\_imane](https://discuss.elastic.co/u/elbesraoui_imane)\ **Replies:** 3\ **Last updated:** [May 28, 2017, 6:56pm UTC](https://discuss.elastic.co/t/grok-patterns-for-syslog-configuration-in-logstash/86938 "2017-05-28T18:56:30Z")
Hi,I recentely started working with the ELK stack but I have a problem for my syslog logstash configuration. Can you give me hand pleaase ? I don't know what's the problem I'm having in my config file, but when i do th… --- ## [Problem access to value of some data](https://discuss.elastic.co/t/problem-access-to-value-of-some-data/86943)
**Author:** [@Gudo](https://discuss.elastic.co/u/Gudo)\ **Replies:** 2\ **Last updated:** [May 28, 2017, 6:55pm UTC](https://discuss.elastic.co/t/problem-access-to-value-of-some-data/86943 "2017-05-28T18:55:19Z")
Hello everyone! i have problem to access some date my output is like that: "alternance" =\> \[ \[0\] { "country" =\> "value" } i made alter filter and also mutate to access value but what i see… --- ## [Logstash pipeline filter by date](https://discuss.elastic.co/t/logstash-pipeline-filter-by-date/86969)
**Author:** [@ezequielo](https://discuss.elastic.co/u/ezequielo)\ **Replies:** 1\ **Last updated:** [May 28, 2017, 6:53pm UTC](https://discuss.elastic.co/t/logstash-pipeline-filter-by-date/86969 "2017-05-28T18:53:50Z")
Hi all, I have created a logstash pipeline that fetches data from a database (MySQL) and load it into ES. So it basically consists of a select over a couple of tables and output it to ES. I have noticed that lately th… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=974) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=976)