# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1072 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1073 --- ## [How to config logstash to listen privilage port without root access](https://discuss.elastic.co/t/how-to-config-logstash-to-listen-privilage-port-without-root-access/67080)
**Author:** [@bopa](https://discuss.elastic.co/u/bopa)\ **Replies:** 1\ **Last updated:** [November 28, 2016, 8:14am UTC](https://discuss.elastic.co/t/how-to-config-logstash-to-listen-privilage-port-without-root-access/67080 "2016-11-28T08:14:46Z")
I want to collect syslogs from cisco switch which will use UDP 514 to send syslogs. I tried setcap command and port forwarding on firewall. But failed. I used following commands, setcap - setcap cap\_net\_bind\_service=… --- ## [LS 2.3 -\> 5.0 TLS trouble, PKCS#8 and cipher\_suites](https://discuss.elastic.co/t/ls-2-3-5-0-tls-trouble-pkcs-8-and-cipher-suites/67309)
**Author:** [@gregoryo](https://discuss.elastic.co/u/gregoryo)\ **Replies:** 0\ **Last updated:** [November 28, 2016, 7:20am UTC](https://discuss.elastic.co/t/ls-2-3-5-0-tls-trouble-pkcs-8-and-cipher-suites/67309 "2016-11-28T07:20:57Z")
On FreeBSD 10, I had TLS working for encrypted communication between previous versions of Logstash and both Filebeat and Metricbeat. I've recently upgraded ELK to the versions below: Logstash 2.3 -\> 5.0 Filebeat 1.2 … --- ## [How to overwrite Log time values with @timestamp](https://discuss.elastic.co/t/how-to-overwrite-log-time-values-with-timestamp/66689)
**Author:** [@Vivin\_Pai](https://discuss.elastic.co/u/Vivin_Pai)\ **Replies:** 2\ **Last updated:** [November 28, 2016, 7:05am UTC](https://discuss.elastic.co/t/how-to-overwrite-log-time-values-with-timestamp/66689 "2016-11-28T07:05:01Z")
I have created a conf file for Logstash. input { file { path =\> "/advdata/tmp/Error\_Trace\_blrhpbl\_SC1\*.txt" start\_position =\> "beginning" } } filter { date { match =\> \[ "logdate",… --- ## [Logstash disconnects beat after 15 seconds](https://discuss.elastic.co/t/logstash-disconnects-beat-after-15-seconds/67187)
**Author:** [@raja3000](https://discuss.elastic.co/u/raja3000)\ **Replies:** 1\ **Last updated:** [November 28, 2016, 7:04am UTC](https://discuss.elastic.co/t/logstash-disconnects-beat-after-15-seconds/67187 "2016-11-28T07:04:55Z")
I am using filebeat and a custom beat based 5.0.1 and logstash with logstash-input-beat (v3.1.8). Filebeat is working fine. But I am seeing following issues with my custom beat 2016-11-25T11:06:21Z ERR Sync. Failed … --- ## [Kibana shows Multiple entries for the same tag](https://discuss.elastic.co/t/kibana-shows-multiple-entries-for-the-same-tag/67307)
**Author:** [@Vivin\_Pai](https://discuss.elastic.co/u/Vivin_Pai)\ **Replies:** 0\ **Last updated:** [November 28, 2016, 7:04am UTC](https://discuss.elastic.co/t/kibana-shows-multiple-entries-for-the-same-tag/67307 "2016-11-28T07:04:02Z")
Hello All, I have written 2 conf files for Log stash and placed under the path /etc/logstash/conf.d 1st conf file \[root@blrvmelk-01 conf.d\]# cat logstash-Error1.conf input { file { path =\> "/advdata/tmp/… --- ## [Grok parsing for multiline logs (winlogbeat)](https://discuss.elastic.co/t/grok-parsing-for-multiline-logs-winlogbeat/67302)
**Author:** [@Houss](https://discuss.elastic.co/u/Houss)\ **Replies:** 2\ **Last updated:** [November 28, 2016, 6:31am UTC](https://discuss.elastic.co/t/grok-parsing-for-multiline-logs-winlogbeat/67302 "2016-11-28T06:31:28Z")
Hello, I'm having problem making the grok parsing work for multiline log. I've tried \\n or \\r or both and can't seem to make it work. Here's an example : Niveau d’emprunt d’identité : Emprunt d’identité Nouvelle o… --- ## [GROK filter with regular expression not working](https://discuss.elastic.co/t/grok-filter-with-regular-expression-not-working/67211)
**Author:** [@Hans](https://discuss.elastic.co/u/Hans)\ **Replies:** 3\ **Last updated:** [November 28, 2016, 6:30am UTC](https://discuss.elastic.co/t/grok-filter-with-regular-expression-not-working/67211 "2016-11-28T06:30:10Z")
Hi, I have a regular expression in a GROK filter, where the first filter is working however the next 5 filters are not working. Could someone kindly look at the filter and possibly point out where I am going wrong? gro… --- ## [Logstash date extraction help](https://discuss.elastic.co/t/logstash-date-extraction-help/67010)
**Author:** [@FrankC](https://discuss.elastic.co/u/FrankC)\ **Replies:** 2\ **Last updated:** [November 28, 2016, 4:33am UTC](https://discuss.elastic.co/t/logstash-date-extraction-help/67010 "2016-11-28T04:33:09Z")
Hello, I am stuck with logstash and need a little help. Each line starts with the event time stamp is multiline. Message: Tue Nov 01 00:07:24 2016 Thread 1 advanced to log sequence 1234 (foo) Current log# 2 seq#… --- ## [Convert one filed from CSV column header to time stamp when loading csv to elasticsearch](https://discuss.elastic.co/t/convert-one-filed-from-csv-column-header-to-time-stamp-when-loading-csv-to-elasticsearch/67220)
**Author:** [@shilong](https://discuss.elastic.co/u/shilong)\ **Replies:** 2\ **Last updated:** [November 28, 2016, 5:02am UTC](https://discuss.elastic.co/t/convert-one-filed-from-csv-column-header-to-time-stamp-when-loading-csv-to-elasticsearch/67220 "2016-11-28T05:02:31Z")
no. count sub-category ranking date 1 23 sd N 2016-10-23 The above is a simple snippet for CSV data. I use logstash to ship the data to elasticsearch vis csv filter. I want to make date as the timestamp on ela… --- ## [AmazonLogstashPlugin to AmazonElasticsearch documentId question](https://discuss.elastic.co/t/amazonlogstashplugin-to-amazonelasticsearch-documentid-question/34550)
**Author:** [@skanjila](https://discuss.elastic.co/u/skanjila)\ **Replies:** 3\ **Last updated:** [November 28, 2016, 4:26am UTC](https://discuss.elastic.co/t/amazonlogstashplugin-to-amazonelasticsearch-documentid-question/34550 "2016-11-28T04:26:12Z")
Hello Folks, First time trying to do this logstash workflow so please bear with me, I am trying to setup the logstash dynamodb plugin (http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.DynamoDBLogstas… --- ## [How to deal with the window when logstash is down and logs may be lost](https://discuss.elastic.co/t/how-to-deal-with-the-window-when-logstash-is-down-and-logs-may-be-lost/67163)
**Author:** [@wangshuogr](https://discuss.elastic.co/u/wangshuogr)\ **Replies:** 1\ **Last updated:** [November 28, 2016, 2:55am UTC](https://discuss.elastic.co/t/how-to-deal-with-the-window-when-logstash-is-down-and-logs-may-be-lost/67163 "2016-11-28T02:55:37Z")
The situation is like this. I run a logstash docker to monitor the log file example.log and logstash consumes the logs to Elasticsearch. There is a rotation for log files. When the file is large enough it is archived an… --- ## [Nested JSON for hostname with packetbeat & logstash](https://discuss.elastic.co/t/nested-json-for-hostname-with-packetbeat-logstash/66761)
**Author:** [@tetsuto](https://discuss.elastic.co/u/tetsuto)\ **Replies:** 2\ **Last updated:** [November 27, 2016, 8:45pm UTC](https://discuss.elastic.co/t/nested-json-for-hostname-with-packetbeat-logstash/66761 "2016-11-27T20:45:53Z")
I've been trying for quite some time to get this to work with no luck - I'm using the Packetbeat plugin to record http data (and headers). At present it's getting them currently - however the host is nested within the J… --- ## [Reading from a deleted file?](https://discuss.elastic.co/t/reading-from-a-deleted-file/67207)
**Author:** [@gcsaba2](https://discuss.elastic.co/u/gcsaba2)\ **Replies:** 1\ **Last updated:** [November 27, 2016, 8:43pm UTC](https://discuss.elastic.co/t/reading-from-a-deleted-file/67207 "2016-11-27T20:43:28Z")
Hello, I'm curious about one scenario on linux. As you know, on linux if I have a file with two processes having a handle to it, then the file won't get deleted permanently until both handles are unlinked. Meaning: if … --- ## [Any hardware requirements for logstash and beats?](https://discuss.elastic.co/t/any-hardware-requirements-for-logstash-and-beats/67273)
**Author:** [@shilong](https://discuss.elastic.co/u/shilong)\ **Replies:** 1\ **Last updated:** [November 27, 2016, 8:35pm UTC](https://discuss.elastic.co/t/any-hardware-requirements-for-logstash-and-beats/67273 "2016-11-27T20:35:38Z")
For some reason, I use windows to run logstash and might be able to get stuck on a few of things to deal with, things like sincedb\_path and performance issue. I am wondering how to run logstash in the back with a powe… --- ## [Suricata dashboard on 5.0 - Ruby issue](https://discuss.elastic.co/t/suricata-dashboard-on-5-0-ruby-issue/66946)
**Author:** [@Nick\_Minter](https://discuss.elastic.co/u/Nick_Minter)\ **Replies:** 1\ **Last updated:** [November 27, 2016, 8:25pm UTC](https://discuss.elastic.co/t/suricata-dashboard-on-5-0-ruby-issue/66946 "2016-11-27T20:25:52Z")
Hi All, On the new version of Logstash the ruby field references have changed. As such the ruby code in the Logstash config here: https://redmine.openinfosecfoundation.org/projects/suricata/wiki/\_Logstash\_Kibana\_and\_… --- ## [Errno::ENOENT: No such file or directory - /var/app/ondeck](https://discuss.elastic.co/t/errno-enoent-no-such-file-or-directory-var-app-ondeck/66945)
**Author:** [@khteh](https://discuss.elastic.co/u/khteh)\ **Replies:** 1\ **Last updated:** [November 27, 2016, 8:24pm UTC](https://discuss.elastic.co/t/errno-enoent-no-such-file-or-directory-var-app-ondeck/66945 "2016-11-27T20:24:36Z")
I deploy logstash 5.0.1 on AWS Elastic Beanstalk. Unfortunately, logstash fails to start up with the following error message: Errno::ENOENT: No such file or directory - /var/app/ondeck chdir at org/jr… --- ## [Codec multiline with redis](https://discuss.elastic.co/t/codec-multiline-with-redis/66827)
**Author:** [@nick76](https://discuss.elastic.co/u/nick76)\ **Replies:** 3\ **Last updated:** [November 27, 2016, 8:09pm UTC](https://discuss.elastic.co/t/codec-multiline-with-redis/66827 "2016-11-27T20:09:17Z")
Hello all, in my setup I had to use the redis aggregator: varnish+apache+tomcat (filebeat) --\> redis \<-- |fw|-- logstash 5.0 in this setup I got different log types from different sources that need to be parsed correctly… --- ## [Orcale (XMLTYPE) attribute not indexing in elasticsearch, logstash filter not working](https://discuss.elastic.co/t/orcale-xmltype-attribute-not-indexing-in-elasticsearch-logstash-filter-not-working/66858)
**Author:** [@shamimhaider33](https://discuss.elastic.co/u/shamimhaider33)\ **Replies:** 1\ **Last updated:** [November 27, 2016, 8:05pm UTC](https://discuss.elastic.co/t/orcale-xmltype-attribute-not-indexing-in-elasticsearch-logstash-filter-not-working/66858 "2016-11-27T20:05:09Z")
I have been trying to parse and index this (XMLTYPE) attribute (ilmd) which I am getting from a Oracle DB when I fire this query, SELECT event\_event\_dt, event\_event\_tm, ilmd FROM event Here is the xml document which is… --- ## [Logstash/Grok not processing part of a custom filter](https://discuss.elastic.co/t/logstash-grok-not-processing-part-of-a-custom-filter/66559)
**Author:** [@brian.jones](https://discuss.elastic.co/u/brian.jones)\ **Replies:** 2\ **Last updated:** [November 27, 2016, 8:01pm UTC](https://discuss.elastic.co/t/logstash-grok-not-processing-part-of-a-custom-filter/66559 "2016-11-27T20:01:32Z")
Not sure if I am missing something, or just going about this all wrong, but some of the syslog events from our ASA just don't seem to filter/parse correctly. Relevant syslog events: "message":"\<150\>Nov 18 2016 15:50:2… --- ## [I want to pay for getting my data onto Kibana,](https://discuss.elastic.co/t/i-want-to-pay-for-getting-my-data-onto-kibana/67256)
**Author:** [@manocs](https://discuss.elastic.co/u/manocs)\ **Replies:** 1\ **Last updated:** [November 27, 2016, 5:52am UTC](https://discuss.elastic.co/t/i-want-to-pay-for-getting-my-data-onto-kibana/67256 "2016-11-27T05:52:07Z")
I am a novice user, and I have spent hours trying to do basic things like download logstash and get it to run, which I haven't been able to do, so I thought there must be some sort of service provider that can upload my … --- ## [Logstash date difference in minutes](https://discuss.elastic.co/t/logstash-date-difference-in-minutes/67243)
**Author:** [@Dallas\_Toth](https://discuss.elastic.co/u/Dallas_Toth)\ **Replies:** 1\ **Last updated:** [November 26, 2016, 5:11pm UTC](https://discuss.elastic.co/t/logstash-date-difference-in-minutes/67243 "2016-11-26T17:11:19Z")
I am looking for some direction on filters. I have a event with a startdatetime and enddatetime I need to add a field to the same event which is the difference in minutes. Elapse won't work since the startdatetime and en… --- ## [Issue with conditionals in logstash with fields from Kafka ----\> FileBeat prospectors](https://discuss.elastic.co/t/issue-with-conditionals-in-logstash-with-fields-from-kafka-filebeat-prospectors/67217)
**Author:** [@rodher](https://discuss.elastic.co/u/rodher)\ **Replies:** 0\ **Last updated:** [November 25, 2016, 8:15pm UTC](https://discuss.elastic.co/t/issue-with-conditionals-in-logstash-with-fields-from-kafka-filebeat-prospectors/67217 "2016-11-25T20:15:17Z")
Hello experts I have the following scenario: FileBeat ----\> Kafka -----\> Logstash -----\> Elastic ----\> Kibana In Filebeat I have 2 prospectors the in YML file,,. and I've added some fields to identify in logstash t… --- ## [Use logstash to combine fields of nested object into one array of values](https://discuss.elastic.co/t/use-logstash-to-combine-fields-of-nested-object-into-one-array-of-values/67192)
**Author:** [@jettro](https://discuss.elastic.co/u/jettro)\ **Replies:** 1\ **Last updated:** [November 25, 2016, 3:37pm UTC](https://discuss.elastic.co/t/use-logstash-to-combine-fields-of-nested-object-into-one-array-of-values/67192 "2016-11-25T15:37:41Z")
Hi I am having a challenge which I did not find a solution for yet. I am using logstash to parse some input documents and I would like to do the following: { "groups": \[ { "id": 1, "name": "one" }… --- ## [Logstash output exec - stdout redirect impossible?](https://discuss.elastic.co/t/logstash-output-exec-stdout-redirect-impossible/64709)
**Author:** [@paulkeogh](https://discuss.elastic.co/u/paulkeogh)\ **Replies:** 2\ **Last updated:** [November 25, 2016, 3:36pm UTC](https://discuss.elastic.co/t/logstash-output-exec-stdout-redirect-impossible/64709 "2016-11-25T15:36:22Z")
Hi, Based on my understanding of the code, it looks impossible to execute a command that redirects stdout to a file for example - so; command \> /tmp/myoutput won't work. This is because of the line; Open3.popen3(… --- ## [How to handle JSON arrays so that they're searchable in Kibana?](https://discuss.elastic.co/t/how-to-handle-json-arrays-so-that-theyre-searchable-in-kibana/67003)
**Author:** [@shadyabhi](https://discuss.elastic.co/u/shadyabhi)\ **Replies:** 1\ **Last updated:** [November 25, 2016, 3:33pm UTC](https://discuss.elastic.co/t/how-to-handle-json-arrays-so-that-theyre-searchable-in-kibana/67003 "2016-11-25T15:33:02Z")
We’ve JSON documents that contain arrays. # Doc 1 { "test": { "steps": \[{ "response\_time": "100" }, { "response\_time": "101" }\] } } # Doc 2 { "test": { … --- ## [Logstash replace: contatenate string with other field values](https://discuss.elastic.co/t/logstash-replace-contatenate-string-with-other-field-values/66731)
**Author:** [@slinky](https://discuss.elastic.co/u/slinky)\ **Replies:** 5\ **Last updated:** [November 25, 2016, 3:02pm UTC](https://discuss.elastic.co/t/logstash-replace-contatenate-string-with-other-field-values/66731 "2016-11-25T15:02:21Z")
In the docs there is this replace example: filter { mutate { replace =\> { "message" =\> "%{source\_host}: My new message" } } } Now I wanna do something like this: filter { mutate { replace =\> { "autono… --- ## [Mutate filter plugin: convert to keyword and text?](https://discuss.elastic.co/t/mutate-filter-plugin-convert-to-keyword-and-text/66936)
**Author:** [@GrahamHannington](https://discuss.elastic.co/u/GrahamHannington)\ **Replies:** 5\ **Last updated:** [November 25, 2016, 8:21am UTC](https://discuss.elastic.co/t/mutate-filter-plugin-convert-to-keyword-and-text/66936 "2016-11-25T08:21:36Z")
From the current mutate convert documentation: Valid conversion targets are: integer, float, string, and boolean. I expected to see the new-for-5.0 keyword and text types in that list. And perhaps a note about the st… --- ## [JDBC connection remain open after data insertion in Elastic search](https://discuss.elastic.co/t/jdbc-connection-remain-open-after-data-insertion-in-elastic-search/67092)
**Author:** [@toxicashu](https://discuss.elastic.co/u/toxicashu)\ **Replies:** 0\ **Last updated:** [November 24, 2016, 10:44am UTC](https://discuss.elastic.co/t/jdbc-connection-remain-open-after-data-insertion-in-elastic-search/67092 "2016-11-24T10:44:41Z")
Hello, We are inserting data in Elastic search from multiple postgresql tables using Logstash. Here is a demo view of one of the config file for doing it: input { jdbc { jdbc\_connection\_string =\> "Connectio… --- ## [Grok terminating Logstash 5.0](https://discuss.elastic.co/t/grok-terminating-logstash-5-0/64307)
**Author:** [@sjivan](https://discuss.elastic.co/u/sjivan)\ **Replies:** 4\ **Last updated:** [November 24, 2016, 10:02pm UTC](https://discuss.elastic.co/t/grok-terminating-logstash-5-0/64307 "2016-11-24T22:02:45Z")
Hi, I'm upgrading from LS 2.2.2 to LS 5.0 and as part of the upgrade moved the multiline filter to the file input codec but the grok filter remained unchanged. The grok pattern worked fine with LS 2.2.2 however when run… --- ## [Can geoip source be a sub-field?](https://discuss.elastic.co/t/can-geoip-source-be-a-sub-field/66951)
**Author:** [@Balsam](https://discuss.elastic.co/u/Balsam)\ **Replies:** 2\ **Last updated:** [November 24, 2016, 5:40pm UTC](https://discuss.elastic.co/t/can-geoip-source-be-a-sub-field/66951 "2016-11-24T17:40:58Z")
Hello everyone, I'm new to ELK. I'm learning to use it to store and analyze Suricata events. Everything is working except I can't get geoip working. The problem is the geoip source is from a sub-field and when I put th… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1071) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1073)