# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=851 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 852 --- ## [Date mapping issue](https://discuss.elastic.co/t/date-mapping-issue/112899)
**Author:** [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Replies:** 16\ **Last updated:** [December 22, 2017, 1:20pm UTC](https://discuss.elastic.co/t/date-mapping-issue/112899 "2017-12-22T13:20:15Z")
I've been getting this warning when running my Logstash pipeline. I'm using the date filter on this field and math this pattern: yyyy-MM-dd HH:mm:ss so I'm not sure why I'm seeing this warning. \[2017-12-21T21:27:55,443\]… --- ## [Kibanaclient - Error when executing Kibana client request](https://discuss.elastic.co/t/kibanaclient-error-when-executing-kibana-client-request/112990)
**Author:** [@antony.y](https://discuss.elastic.co/u/antony.y)\ **Replies:** 3\ **Last updated:** [December 22, 2017, 12:09pm UTC](https://discuss.elastic.co/t/kibanaclient-error-when-executing-kibana-client-request/112990 "2017-12-22T12:09:02Z")
I am now using netflow module in my logstash . And i using " bin/logstash --modules netflow --setup -M netflow.var.input.udp.port=2055 -M var.elasticsearch.hosts="192.168.88.120:9200" -M var.kibana.host="192.168.88.120:… --- ## [Debug Logstash hot\_threads](https://discuss.elastic.co/t/debug-logstash-hot-threads/112131)
**Author:** [@bmh.01](https://discuss.elastic.co/u/bmh.01)\ **Replies:** 3\ **Last updated:** [December 22, 2017, 11:08am UTC](https://discuss.elastic.co/t/debug-logstash-hot-threads/112131 "2017-12-22T11:08:34Z")
I've got an issue since upgrading to v6 where the logstash process will use 100% of a CPU core even though it is idle with no events being processed. I've checked through the JVM heap and whilst I'd like it to have more … --- ## [Issue with Elasticsearch mapping](https://discuss.elastic.co/t/issue-with-elasticsearch-mapping/112903)
**Author:** [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Replies:** 1\ **Last updated:** [December 22, 2017, 10:30am UTC](https://discuss.elastic.co/t/issue-with-elasticsearch-mapping/112903 "2017-12-22T10:30:37Z")
I've been getting these errors nonstop all day and haven't been able to figure out a solution. From investigating the issue, it looks like our data input is giving us a JSON object that I want to parse with the JSON filt… --- ## [New logstash installation](https://discuss.elastic.co/t/new-logstash-installation/112983)
**Author:** [@Kristoffer\_Fagerlund](https://discuss.elastic.co/u/Kristoffer_Fagerlund)\ **Replies:** 1\ **Last updated:** [December 22, 2017, 10:23am UTC](https://discuss.elastic.co/t/new-logstash-installation/112983 "2017-12-22T10:23:46Z")
Issue solved! --- ## [Auth problem after installing X-pack on logstash](https://discuss.elastic.co/t/auth-problem-after-installing-x-pack-on-logstash/111936)
**Author:** [@Nima\_van\_Persien](https://discuss.elastic.co/u/Nima_van_Persien)\ **Replies:** 1\ **Last updated:** [December 22, 2017, 9:57am UTC](https://discuss.elastic.co/t/auth-problem-after-installing-x-pack-on-logstash/111936 "2017-12-22T09:57:14Z")
Hi community, I have recently installed xpack on Elasticsearch,Kibana and logstash as well. but I don't receive any data more! this is the log which I receive, tail -f /var/log/logstash/logstash-plain.log \[2017-12-1… --- ## [Cluster configuarton for ELK stack with Kafka](https://discuss.elastic.co/t/cluster-configuarton-for-elk-stack-with-kafka/112823)
**Author:** [@SJN8](https://discuss.elastic.co/u/SJN8)\ **Replies:** 4\ **Last updated:** [December 22, 2017, 9:29am UTC](https://discuss.elastic.co/t/cluster-configuarton-for-elk-stack-with-kafka/112823 "2017-12-22T09:29:54Z")
Hi All, I am done with standalone testing of ELK with Kafka now i want to implement Full Clustered ELK + Kafka . Please share docs and imp points for the same . --- ## [Multiline plugin not merging lines and throwing \_grokparsefailure](https://discuss.elastic.co/t/multiline-plugin-not-merging-lines-and-throwing-grokparsefailure/112799)
**Author:** [@vinit](https://discuss.elastic.co/u/vinit)\ **Replies:** 4\ **Last updated:** [December 22, 2017, 7:57am UTC](https://discuss.elastic.co/t/multiline-plugin-not-merging-lines-and-throwing-grokparsefailure/112799 "2017-12-22T07:57:24Z")
Hi, In logstash 1.x, used multiline plugin to merge all lines those are not starting with Timestamp and found issue that in some cases it didn't worked. Filter: multiline { patterns\_dir =\> "/u01/data/conf/patterns" … --- ## [Logstash configuration problem](https://discuss.elastic.co/t/logstash-configuration-problem/112858)
**Author:** [@thierry-decker](https://discuss.elastic.co/u/thierry-decker)\ **Replies:** 2\ **Last updated:** [December 22, 2017, 7:57am UTC](https://discuss.elastic.co/t/logstash-configuration-problem/112858 "2017-12-22T07:57:14Z")
I'm currently working on a Logstash configuration. Working on version 6.1. My problem is that I want my logstash instance to receive events from both NetFlow module and rsyslog senders. Configuring NetFlow module in l… --- ## [Is it possiable to add condition statements to logstash](https://discuss.elastic.co/t/is-it-possiable-to-add-condition-statements-to-logstash/112940)
**Author:** [@ent\_1](https://discuss.elastic.co/u/ent_1)\ **Replies:** 1\ **Last updated:** [December 22, 2017, 6:32am UTC](https://discuss.elastic.co/t/is-it-possiable-to-add-condition-statements-to-logstash/112940 "2017-12-22T06:32:34Z")
Hello, i'm trying to load the data from existing index to new index using index. i would like to add some Conditions.. whats is the solution for this .. Regards Dileep. --- ## [CSV field from string to Date](https://discuss.elastic.co/t/csv-field-from-string-to-date/112069)
**Author:** [@Sridhar](https://discuss.elastic.co/u/Sridhar)\ **Replies:** 12\ **Last updated:** [December 22, 2017, 6:20am UTC](https://discuss.elastic.co/t/csv-field-from-string-to-date/112069 "2017-12-22T06:20:53Z")
Hi, I have to convert the Run\_date of the CSV file as date but its populating as string. Please help me..!! Sample CSV file; 30-NOV-17,GH,381,CUSTOMERS 30-NOV-17,GH,50047,TRANSACTIONS 30-NOV-17,IQ,0,TRANSACTIONS 30… --- ## [Grok working, but how do I put it in a pattern?](https://discuss.elastic.co/t/grok-working-but-how-do-i-put-it-in-a-pattern/112934)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 2\ **Last updated:** [December 22, 2017, 6:20am UTC](https://discuss.elastic.co/t/grok-working-but-how-do-i-put-it-in-a-pattern/112934 "2017-12-22T06:20:45Z")
Hi, Stupid question but how can I create a custom pattern and call on it? I've used the grok debugger to make sure everything gets parsed correctly but I'm not sure how to go from there. Log data (events come in per li… --- ## [Logstash-Filter-Elasticsearch Slow](https://discuss.elastic.co/t/logstash-filter-elasticsearch-slow/110583)
**Author:** [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\ **Replies:** 5\ **Last updated:** [December 22, 2017, 1:10am UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-slow/110583 "2017-12-22T01:10:31Z")
Hi, I am getting slow performance when using logstash-filter-elasticsearch. Even with small volumes of data. I've tried this in multiple versions from 5.3 up to 6.0 of both ES & LS. By way of example here is a config f… --- ## [Logstash configuration file](https://discuss.elastic.co/t/logstash-configuration-file/112432)
**Author:** [@Amir\_P](https://discuss.elastic.co/u/Amir_P)\ **Replies:** 2\ **Last updated:** [December 21, 2017, 9:55pm UTC](https://discuss.elastic.co/t/logstash-configuration-file/112432 "2017-12-21T21:55:20Z")
Hello, I wrote a logstash configuration file to sql server (a simple one, taking the table as is and index it), but i want to store each row (documents) from the table as an array in elastic. Do I need to write it at th… --- ## [Running 2 instances of Logstash on Windows machine](https://discuss.elastic.co/t/running-2-instances-of-logstash-on-windows-machine/112884)
**Author:** [@Jay1](https://discuss.elastic.co/u/Jay1)\ **Replies:** 3\ **Last updated:** [December 21, 2017, 9:40pm UTC](https://discuss.elastic.co/t/running-2-instances-of-logstash-on-windows-machine/112884 "2017-12-21T21:40:06Z")
I am trying to create a local setup with 2 instances of logstash running on windows 7 to do some validations ( this is a part of a bigger work of creating a clustered prod environment) . I tried creating a copy of logst… --- ## [Logstash is failing with response code 403 trying to publish events to closed filebeat indexes](https://discuss.elastic.co/t/logstash-is-failing-with-response-code-403-trying-to-publish-events-to-closed-filebeat-indexes/112839)
**Author:** [@Yuri\_Sibirski](https://discuss.elastic.co/u/Yuri_Sibirski)\ **Replies:** 1\ **Last updated:** [December 21, 2017, 7:33pm UTC](https://discuss.elastic.co/t/logstash-is-failing-with-response-code-403-trying-to-publish-events-to-closed-filebeat-indexes/112839 "2017-12-21T19:33:19Z")
Dear ELK specialists, for some reason i am seeing my logstash is failing with response code 403 by trying to publish events into closed indexes. Here are some entries from logstash-plain.log. Do you have any idea why it … --- ## [Logstash doesn't run as a service only manually](https://discuss.elastic.co/t/logstash-doesnt-run-as-a-service-only-manually/112652)
**Author:** [@Kristoffer\_Fagerlund](https://discuss.elastic.co/u/Kristoffer_Fagerlund)\ **Replies:** 2\ **Last updated:** [December 21, 2017, 6:41pm UTC](https://discuss.elastic.co/t/logstash-doesnt-run-as-a-service-only-manually/112652 "2017-12-21T18:41:05Z")
logstash is only listening to the confed port and receiving messages if Istart it manually with : sudo bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/logstash.conf Starting it as a service and it do… --- ## [File-input plugin unable to detect changes in nginx access log](https://discuss.elastic.co/t/file-input-plugin-unable-to-detect-changes-in-nginx-access-log/112865)
**Author:** [@mattjourard](https://discuss.elastic.co/u/mattjourard)\ **Replies:** 0\ **Last updated:** [December 21, 2017, 5:39pm UTC](https://discuss.elastic.co/t/file-input-plugin-unable-to-detect-changes-in-nginx-access-log/112865 "2017-12-21T17:39:37Z")
Hi, I've been trying to get the file input plugin to work with the nginx access log to no avail. I've confirmed that the logstash user has read and execute access to /var/log/nginx/access.log, and I'm able to use sudo … --- ## [Why changing logstash filter config caused i/o timeout event](https://discuss.elastic.co/t/why-changing-logstash-filter-config-caused-i-o-timeout-event/109286)
**Author:** [@Yuri\_Sibirski](https://discuss.elastic.co/u/Yuri_Sibirski)\ **Replies:** 6\ **Last updated:** [December 21, 2017, 5:36pm UTC](https://discuss.elastic.co/t/why-changing-logstash-filter-config-caused-i-o-timeout-event/109286 "2017-12-21T17:36:44Z")
Hi there, i am really confused why changing my 10-syslog-filter.conf file caused i/o timeout event on all the clients and no data was saved. Initial config is: filter { if \[type\] == "syslog" { grok { match =\>… --- ## [JDBC\_Streaming informations](https://discuss.elastic.co/t/jdbc-streaming-informations/112489)
**Author:** [@AH\_M7](https://discuss.elastic.co/u/AH_M7)\ **Replies:** 6\ **Last updated:** [December 21, 2017, 5:33pm UTC](https://discuss.elastic.co/t/jdbc-streaming-informations/112489 "2017-12-21T17:33:56Z")
Hi, I'm actually working on a Logstash demo. The idea is simple : I use the Kafka Input Method to read my String (message) AND i'm trying to do a SQL query with the JDBC\_Streaming filter (if possible not with ruby\_co… --- ## [Grok performance](https://discuss.elastic.co/t/grok-performance/112577)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 4\ **Last updated:** [December 21, 2017, 4:04pm UTC](https://discuss.elastic.co/t/grok-performance/112577 "2017-12-21T16:04:55Z")
hi there! i use multiple grok match patterns in my logstash filter. eg: filter { grok { # grok1 break\_on\_match =\> true match =\> { "message" =\> \[ "^(?\\\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2},\\d{1,3}).?(… --- ## [\[SOLVED\] Chaining Pipelines Together](https://discuss.elastic.co/t/solved-chaining-pipelines-together/112708)
**Author:** [@bgeels](https://discuss.elastic.co/u/bgeels)\ **Replies:** 4\ **Last updated:** [December 21, 2017, 3:26pm UTC](https://discuss.elastic.co/t/solved-chaining-pipelines-together/112708 "2017-12-21T15:26:17Z")
Hey All, Has anyone tried to chain multiple pipelines together on a single host? I'm trying to send from http-\>A-\>B-\>C-\>logfile. Connecting the pipelines together via the tcp input/output plugins. I have debug level lo… --- ## [Sincedb write error despite proper permissions](https://discuss.elastic.co/t/sincedb-write-error-despite-proper-permissions/112259)
**Author:** [@makibroshett](https://discuss.elastic.co/u/makibroshett)\ **Replies:** 1\ **Last updated:** [December 21, 2017, 2:47pm UTC](https://discuss.elastic.co/t/sincedb-write-error-despite-proper-permissions/112259 "2017-12-21T14:47:28Z")
Hi, I installed LS 6.0 on Linux (2.6.18, centos). I have an error logged whenever LS attempts to write to a sincedb (not in the default folder), could you please help me finding out what is wrong ? All kind of help is … --- ## [Logstash debugger tool shows matched for filter but new fields don't appear on Kibana](https://discuss.elastic.co/t/logstash-debugger-tool-shows-matched-for-filter-but-new-fields-dont-appear-on-kibana/110426)
**Author:** [@Yuri\_Sibirski](https://discuss.elastic.co/u/Yuri_Sibirski)\ **Replies:** 2\ **Last updated:** [December 21, 2017, 2:33pm UTC](https://discuss.elastic.co/t/logstash-debugger-tool-shows-matched-for-filter-but-new-fields-dont-appear-on-kibana/110426 "2017-12-21T14:33:07Z")
I am having an issue with my logstash and grok filters. When i use logstash debugger tool and pasting log entry and filter to test the parser it checks out OK and gives me an expected output with extra fields that i am l… --- ## [Parsing a directory containing csv files as input (all of them new and ignore\_older=0)](https://discuss.elastic.co/t/parsing-a-directory-containing-csv-files-as-input-all-of-them-new-and-ignore-older-0/112819)
**Author:** [@Amouzigh\_Driss](https://discuss.elastic.co/u/Amouzigh_Driss)\ **Replies:** 1\ **Last updated:** [December 21, 2017, 12:47pm UTC](https://discuss.elastic.co/t/parsing-a-directory-containing-csv-files-as-input-all-of-them-new-and-ignore-older-0/112819 "2017-12-21T12:47:27Z")
Hello, I'm trying to parse csv files and send the content to ElasticSearch. With one file it's OK. But when telling logstash to parse the entier directory it's not OK. Here's the config for one file: input { file { … --- ## [Remove\_field in csv filter section](https://discuss.elastic.co/t/remove-field-in-csv-filter-section/112683)
**Author:** [@Amouzigh\_Driss](https://discuss.elastic.co/u/Amouzigh_Driss)\ **Replies:** 6\ **Last updated:** [December 21, 2017, 12:33pm UTC](https://discuss.elastic.co/t/remove-field-in-csv-filter-section/112683 "2017-12-21T12:33:14Z")
Hello, I'm trying to remove "extra" fileds while parsing csv files (separator is ; ). My filter section is that one: filter { csv { separator =\> ";" source =\> message columns =\> \["ID","Value","Timestamp","DateA… --- ## [Unable to connect derbydb through jdbc](https://discuss.elastic.co/t/unable-to-connect-derbydb-through-jdbc/112751)
**Author:** [@errupeshmca](https://discuss.elastic.co/u/errupeshmca)\ **Replies:** 1\ **Last updated:** [December 21, 2017, 12:15pm UTC](https://discuss.elastic.co/t/unable-to-connect-derbydb-through-jdbc/112751 "2017-12-21T12:15:47Z")
i need to extract data from tables which is DERBy db but i amd Facing issue with connecting derbydb through jdbc, Below is the configuration of mine. elasticsearch version : 5.6.5 logstash version: 5.6.5 jdbc { jdbc… --- ## [Convert date from string to datetime format](https://discuss.elastic.co/t/convert-date-from-string-to-datetime-format/112774)
**Author:** [@vuxuanlai](https://discuss.elastic.co/u/vuxuanlai)\ **Replies:** 3\ **Last updated:** [December 21, 2017, 9:48am UTC](https://discuss.elastic.co/t/convert-date-from-string-to-datetime-format/112774 "2017-12-21T09:48:00Z")
Hi there, I have logstash filter with date as below: date { match =\> \["date","MMM dd yyyy HH:mm:ss", "MMM d yyyy HH:mm:ss"\] target =\> "date" } but when i visualize data on kibana, i got "date" field which still g… --- ## [GeoIP is not discoverable by elasticsearch and kibana](https://discuss.elastic.co/t/geoip-is-not-discoverable-by-elasticsearch-and-kibana/112781)
**Author:** [@yash\_mangla](https://discuss.elastic.co/u/yash_mangla)\ **Replies:** 0\ **Last updated:** [December 21, 2017, 9:30am UTC](https://discuss.elastic.co/t/geoip-is-not-discoverable-by-elasticsearch-and-kibana/112781 "2017-12-21T09:30:20Z")
Hello Guyz, I am currently working on ELK stack version 5.6.3. I have JSON logs which contains 2 level json. Which can be consider as nested JSON in technical terms. This log contain geoip coordinates. I have written… --- ## [Netflow v9 is not being processed](https://discuss.elastic.co/t/netflow-v9-is-not-being-processed/112776)
**Author:** [@geertn444](https://discuss.elastic.co/u/geertn444)\ **Replies:** 1\ **Last updated:** [December 21, 2017, 9:27am UTC](https://discuss.elastic.co/t/netflow-v9-is-not-being-processed/112776 "2017-12-21T09:27:07Z")
Hello, I am using 6.x logstash with default netflow plugin. I am trying to read netflow v9 records from Riverbed Steelhead. V5 works, but when i switch to v9...nothing is displayed (if i run interactively with -f). In… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=850) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=852)