# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=883 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 884 --- ## [Sql\_last\_value with tracking column](https://discuss.elastic.co/t/sql-last-value-with-tracking-column/105639)
**Author:** [@mkarmi](https://discuss.elastic.co/u/mkarmi)\ **Replies:** 1\ **Last updated:** [October 31, 2017, 6:16am UTC](https://discuss.elastic.co/t/sql-last-value-with-tracking-column/105639 "2017-10-31T06:16:12Z")
I'm little confused with the "clean\_run" option. from documentation it says "If clean\_run is set to true, this value will be ignored and sql\_last\_value will be set to Jan 1, 1970, or 0 if use\_column\_value is true, as if … --- ## [Csv filter convert to date doesn't work](https://discuss.elastic.co/t/csv-filter-convert-to-date-doesnt-work/105609)
**Author:** [@szgaljic](https://discuss.elastic.co/u/szgaljic)\ **Replies:** 4\ **Last updated:** [October 31, 2017, 6:04am UTC](https://discuss.elastic.co/t/csv-filter-convert-to-date-doesnt-work/105609 "2017-10-31T06:04:18Z")
I have a column called Birthday that is formatted as dd/MM/yyyy (i.e., 3/8/1983) and would like Logstash to treat this as a date when passing to Elasticsearch so the mapping is auto-generated as a date. This does not wo… --- ## [Logstash saves to two different indexes when adding netflow module](https://discuss.elastic.co/t/logstash-saves-to-two-different-indexes-when-adding-netflow-module/105683)
**Author:** [@zentavr](https://discuss.elastic.co/u/zentavr)\ **Replies:** 4\ **Last updated:** [October 31, 2017, 4:47am UTC](https://discuss.elastic.co/t/logstash-saves-to-two-different-indexes-when-adding-netflow-module/105683 "2017-10-31T04:47:23Z")
I enabled netflow module in logstash 5.6.3 several days ago and today started to stream the flows into logstash. Before that I had collected the logs via filebeat already. What I noticed is that filebeat's stuff appeare… --- ## [Logstash 5.6.3 not working with Java 9](https://discuss.elastic.co/t/logstash-5-6-3-not-working-with-java-9/105830)
**Author:** [@Seyf\_Souissi](https://discuss.elastic.co/u/Seyf_Souissi)\ **Replies:** 1\ **Last updated:** [October 31, 2017, 3:13am UTC](https://discuss.elastic.co/t/logstash-5-6-3-not-working-with-java-9/105830 "2017-10-31T03:13:01Z")
Hi, I'm new to elastic and was trying to get Logstash working on Ubuntu. The JDK installed successfully, elastic and KIbana are working correctly. java -version java version "9.0.1" Java(TM) SE Runtime Environment (bu… --- ## [Best Practices for supporting Test vs Production environments and different data sources](https://discuss.elastic.co/t/best-practices-for-supporting-test-vs-production-environments-and-different-data-sources/105512)
**Author:** [@jonsrose](https://discuss.elastic.co/u/jonsrose)\ **Replies:** 2\ **Last updated:** [October 30, 2017, 10:50pm UTC](https://discuss.elastic.co/t/best-practices-for-supporting-test-vs-production-environments-and-different-data-sources/105512 "2017-10-30T22:50:53Z")
I have couple of related questions Test vs Prod environments I need to manage 2 shared environments for Elastic Search, one for testing and one for production. I am interested in what are the best practices for managi… --- ## [ELK as AppD replacement?](https://discuss.elastic.co/t/elk-as-appd-replacement/105812)
**Author:** [@Jack\_Judge](https://discuss.elastic.co/u/Jack_Judge)\ **Replies:** 1\ **Last updated:** [October 30, 2017, 10:46pm UTC](https://discuss.elastic.co/t/elk-as-appd-replacement/105812 "2017-10-30T22:46:09Z")
I know I'm comparing apples to oranges. We're an AppD / Splunk house and I'm looking into the feasibility of replacing them with ELK. We can reproduce almost all the functionality we need from the two, except for AppD's… --- ## [Logstash JDBC Input Unable to Connect to Multiple DB](https://discuss.elastic.co/t/logstash-jdbc-input-unable-to-connect-to-multiple-db/104461)
**Author:** [@Kareny](https://discuss.elastic.co/u/Kareny)\ **Replies:** 2\ **Last updated:** [October 30, 2017, 10:05pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-unable-to-connect-to-multiple-db/104461 "2017-10-30T22:05:19Z")
Hi, I'm currently working on getting the data from multiple DBs using the JDBC input plugin. As the config shown below, I have separate jdbc configs for MySQL and MSSQL. " input { jdbc { jdbc\_driver\_library =\>"/opt/j… --- ## [Logstash saving to two different indexes at the same time](https://discuss.elastic.co/t/logstash-saving-to-two-different-indexes-at-the-same-time/105666)
**Author:** [@Rafael\_Carvallo](https://discuss.elastic.co/u/Rafael_Carvallo)\ **Replies:** 2\ **Last updated:** [October 30, 2017, 9:56pm UTC](https://discuss.elastic.co/t/logstash-saving-to-two-different-indexes-at-the-same-time/105666 "2017-10-30T21:56:13Z")
Hi, I'd like to request some help with logstash, currently I'm trying to use it as a rsyslog gatherer, basically I've got this setup: Devices Reporting to RSYSLOG RSYSLOG sending the logs to Logstash as JSON Logstash… --- ## [Logstash Output plugin with HTTP-API Key](https://discuss.elastic.co/t/logstash-output-plugin-with-http-api-key/105808)
**Author:** [@MuthuV](https://discuss.elastic.co/u/MuthuV)\ **Replies:** 0\ **Last updated:** [October 30, 2017, 8:47pm UTC](https://discuss.elastic.co/t/logstash-output-plugin-with-http-api-key/105808 "2017-10-30T20:47:20Z")
Hi Team, Here is my scenario, I am currently doing POC to read error records from oracle Database tables and pumping to ElasticSearch - DataLens. I am using Logstash-5.6.3 currently, Added the JDBC Input plugins, Able… --- ## [Not possible to apply grok filter to kafka.key](https://discuss.elastic.co/t/not-possible-to-apply-grok-filter-to-kafka-key/105620)
**Author:** [@ashvita](https://discuss.elastic.co/u/ashvita)\ **Replies:** 1\ **Last updated:** [October 30, 2017, 8:25pm UTC](https://discuss.elastic.co/t/not-possible-to-apply-grok-filter-to-kafka-key/105620 "2017-10-30T20:25:57Z")
Hi, I am trying to apply grok filter to kafka.key but it isnt getting applied. filter{ grok{ match =\> \['kafka.key' , '%{GREEDYDATA: IP Address}'\] break\_on\_match =\> false } } I am using this . There is no failure … --- ## [Logstash Input Filter By Date](https://discuss.elastic.co/t/logstash-input-filter-by-date/105779)
**Author:** [@maddie0726](https://discuss.elastic.co/u/maddie0726)\ **Replies:** 2\ **Last updated:** [October 30, 2017, 8:22pm UTC](https://discuss.elastic.co/t/logstash-input-filter-by-date/105779 "2017-10-30T20:22:44Z")
I have Elasticsearch Kibana and Logstash version 5.6.2 all running on WIndows 2012 R2. I am reading input from a .txt file. I have one .txt file for the current day that is always updating with new, live events and an ar… --- ## [Grok parse failed, works good with grok debugger](https://discuss.elastic.co/t/grok-parse-failed-works-good-with-grok-debugger/105473)
**Author:** [@sarikonl](https://discuss.elastic.co/u/sarikonl)\ **Replies:** 5\ **Last updated:** [October 30, 2017, 8:01pm UTC](https://discuss.elastic.co/t/grok-parse-failed-works-good-with-grok-debugger/105473 "2017-10-30T20:01:54Z")
Can someone please help me with this. I tried almost all possible ways but no luck. My grok debugger works perfect but get "beats\_input\_codec\_plain\_applied, failedparse" error in Kibana from logstash. This is the grok I… --- ## [Ruby filter: Ruby exception occurred: undefined method \`\*' for nil:NilClass](https://discuss.elastic.co/t/ruby-filter-ruby-exception-occurred-undefined-method-for-nil-nilclass/105699)
**Author:** [@nicklaw](https://discuss.elastic.co/u/nicklaw)\ **Replies:** 2\ **Last updated:** [October 30, 2017, 6:53pm UTC](https://discuss.elastic.co/t/ruby-filter-ruby-exception-occurred-undefined-method-for-nil-nilclass/105699 "2017-10-30T18:53:46Z")
My circumstance is a little different to this post (I think). I have the below filter that uses ruby to convert Nginx access request\_time's and upsteam\_response\_time's to milliseconds: filter { # NGINX ACCESS GROK … --- ## [Database sync using document\_id](https://discuss.elastic.co/t/database-sync-using-document-id/105650)
**Author:** [@karnamonkster](https://discuss.elastic.co/u/karnamonkster)\ **Replies:** 2\ **Last updated:** [October 30, 2017, 4:29pm UTC](https://discuss.elastic.co/t/database-sync-using-document-id/105650 "2017-10-30T16:29:29Z")
Hello, I am trying to sync a database with around 100,000 records. Now these records are flags respective to number of devices which keep on updating. For me the use case is to see the latest status of that device and… --- ## [Truncation of field values in grok filter](https://discuss.elastic.co/t/truncation-of-field-values-in-grok-filter/105531)
**Author:** [@Rupesh\_Ranjan](https://discuss.elastic.co/u/Rupesh_Ranjan)\ **Replies:** 1\ **Last updated:** [October 30, 2017, 3:12pm UTC](https://discuss.elastic.co/t/truncation-of-field-values-in-grok-filter/105531 "2017-10-30T15:12:54Z")
How can i truncate a field value in grok? e.g. I have a field "value: abcdefgh1234567890" in logstash(obtained from message after grok filter) match =\> { "message" =\> "%{GREEDYDATA}value=%{WORD:value}"} and i want t… --- ## [Create custom index using custom mapping field via logstash](https://discuss.elastic.co/t/create-custom-index-using-custom-mapping-field-via-logstash/105757)
**Author:** [@Aner\_Izraeli](https://discuss.elastic.co/u/Aner_Izraeli)\ **Replies:** 7\ **Last updated:** [October 30, 2017, 2:56pm UTC](https://discuss.elastic.co/t/create-custom-index-using-custom-mapping-field-via-logstash/105757 "2017-10-30T14:56:58Z")
Hello there, im using logstash with matching my custom fields via multigrok.conf. example: match =\> { "message" =\> "(?\AUTORUN) (?.\*?)|%{IPORHOST:machineName}|%{DATA:userName}|%{DATA:domainname}|%{DATA:os}|%{… --- ## [Logstash - Unsupported platform: x86\_64-windows](https://discuss.elastic.co/t/logstash-unsupported-platform-x86-64-windows/105751)
**Author:** [@Sanat\_Sathyan](https://discuss.elastic.co/u/Sanat_Sathyan)\ **Replies:** 2\ **Last updated:** [October 30, 2017, 1:33pm UTC](https://discuss.elastic.co/t/logstash-unsupported-platform-x86-64-windows/105751 "2017-10-30T13:33:00Z")
Whenever i try to install a plugin through "logstash-plugin", I get this error.. I use Windows 10 - 64 bit, and i have Java 9.0.1 installed. \[image\] --- ## [Exclude if not in array](https://discuss.elastic.co/t/exclude-if-not-in-array/105552)
**Author:** [@henrilabarre](https://discuss.elastic.co/u/henrilabarre)\ **Replies:** 2\ **Last updated:** [October 30, 2017, 1:26pm UTC](https://discuss.elastic.co/t/exclude-if-not-in-array/105552 "2017-10-30T13:26:48Z")
Hi here is my source: { "telephone" =\> { "fine" =\> { "date" =\> "2017-10-26T16:54:28.477Z", "value" =\> "0387931080" } }, "TEL" =\> "0387931080", } I use this conf file to add … --- ## [Not Getting Syslog messages After Reload of logstash pipeline](https://discuss.elastic.co/t/not-getting-syslog-messages-after-reload-of-logstash-pipeline/105762)
**Author:** [@amitdighe235](https://discuss.elastic.co/u/amitdighe235)\ **Replies:** 0\ **Last updated:** [October 30, 2017, 1:06pm UTC](https://discuss.elastic.co/t/not-getting-syslog-messages-after-reload-of-logstash-pipeline/105762 "2017-10-30T13:06:35Z")
My logstash pipeline accepts Syslog messages at defined port and there are two log syslog sources for sending messages to this pipeline . So when I change the configuration pipeline , the new Changed configuration is loa… --- ## [How to extract partial match using grok filtering?](https://discuss.elastic.co/t/how-to-extract-partial-match-using-grok-filtering/105649)
**Author:** [@AI\_Rumman](https://discuss.elastic.co/u/AI_Rumman)\ **Replies:** 1\ **Last updated:** [October 30, 2017, 12:45pm UTC](https://discuss.elastic.co/t/how-to-extract-partial-match-using-grok-filtering/105649 "2017-10-30T12:45:11Z")
Hi, I am trying to write a logstash conf file with grok filter to find out my cassandra clusters name and datacenter name. For example: Initial value: "metric\_path": "zabbix\_test-us-east-2-node0.KeyCache.HitRate" Her… --- ## [Is there a way to know how to calculate RAM for Logstash](https://discuss.elastic.co/t/is-there-a-way-to-know-how-to-calculate-ram-for-logstash/105516)
**Author:** [@Neal\_Shan](https://discuss.elastic.co/u/Neal_Shan)\ **Replies:** 4\ **Last updated:** [October 27, 2017, 8:48am UTC](https://discuss.elastic.co/t/is-there-a-way-to-know-how-to-calculate-ram-for-logstash/105516 "2017-10-27T08:48:19Z")
I was running one Logstash JDBC instance to sync 3 tables to ES every 5 minuts, but now we need to sync 8 tables. Sync 8 takes too long. So I tried to run multi instances of Logstash on single server. It doesn't help. T… --- ## [Starting listener on port commented out in config ...?](https://discuss.elastic.co/t/starting-listener-on-port-commented-out-in-config/105641)
**Author:** [@irom77](https://discuss.elastic.co/u/irom77)\ **Replies:** 1\ **Last updated:** [October 30, 2017, 12:43pm UTC](https://discuss.elastic.co/t/starting-listener-on-port-commented-out-in-config/105641 "2017-10-30T12:43:10Z")
I commented one of two my inputs in config file, but I can see in logs that it is still starting... 13:59:39.989 \[\[main\]\"0.0.0.0:11514"} 13:59:39.995 \[\[… --- ## [Help With Custom Grok Parsing](https://discuss.elastic.co/t/help-with-custom-grok-parsing/105705)
**Author:** [@uppaljs](https://discuss.elastic.co/u/uppaljs)\ **Replies:** 1\ **Last updated:** [October 30, 2017, 12:41pm UTC](https://discuss.elastic.co/t/help-with-custom-grok-parsing/105705 "2017-10-30T12:41:22Z")
Hi, My log is coming in the format ( its either OK for successful with a set of value or ERR for failure with a different set of values ) ok{fus=192.168.20.1, ful=44001, fui=672a000249d481f10000000c59f5efe3, fuc=0, … --- ## [Varnish client and backend in the same log file](https://discuss.elastic.co/t/varnish-client-and-backend-in-the-same-log-file/105759)
**Author:** [@covidium](https://discuss.elastic.co/u/covidium)\ **Replies:** 0\ **Last updated:** [October 30, 2017, 12:26pm UTC](https://discuss.elastic.co/t/varnish-client-and-backend-in-the-same-log-file/105759 "2017-10-30T12:26:39Z")
Hello, For some time now I'm trying to create a logstash config that will place information from 2 lines in the same document. As you may probably know, Varnish can be configured to log into varnishncsa for both client… --- ## [Auto Update logs without running logstash](https://discuss.elastic.co/t/auto-update-logs-without-running-logstash/104855)
**Author:** [@Samreen\_Zeeshan](https://discuss.elastic.co/u/Samreen_Zeeshan)\ **Replies:** 9\ **Last updated:** [October 30, 2017, 11:31am UTC](https://discuss.elastic.co/t/auto-update-logs-without-running-logstash/104855 "2017-10-30T11:31:09Z")
I have logs stored in a MS SQL table and i have migrated all logs by running logstash once. Now what i want is if i update a particular record in SQL table that should be reflected in Kibana visualizations without resta… --- ## [Elasticsearch output with a time period](https://discuss.elastic.co/t/elasticsearch-output-with-a-time-period/105746)
**Author:** [@omer](https://discuss.elastic.co/u/omer)\ **Replies:** 0\ **Last updated:** [October 30, 2017, 10:57am UTC](https://discuss.elastic.co/t/elasticsearch-output-with-a-time-period/105746 "2017-10-30T10:57:27Z")
Hi all, I have log files with long time period such as 6 months or more. However, i need, for example, one day of it. Loading all data to Kibana and then visualize for that day is expensive in terms of memory and disc s… --- ## [Unable to Send static log files to elasticsearch](https://discuss.elastic.co/t/unable-to-send-static-log-files-to-elasticsearch/105643)
**Author:** [@kpavankulakarni](https://discuss.elastic.co/u/kpavankulakarni)\ **Replies:** 3\ **Last updated:** [October 30, 2017, 10:07am UTC](https://discuss.elastic.co/t/unable-to-send-static-log-files-to-elasticsearch/105643 "2017-10-30T10:07:44Z")
Hi Everyone, Am starting ELK project for first time. I have installed logstash, elasticsearch, kibana. All in one single lab server. I have couple of usecases to work on, So i got couple of files from app team. Basica… --- ## [How to know the log comme from which file](https://discuss.elastic.co/t/how-to-know-the-log-comme-from-which-file/105211)
**Author:** [@seeva](https://discuss.elastic.co/u/seeva)\ **Replies:** 8\ **Last updated:** [October 30, 2017, 9:47am UTC](https://discuss.elastic.co/t/how-to-know-the-log-comme-from-which-file/105211 "2017-10-30T09:47:28Z")
Hi, I am using filebeat,logstash,ES and kibana 5.6.3. I want to be able to see the log source file name in kibana, but i am not being able to to. i have all information but do not have the file source name , the log are… --- ## [Can't parse httpd combined logs when ident is set](https://discuss.elastic.co/t/cant-parse-httpd-combined-logs-when-ident-is-set/105732)
**Author:** [@sdumetz](https://discuss.elastic.co/u/sdumetz)\ **Replies:** 0\ **Last updated:** [October 30, 2017, 9:30am UTC](https://discuss.elastic.co/t/cant-parse-httpd-combined-logs-when-ident-is-set/105732 "2017-10-30T09:30:36Z")
I have those logs (OVH shared hosting), that set the ident field of log lines to: example.com:443. It's only on a fraction of the logs, most of them set to example.com or www.example.com as they should (I think?). A ful… --- ## [Update multiple documents using script is not working](https://discuss.elastic.co/t/update-multiple-documents-using-script-is-not-working/105728)
**Author:** [@aliaqdas1](https://discuss.elastic.co/u/aliaqdas1)\ **Replies:** 0\ **Last updated:** [October 30, 2017, 8:09am UTC](https://discuss.elastic.co/t/update-multiple-documents-using-script-is-not-working/105728 "2017-10-30T08:09:16Z")
Hi All, I'm trying to update multiple documents using list of document ids by storing their id's in map/list, and then at specific event call "update" for each of that document id. I've tried following conf code for ou… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=882) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=884)