# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=924 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 925 --- ## [Trying to build logstash 5.5.1 docker image but fail due to missing libzmq5 package](https://discuss.elastic.co/t/trying-to-build-logstash-5-5-1-docker-image-but-fail-due-to-missing-libzmq5-package/97079)
**Author:** [@mathias](https://discuss.elastic.co/u/mathias)\ **Replies:** 1\ **Last updated:** [August 17, 2017, 12:47pm UTC](https://discuss.elastic.co/t/trying-to-build-logstash-5-5-1-docker-image-but-fail-due-to-missing-libzmq5-package/97079 "2017-08-17T12:47:15Z")
Hi, I try to upgrade logstash from 5.4.0 which use libzmq3 to logstash 5.5.1 which uses libzmg5. I fail to build the contianer due to missing package “E: Unable to locate package libzmq5”. docker build --label=cran-5g… --- ## [How to split DNS request and retrive the domain](https://discuss.elastic.co/t/how-to-split-dns-request-and-retrive-the-domain/97074)
**Author:** [@mikygee](https://discuss.elastic.co/u/mikygee)\ **Replies:** 1\ **Last updated:** [August 17, 2017, 11:46am UTC](https://discuss.elastic.co/t/how-to-split-dns-request-and-retrive-the-domain/97074 "2017-08-17T11:46:49Z")
Hello, I’ve got a configuration that seems to work fine filter { if \[program\] == "named" { grok { break\_on\_match =\> true patterns\_dir =\> "/etc/logstash/conf.d/patterns" match =\> \[ "message… --- ## [How to parse the JSON Column dynamically using logstash](https://discuss.elastic.co/t/how-to-parse-the-json-column-dynamically-using-logstash/94752)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 1\ **Last updated:** [August 17, 2017, 9:30am UTC](https://discuss.elastic.co/t/how-to-parse-the-json-column-dynamically-using-logstash/94752 "2017-08-17T09:30:10Z")
Hi All I have a RESTful API (given below), when i index it , i get only 1 column in Elasticsearch as "message" http://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b1b15e88fa797225412429c1c50c122a1 I… --- ## [Logstash-input-jdbc plugin will not install](https://discuss.elastic.co/t/logstash-input-jdbc-plugin-will-not-install/97369)
**Author:** [@bilpor](https://discuss.elastic.co/u/bilpor)\ **Replies:** 1\ **Last updated:** [August 17, 2017, 9:24am UTC](https://discuss.elastic.co/t/logstash-input-jdbc-plugin-will-not-install/97369 "2017-08-17T09:24:56Z")
0 down vote favorite I have elasticsearch 5.5.1 installed and running. I'm now trying to install logstash and required plugins to use against SQL Server. I'm trying to install the logstash-input-jdbc plugin but when I… --- ## [Elapsed time between consecutive logs](https://discuss.elastic.co/t/elapsed-time-between-consecutive-logs/97216)
**Author:** [@Robert\_Kovacs](https://discuss.elastic.co/u/Robert_Kovacs)\ **Replies:** 4\ **Last updated:** [August 17, 2017, 9:23am UTC](https://discuss.elastic.co/t/elapsed-time-between-consecutive-logs/97216 "2017-08-17T09:23:09Z")
I know a quite similar question was raised in Time between timestamp but a clear answer wasn’t given there. I would like to calculate the difference between every consecutive logs timestamp and insert the result as a ne… --- ## [Converting string(Time) field to date field...!](https://discuss.elastic.co/t/converting-string-time-field-to-date-field/97339)
**Author:** [@DharaniKumar](https://discuss.elastic.co/u/DharaniKumar)\ **Replies:** 4\ **Last updated:** [August 17, 2017, 9:22am UTC](https://discuss.elastic.co/t/converting-string-time-field-to-date-field/97339 "2017-08-17T09:22:58Z")
Hi I’ve a Time field in my log file which is a string type, i want that field to be date field. #my logstash filter config file kv{ value\_split =\> “=” } date { match =\> \[“Time”,“yyyy-MM-dd\_HH:mm:ss.SSS”\] target =\>… --- ## [Logstash json in text message](https://discuss.elastic.co/t/logstash-json-in-text-message/97285)
**Author:** [@annaK](https://discuss.elastic.co/u/annaK)\ **Replies:** 5\ **Last updated:** [August 17, 2017, 9:19am UTC](https://discuss.elastic.co/t/logstash-json-in-text-message/97285 "2017-08-17T09:19:16Z")
Hi all :slight\_smile: I have a log line that looks like this: time='2017-08-16 12:39:43.22253194 +0000 UTC' container\_name='/r-stack' source='stdout' data='{"name":"stack","hostname":"bc202dc10bc9","pid":28,"level":50,… --- ## [Logstash match IP range](https://discuss.elastic.co/t/logstash-match-ip-range/96086)
**Author:** [@ilya](https://discuss.elastic.co/u/ilya)\ **Replies:** 1\ **Last updated:** [August 7, 2017, 10:51am UTC](https://discuss.elastic.co/t/logstash-match-ip-range/96086 "2017-08-07T10:51:56Z")
Hello, Is possible to find is any IP address match IP range which is not full network range? Example: Is 192.168.1.5 in \[192.168.0.1-192.168.1.14\] --- ## [How to fetch first value in json file if field is coming up multiple times](https://discuss.elastic.co/t/how-to-fetch-first-value-in-json-file-if-field-is-coming-up-multiple-times/97357)
**Author:** [@deepanshu\_mittal](https://discuss.elastic.co/u/deepanshu_mittal)\ **Replies:** 1\ **Last updated:** [August 17, 2017, 8:40am UTC](https://discuss.elastic.co/t/how-to-fetch-first-value-in-json-file-if-field-is-coming-up-multiple-times/97357 "2017-08-17T08:40:01Z")
Hi, If my input json file is setting 1 field value twice in the file . Can we fetch particular value in that case? Eg : “message”: “{“modelVersion”:” 12.2.0.1.3\_20170331", “RackType”:“N/A”,“EngineeredSystems”:“Big Dat… --- ## [How to process multiple files in logstash](https://discuss.elastic.co/t/how-to-process-multiple-files-in-logstash/97148)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 4\ **Last updated:** [August 17, 2017, 7:41am UTC](https://discuss.elastic.co/t/how-to-process-multiple-files-in-logstash/97148 "2017-08-17T07:41:29Z")
I have some million json files that i separated in multiple sub folders. Basically these are Cloudtrail data for a year. My Logstash version is 5.5 and this is a 8 core , 32 GB system. The problem is that when i run my … --- ## [Exclude a sequence of logs](https://discuss.elastic.co/t/exclude-a-sequence-of-logs/97354)
**Author:** [@Hisushi](https://discuss.elastic.co/u/Hisushi)\ **Replies:** 0\ **Last updated:** [August 17, 2017, 7:27am UTC](https://discuss.elastic.co/t/exclude-a-sequence-of-logs/97354 "2017-08-17T07:27:23Z")
Hi everybody, I’d like to exclude a given sequence of logs in Logstash or pass them to a separate index as they are created when the logfile itself is generated by a predefined procedure. Is this even possible? My appro… --- ## [Logstash using grok filter to modify values](https://discuss.elastic.co/t/logstash-using-grok-filter-to-modify-values/97193)
**Author:** [@mulgar](https://discuss.elastic.co/u/mulgar)\ **Replies:** 2\ **Last updated:** [August 17, 2017, 7:24am UTC](https://discuss.elastic.co/t/logstash-using-grok-filter-to-modify-values/97193 "2017-08-17T07:24:29Z")
I haven’t used grok filter before and having a little trouble getting the results I need. Ultimately I’m pulling some data into ElasticSearch from a database query using Logstash and the JDBC input plugin. Mostly irrelev… --- ## [When using queue conf, Logstash hangs](https://discuss.elastic.co/t/when-using-queue-conf-logstash-hangs/96967)
**Author:** [@Frederico\_Ferreira](https://discuss.elastic.co/u/Frederico_Ferreira)\ **Replies:** 0\ **Last updated:** [August 14, 2017, 12:53pm UTC](https://discuss.elastic.co/t/when-using-queue-conf-logstash-hangs/96967 "2017-08-14T12:53:28Z")
Hello, i’m setting up logstash and want to use the queue configuration. I’m using the http input plugin and, everytime that i setup the queue (persisted) configuration, when i use curl to test it, it hangs, like, forev… --- ## [Can i convert latin-1 (ISO8859\_1) into UTF-8 available?](https://discuss.elastic.co/t/can-i-convert-latin-1-iso8859-1-into-utf-8-available/96898)
**Author:** [@Prapat\_Polchan](https://discuss.elastic.co/u/Prapat_Polchan)\ **Replies:** 10\ **Last updated:** [August 17, 2017, 7:14am UTC](https://discuss.elastic.co/t/can-i-convert-latin-1-iso8859-1-into-utf-8-available/96898 "2017-08-17T07:14:23Z")
i using JDBC input on logstash but can't read latin-1 into elasticsearch anyone can convert guide for me ? this result problem. --- ## [Logstash 2.4 supports which version of Graylog](https://discuss.elastic.co/t/logstash-2-4-supports-which-version-of-graylog/97345)
**Author:** [@akshay277](https://discuss.elastic.co/u/akshay277)\ **Replies:** 0\ **Last updated:** [August 17, 2017, 6:29am UTC](https://discuss.elastic.co/t/logstash-2-4-supports-which-version-of-graylog/97345 "2017-08-17T06:29:34Z")
Hi, I am planning on upgrading our Logstash setup to 2.4. But before doing that, I would like to know the versions of Graylog that are supported by Logstash 2.4. Currently I am using logstash 1.5.4 and Graylog 1.3.3 … --- ## [Index name from log date not from system time](https://discuss.elastic.co/t/index-name-from-log-date-not-from-system-time/96782)
**Author:** [@hasan\_sharekhan](https://discuss.elastic.co/u/hasan_sharekhan)\ **Replies:** 4\ **Last updated:** [August 17, 2017, 6:09am UTC](https://discuss.elastic.co/t/index-name-from-log-date-not-from-system-time/96782 "2017-08-17T06:09:27Z")
Hi, Today i.e on 11-Aug-2017 I am pushing back dated logs into elastic search via logstash via filebeats I want to create index names of log dates not of current date. Back dates logs are from Mar-Jul 2017, but for al… --- ## [Elasticsearch Input Seems To Not Use Provided Query - It Just Dumps All docs in Index](https://discuss.elastic.co/t/elasticsearch-input-seems-to-not-use-provided-query-it-just-dumps-all-docs-in-index/97157)
**Author:** [@cb2015](https://discuss.elastic.co/u/cb2015)\ **Replies:** 1\ **Last updated:** [August 17, 2017, 5:12am UTC](https://discuss.elastic.co/t/elasticsearch-input-seems-to-not-use-provided-query-it-just-dumps-all-docs-in-index/97157 "2017-08-17T05:12:41Z")
Hello, I am trying to use the Elasticsearch input but keep running into trouble. It seems like the input does not care about the query I give it. Instead, I just see logstash spew out every document in the index. This i… --- ## [Parsing data from the message field of incoming json](https://discuss.elastic.co/t/parsing-data-from-the-message-field-of-incoming-json/97327)
**Author:** [@davidmnoriega](https://discuss.elastic.co/u/davidmnoriega)\ **Replies:** 1\ **Last updated:** [August 17, 2017, 3:46am UTC](https://discuss.elastic.co/t/parsing-data-from-the-message-field-of-incoming-json/97327 "2017-08-17T03:46:29Z")
I’ve configured some services to send their logs to rsyslog, which then in turns sends them to logstash as json. The issue is these services have their own format, they include their own timestamp, etc. { "\_index": "s… --- ## [Kafka Input not committing and causing OOM](https://discuss.elastic.co/t/kafka-input-not-committing-and-causing-oom/97178)
**Author:** [@jalaziz](https://discuss.elastic.co/u/jalaziz)\ **Replies:** 1\ **Last updated:** [August 17, 2017, 12:31am UTC](https://discuss.elastic.co/t/kafka-input-not-committing-and-causing-oom/97178 "2017-08-17T00:31:44Z")
We recently upgraded to ES 5.5.1 and Logstash 5.5.1 from ES 2.4 and Logstash 2.4. Our pipeline is pretty simple: a kafka input with multiple consumer threads subscribing to a pattern, a few filters, and an ES output. I… --- ## [LogStash config not parsing out log events even though config checks out in Grok Debugger](https://discuss.elastic.co/t/logstash-config-not-parsing-out-log-events-even-though-config-checks-out-in-grok-debugger/96653)
**Author:** [@cchooks2](https://discuss.elastic.co/u/cchooks2)\ **Replies:** 5\ **Last updated:** [August 16, 2017, 8:51pm UTC](https://discuss.elastic.co/t/logstash-config-not-parsing-out-log-events-even-though-config-checks-out-in-grok-debugger/96653 "2017-08-16T20:51:08Z")
First post and hoping this is the forum to post questions … I have the following GROK filter: if (\[service\] == “MY\_SERVICE”){ if(\[attributes\]\[file\_path\] == “/apps/log/mydoc/wso2carbon.log”) { grok { patterns\_dir =\> … --- ## [Day of week changes during the day based on timezone](https://discuss.elastic.co/t/day-of-week-changes-during-the-day-based-on-timezone/96875)
**Author:** [@Ricardo\_Jorge](https://discuss.elastic.co/u/Ricardo_Jorge)\ **Replies:** 8\ **Last updated:** [August 16, 2017, 6:42pm UTC](https://discuss.elastic.co/t/day-of-week-changes-during-the-day-based-on-timezone/96875 "2017-08-16T18:42:17Z")
I have a log from where I want to extract week day ( just the number ). It works as I expect up to the hour that timezone is reached. I live at UTC -0300 so before 9 PM week day is OK but after 9 PM week day changes to… --- ## [Multiple elasticsearch input](https://discuss.elastic.co/t/multiple-elasticsearch-input/97180)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 4\ **Last updated:** [August 16, 2017, 5:05pm UTC](https://discuss.elastic.co/t/multiple-elasticsearch-input/97180 "2017-08-16T17:05:16Z")
how to input 2 index using elasticsearch input plugin? TIA --- ## [Cannot disable certificate verification for logstash monitoring](https://discuss.elastic.co/t/cannot-disable-certificate-verification-for-logstash-monitoring/97294)
**Author:** [@bt\_scotth](https://discuss.elastic.co/u/bt_scotth)\ **Replies:** 0\ **Last updated:** [August 16, 2017, 4:00pm UTC](https://discuss.elastic.co/t/cannot-disable-certificate-verification-for-logstash-monitoring/97294 "2017-08-16T16:00:04Z")
I am setting up my cluster with encryption and need to disable certificate verification. All settings work correctly except for xpack monitoring of logstash. There is no setting to disable certification verification for … --- ## [Push mongodb data to elasticsearch using logstash](https://discuss.elastic.co/t/push-mongodb-data-to-elasticsearch-using-logstash/97290)
**Author:** [@sa-shukla](https://discuss.elastic.co/u/sa-shukla)\ **Replies:** 0\ **Last updated:** [August 16, 2017, 3:24pm UTC](https://discuss.elastic.co/t/push-mongodb-data-to-elasticsearch-using-logstash/97290 "2017-08-16T15:24:29Z")
I tried to configure logstash-input-mongodb plugin to crawl mongodb data into elasticsearch, but getting following error message. My Environment is - ELK - 5.5 logstash-input-mongodb plugin - 0.4.1 \[2017-08-16T14:58:… --- ## [Installing logstash-5.5.1 on windows](https://discuss.elastic.co/t/installing-logstash-5-5-1-on-windows/97244)
**Author:** [@bilpor](https://discuss.elastic.co/u/bilpor)\ **Replies:** 0\ **Last updated:** [August 16, 2017, 11:13am UTC](https://discuss.elastic.co/t/installing-logstash-5-5-1-on-windows/97244 "2017-08-16T11:13:04Z")
HI, I need to make use of logstash as I want to be able to index data residing in SQLServer 2012 tables on a windows 2012 r2 server. Looking at the following link Installing Logstash there is no reference to installing … --- ## [New timestamp using dynamic timezone not working](https://discuss.elastic.co/t/new-timestamp-using-dynamic-timezone-not-working/97166)
**Author:** [@bitesize](https://discuss.elastic.co/u/bitesize)\ **Replies:** 3\ **Last updated:** [August 16, 2017, 2:43pm UTC](https://discuss.elastic.co/t/new-timestamp-using-dynamic-timezone-not-working/97166 "2017-08-16T14:43:36Z")
I am trying to add a field called localtimestamp to match the UNIX timestamp received in the json message: {"event": 1,"content": \["test","value1","value2","value3","value4"\],"timestamp": 1502844683} This is my filter … --- ## [Logstash - File mentioned in last\_run\_metadata\_path is not getting created](https://discuss.elastic.co/t/logstash-file-mentioned-in-last-run-metadata-path-is-not-getting-created/96989)
**Author:** [@vincentmarian](https://discuss.elastic.co/u/vincentmarian)\ **Replies:** 1\ **Last updated:** [August 16, 2017, 2:07pm UTC](https://discuss.elastic.co/t/logstash-file-mentioned-in-last-run-metadata-path-is-not-getting-created/96989 "2017-08-16T14:07:07Z")
I am using Logstash 5.4 File mentioned in last\_run\_metadata\_path is not getting created after starting Logstash. Logstash config file: input { jdbc { jdbc\_driver\_library =\> "/app/logstash/dbdrivers/mariadb/mariadb… --- ## [Issue with File input plugin for ASCII C++ program text file format](https://discuss.elastic.co/t/issue-with-file-input-plugin-for-ascii-c-program-text-file-format/97135)
**Author:** [@ankeet.jain](https://discuss.elastic.co/u/ankeet.jain)\ **Replies:** 1\ **Last updated:** [August 16, 2017, 12:22pm UTC](https://discuss.elastic.co/t/issue-with-file-input-plugin-for-ascii-c-program-text-file-format/97135 "2017-08-16T12:22:45Z")
I am facing a problem while reading a file using file input plugin. File input plugin is not able to find the line delimiter for a file and it is reading 500 consecutive lines as a single line. Using file command I che… --- ## [Logstash Shutdownwatcher](https://discuss.elastic.co/t/logstash-shutdownwatcher/97228)
**Author:** [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Replies:** 0\ **Last updated:** [August 16, 2017, 10:00am UTC](https://discuss.elastic.co/t/logstash-shutdownwatcher/97228 "2017-08-16T10:00:34Z")
Hello, I noticed lots of warning in the logs of Logstash. 11:54:50,698\]\[WARN \]\[logstash.shutdownwatcher \] {"inflight\_count"=\>3864... 11:54:55,706\]\[WARN \]\[logstash.shutdownwatcher \] {"inflight\_count"=\>4250... 11:55:00,8… --- ## [Drop everything and only filter for specific fileds](https://discuss.elastic.co/t/drop-everything-and-only-filter-for-specific-fileds/97223)
**Author:** [@sentinel](https://discuss.elastic.co/u/sentinel)\ **Replies:** 6\ **Last updated:** [August 16, 2017, 10:42am UTC](https://discuss.elastic.co/t/drop-everything-and-only-filter-for-specific-fileds/97223 "2017-08-16T10:42:20Z")
In my config file I have a very long list of these drops: if (\[EventType\]=="CachedRateDataStore"){ drop { } } … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=923) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=925)