# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1035 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1036 --- ## [Logstash rotating log - sincedb issue on Windows](https://discuss.elastic.co/t/logstash-rotating-log-sincedb-issue-on-windows/74342)
**Author:** [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\ **Replies:** 4\ **Last updated:** [February 16, 2017, 3:56pm UTC](https://discuss.elastic.co/t/logstash-rotating-log-sincedb-issue-on-windows/74342 "2017-02-16T15:56:56Z")
I am running Logstash 5.2 on Windows7 and I have a rotating log use case, where I maintain a file batchlog.txt everyday and I read this file into logstash and output to ES (daily indices) My file input config: inpu… --- ## [Logstash elasticsearch filter geoip fields nil](https://discuss.elastic.co/t/logstash-elasticsearch-filter-geoip-fields-nil/75224)
**Author:** [@Len\_Rugen](https://discuss.elastic.co/u/Len_Rugen)\ **Replies:** 3\ **Last updated:** [February 16, 2017, 2:31pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-filter-geoip-fields-nil/75224 "2017-02-16T14:31:44Z")
I'm trying to get some of the geoip fields from a prior event using this code: elasticsearch { hosts =\> \[ "myes:9200" \] user =\> "elastic" password =\> "changeme" … --- ## [Tracking of outage duration from ups and downs](https://discuss.elastic.co/t/tracking-of-outage-duration-from-ups-and-downs/75209)
**Author:** [@tarp](https://discuss.elastic.co/u/tarp)\ **Replies:** 2\ **Last updated:** [February 16, 2017, 1:50pm UTC](https://discuss.elastic.co/t/tracking-of-outage-duration-from-ups-and-downs/75209 "2017-02-16T13:50:47Z")
Hi, Recently I added our sitescope alert logs to our Elasticsearch cluster. This is very helpful that shows us when we get down alerts. What I'm not getting is any kind of duration. How long was something down? The s… --- ## [How to transfer original @timestamp from json log to logstash](https://discuss.elastic.co/t/how-to-transfer-original-timestamp-from-json-log-to-logstash/75243)
**Author:** [@sagittarius](https://discuss.elastic.co/u/sagittarius)\ **Replies:** 3\ **Last updated:** [February 16, 2017, 12:52pm UTC](https://discuss.elastic.co/t/how-to-transfer-original-timestamp-from-json-log-to-logstash/75243 "2017-02-16T12:52:13Z")
I am trying to setup filebeat-\>logstash-\>elasticsearch chain and I am having problems with @timestamp which is not being transferred from the logfile and a timestamp when the message arrives into logstash is used instead… --- ## [Automatically Updating the Blacklisted IP on a daily basis](https://discuss.elastic.co/t/automatically-updating-the-blacklisted-ip-on-a-daily-basis/42000)
**Author:** [@praveen\_siem](https://discuss.elastic.co/u/praveen_siem)\ **Replies:** 4\ **Last updated:** [February 16, 2017, 10:44am UTC](https://discuss.elastic.co/t/automatically-updating-the-blacklisted-ip-on-a-daily-basis/42000 "2017-02-16T10:44:26Z")
Hello, We are building a rule "Sucessful Connection to Blacklisted IP". We are doing this using translate filter. We are taking the Blacklisted IPs from SANS or other security websites. Is their any way to automatic… --- ## [\[SOLVED\] Get the real date](https://discuss.elastic.co/t/solved-get-the-real-date/75212)
**Author:** [@Romain\_PZK](https://discuss.elastic.co/u/Romain_PZK)\ **Replies:** 2\ **Last updated:** [February 16, 2017, 10:07am UTC](https://discuss.elastic.co/t/solved-get-the-real-date/75212 "2017-02-16T10:07:00Z")
Hello, I have a logstash running well, but I don't understand how to get the real date instead of the timestamp. I have read this doc Date filter plugin | Logstash Reference \[8.11\] | Elastic The is my config file : … --- ## [Bro IDS with ELK stack](https://discuss.elastic.co/t/bro-ids-with-elk-stack/74637)
**Author:** [@ruby](https://discuss.elastic.co/u/ruby)\ **Replies:** 5\ **Last updated:** [February 16, 2017, 9:40am UTC](https://discuss.elastic.co/t/bro-ids-with-elk-stack/74637 "2017-02-16T09:40:12Z")
I have configured BRO -IDS on PC along with Filebeat Shiipper and from another PC on which I have installed ELK stack, I need to fetch BRO IDS logs. please help to know what to configure on ELK stack..is there any ne… --- ## [Logstash set hour %{+HH} timezone on strings?](https://discuss.elastic.co/t/logstash-set-hour-hh-timezone-on-strings/75236)
**Author:** [@mostolog](https://discuss.elastic.co/u/mostolog)\ **Replies:** 3\ **Last updated:** [February 16, 2017, 9:13am UTC](https://discuss.elastic.co/t/logstash-set-hour-hh-timezone-on-strings/75236 "2017-02-16T09:13:33Z")
Hi When dealing with Logstash dates, anyone can easily convert a text string date field to a real date (with proper TZ and DST) using date filter: date { timezone =\> "Europe/Madrid" match =\> \["mytim… --- ## [Logstash- extraction of data from a column in csv](https://discuss.elastic.co/t/logstash-extraction-of-data-from-a-column-in-csv/75284)
**Author:** [@nilanjana](https://discuss.elastic.co/u/nilanjana)\ **Replies:** 4\ **Last updated:** [February 16, 2017, 7:30am UTC](https://discuss.elastic.co/t/logstash-extraction-of-data-from-a-column-in-csv/75284 "2017-02-16T07:30:28Z")
Hi , I have a csv file with where the is a column called "description" which has got random lines and varies with each record in csv . For one of the record it looks like this : JOB RUN: 'AAL ATLAS ETL ATRMS Recipe' … --- ## [Parsing specific values from logs in logstash configuration](https://discuss.elastic.co/t/parsing-specific-values-from-logs-in-logstash-configuration/75288)
**Author:** [@agrawalsaurabh](https://discuss.elastic.co/u/agrawalsaurabh)\ **Replies:** 8\ **Last updated:** [February 16, 2017, 7:28am UTC](https://discuss.elastic.co/t/parsing-specific-values-from-logs-in-logstash-configuration/75288 "2017-02-16T07:28:14Z")
Hi, I am currently using default logstash configuration to parse the apache server access log file and index it to elasticsearch. However, I don't want to parse all fields in log but say specific ones like IP, Referrer… --- ## [Grok on multiple continues lines](https://discuss.elastic.co/t/grok-on-multiple-continues-lines/73970)
**Author:** [@Govind\_Raj](https://discuss.elastic.co/u/Govind_Raj)\ **Replies:** 18\ **Last updated:** [February 16, 2017, 7:07am UTC](https://discuss.elastic.co/t/grok-on-multiple-continues-lines/73970 "2017-02-16T07:07:08Z")
Hi We have a pattern as follows, if the application works fine then not more 1 occurance of following log line will be present, 2017-02-04 17:25:21 20713 20:25:21.251 timeman(170) Note: NtpdateWrapper:: StartNt… --- ## [Logstash not logging](https://discuss.elastic.co/t/logstash-not-logging/75254)
**Author:** [@timmy8ken](https://discuss.elastic.co/u/timmy8ken)\ **Replies:** 4\ **Last updated:** [February 16, 2017, 3:37am UTC](https://discuss.elastic.co/t/logstash-not-logging/75254 "2017-02-16T03:37:51Z")
Hello, I have Logstash 5.2.0 running on a Server 2016 server and I am trying to enable internal logging and I can't get it to write to a log file. I have tried the following: Edited logstash.yml with the following … --- ## [Error message using logstash(Response code: 429, 500)](https://discuss.elastic.co/t/error-message-using-logstash-response-code-429-500/75264)
**Author:** [@Suin\_You](https://discuss.elastic.co/u/Suin_You)\ **Replies:** 0\ **Last updated:** [February 16, 2017, 2:49am UTC](https://discuss.elastic.co/t/error-message-using-logstash-response-code-429-500/75264 "2017-02-16T02:49:00Z")
I got the two error messages and how could I do? The first problem is \[2017-02-16T11:42:17,332\]\[ERROR\]\[logstash.outputs.elasticsearch\] Got a bad response code from server, but this code is not considered retryable. R… --- ## [Pacemaker Resource Agent for Logstash](https://discuss.elastic.co/t/pacemaker-resource-agent-for-logstash/75257)
**Author:** [@ThomasB](https://discuss.elastic.co/u/ThomasB)\ **Replies:** 0\ **Last updated:** [February 16, 2017, 1:00am UTC](https://discuss.elastic.co/t/pacemaker-resource-agent-for-logstash/75257 "2017-02-16T01:00:13Z")
Hi, I am working on a Project for a logstash fail-over cluster (using pacemaker) and I am looking for an OCF (http://linux-ha.org/wiki/OCF\_Resource\_Agents) resource agent for logstash. My goal is to have Geo-reduntant Ac… --- ## [How to GET all events since a certain time](https://discuss.elastic.co/t/how-to-get-all-events-since-a-certain-time/75244)
**Author:** [@Ben\_Hoffman](https://discuss.elastic.co/u/Ben_Hoffman)\ **Replies:** 1\ **Last updated:** [February 15, 2017, 10:11pm UTC](https://discuss.elastic.co/t/how-to-get-all-events-since-a-certain-time/75244 "2017-02-15T22:11:03Z")
So currently I am using this POST query to get my data via HTTP: { "query": { "match\_all": {} }, "size": 1, "sort": \[{ "@timestamp": { "order": "desc" } }\] } This get's me the data that I want, but only one at a … --- ## [Why does Logstash write into two indices? (default and custom ones)](https://discuss.elastic.co/t/why-does-logstash-write-into-two-indices-default-and-custom-ones/75233)
**Author:** [@Mr.King](https://discuss.elastic.co/u/Mr.King)\ **Replies:** 4\ **Last updated:** [February 15, 2017, 10:10pm UTC](https://discuss.elastic.co/t/why-does-logstash-write-into-two-indices-default-and-custom-ones/75233 "2017-02-15T22:10:11Z")
Hello, I am using Elasticsearch 5.2 and Logstash 5.2 . My problem is that Logstash is writing into the default index (format logstash-2017.02.15) and into my custom one. I just want it to write only into my "logstash… --- ## [New operation for the mutate filter?](https://discuss.elastic.co/t/new-operation-for-the-mutate-filter/75070)
**Author:** [@val](https://discuss.elastic.co/u/val)\ **Replies:** 2\ **Last updated:** [February 15, 2017, 9:41pm UTC](https://discuss.elastic.co/t/new-operation-for-the-mutate-filter/75070 "2017-02-15T21:41:41Z")
I often find myself using the ruby filter below in order to "move" a sub-structure at the root level in order to use it as the main event. ruby { code =\> " event.to\_hash.delete\_if {|k, v| k != 'payload'}… --- ## [Dynamic Bucket Names or Directories in AWS S3 Output](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459)
**Author:** [@umutcan](https://discuss.elastic.co/u/umutcan)\ **Replies:** 9\ **Last updated:** [February 15, 2017, 8:57pm UTC](https://discuss.elastic.co/t/dynamic-bucket-names-or-directories-in-aws-s3-output/2459 "2017-02-15T20:57:03Z")
Hi, I am doing some tests on storing data on AWS S3. I have read the documents and couldn't find a dynamic bucket name or directory option like Elasticsearch output provides in index name. Is there a way to do this? T… --- ## [Must delete files in message queue to start logstash](https://discuss.elastic.co/t/must-delete-files-in-message-queue-to-start-logstash/71907)
**Author:** [@panaman](https://discuss.elastic.co/u/panaman)\ **Replies:** 18\ **Last updated:** [February 15, 2017, 8:06pm UTC](https://discuss.elastic.co/t/must-delete-files-in-message-queue-to-start-logstash/71907 "2017-02-15T20:06:48Z")
For some reason if i restart logstash and there is a few gig of messages in the queue logstash complains that there is a config error \[2017-01-17T20:34:41,732\]\[ERROR\]\[logstash.agent \] fetched an invalid con… --- ## [Set data type to float (vs long) on filesystem.fs.used\_p](https://discuss.elastic.co/t/set-data-type-to-float-vs-long-on-filesystem-fs-used-p/74287)
**Author:** [@itsecureadmin](https://discuss.elastic.co/u/itsecureadmin)\ **Replies:** 1\ **Last updated:** [February 15, 2017, 7:08pm UTC](https://discuss.elastic.co/t/set-data-type-to-float-vs-long-on-filesystem-fs-used-p/74287 "2017-02-15T19:08:24Z")
I am using logstash 2.2 and Elastic Search 1.5 on AWS and am not able to force the filesystem.fs.used\_p value to be stored as a float so that I can check the filesystem used percentage properly. I'm looking for help th… --- ## [Logstash Email throttle not working](https://discuss.elastic.co/t/logstash-email-throttle-not-working/75230)
**Author:** [@amitkashyap](https://discuss.elastic.co/u/amitkashyap)\ **Replies:** 3\ **Last updated:** [February 15, 2017, 7:00pm UTC](https://discuss.elastic.co/t/logstash-email-throttle-not-working/75230 "2017-02-15T19:00:41Z")
I am able to generate mails based on events in log file but I am not able to throttle it. below is my config file. ============================================ input { beats { port =\> 5044 } } filter { … --- ## [Logstash Does not extract timestamp from JSON](https://discuss.elastic.co/t/logstash-does-not-extract-timestamp-from-json/52151)
**Author:** [@sunnysharma](https://discuss.elastic.co/u/sunnysharma)\ **Replies:** 5\ **Last updated:** [February 15, 2017, 6:17pm UTC](https://discuss.elastic.co/t/logstash-does-not-extract-timestamp-from-json/52151 "2017-02-15T18:17:32Z")
Hi Folks, Logstash noob here. I am trying to put some JSON data but logstash is not setting my log data timestamp to @timestamp field. It continues to use the time when the data is read into logstash. Here's the log… --- ## [Trying to use custom patterns without success](https://discuss.elastic.co/t/trying-to-use-custom-patterns-without-success/75119)
**Author:** [@gcranshaw](https://discuss.elastic.co/u/gcranshaw)\ **Replies:** 3\ **Last updated:** [February 15, 2017, 4:08pm UTC](https://discuss.elastic.co/t/trying-to-use-custom-patterns-without-success/75119 "2017-02-15T16:08:20Z")
I have an ELK stack setup on Ubuntu 16.04 all using version 5.2. Everything is working fine. I am using the below 05-ciscoasa-filter.conf file in /etc/logstash/conf.d to sort logs from a Cisco ASA that are being dumped t… --- ## [Logstash elasticsearch filter error TypeError: can't convert nil into String](https://discuss.elastic.co/t/logstash-elasticsearch-filter-error-typeerror-cant-convert-nil-into-string/75125)
**Author:** [@Len\_Rugen](https://discuss.elastic.co/u/Len_Rugen)\ **Replies:** 3\ **Last updated:** [February 15, 2017, 2:17pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-filter-error-typeerror-cant-convert-nil-into-string/75125 "2017-02-15T14:17:19Z")
I'm new to elk, trying to query es from logstash with this: elasticsearch { hosts =\> \[ "my-elk-server:9200" \] user =\> "elastic" password =\> "changeme" i… --- ## [Aggregate events with START but no END](https://discuss.elastic.co/t/aggregate-events-with-start-but-no-end/75208)
**Author:** [@frontman](https://discuss.elastic.co/u/frontman)\ **Replies:** 0\ **Last updated:** [February 15, 2017, 2:09pm UTC](https://discuss.elastic.co/t/aggregate-events-with-start-but-no-end/75208 "2017-02-15T14:09:42Z")
Hi all, I'm new to the ELK family and currently encountering a problem. I'm using the logstash-filter-aggregate to aggregate events that have a clear START event (second field containing 70), but no END event. The task… --- ## [Question about multiple conf files](https://discuss.elastic.co/t/question-about-multiple-conf-files/75186)
**Author:** [@lendoly](https://discuss.elastic.co/u/lendoly)\ **Replies:** 2\ **Last updated:** [February 15, 2017, 1:02pm UTC](https://discuss.elastic.co/t/question-about-multiple-conf-files/75186 "2017-02-15T13:02:15Z")
Hello, I'm having some troubles with my conf file (more than 300 lines and growing) so I decide to split it on severals files and looking for information about that I read that is not possible to duplicate the input o… --- ## [Logstash 5.0 parsing jason](https://discuss.elastic.co/t/logstash-5-0-parsing-jason/72681)
**Author:** [@fabian](https://discuss.elastic.co/u/fabian)\ **Replies:** 8\ **Last updated:** [February 15, 2017, 12:57pm UTC](https://discuss.elastic.co/t/logstash-5-0-parsing-jason/72681 "2017-02-15T12:57:56Z")
I'm new to Elasticsearch 5.0 with Logstash 5.0 and I want to parse some bigger json file. I put an example on Json File. What I try to do is split the json on Data.InstallAnalysis.Analysis.TCommServerInstallAnalysis.\* … --- ## [Common config for file input plugins](https://discuss.elastic.co/t/common-config-for-file-input-plugins/75190)
**Author:** [@Mayank\_Agrawal](https://discuss.elastic.co/u/Mayank_Agrawal)\ **Replies:** 1\ **Last updated:** [February 15, 2017, 12:57pm UTC](https://discuss.elastic.co/t/common-config-for-file-input-plugins/75190 "2017-02-15T12:57:34Z")
Hi, Can I have common config for all the file input plugins in logstash configuration file ? Example: input{ file { type =\> "abc" path =\> "C:/user/logs/\*abc.log\*" sincedb\_path =\> "C:/user/logs/abc… --- ## [Logstash csv - date and file format](https://discuss.elastic.co/t/logstash-csv-date-and-file-format/75083)
**Author:** [@cthomps](https://discuss.elastic.co/u/cthomps)\ **Replies:** 4\ **Last updated:** [February 15, 2017, 12:43pm UTC](https://discuss.elastic.co/t/logstash-csv-date-and-file-format/75083 "2017-02-15T12:43:37Z")
I'm having trouble getting my timestamp field from csv file into the @timestamp in ES. I have tried the timestamp field a few different ways, trying to stay in a standard if possible. In the debug logs i see an error… --- ## [Logstash Filter Pattern for Weblogic application logs](https://discuss.elastic.co/t/logstash-filter-pattern-for-weblogic-application-logs/60078)
**Author:** [@dvkawa](https://discuss.elastic.co/u/dvkawa)\ **Replies:** 5\ **Last updated:** [February 15, 2017, 11:36am UTC](https://discuss.elastic.co/t/logstash-filter-pattern-for-weblogic-application-logs/60078 "2017-02-15T11:36:27Z")
Hi , I am very new to ELK stack and I am trying to parse our weblogic application server logs using logstash and feed that to Elastic Search index and view in Kibana. Although I am able to get that indexed, I am looki… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1034) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1036)