# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=877 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 878 --- ## [How to capture docker logs](https://discuss.elastic.co/t/how-to-capture-docker-logs/107228)
**Author:** [@obodard](https://discuss.elastic.co/u/obodard)\ **Replies:** 0\ **Last updated:** [November 10, 2017, 6:29pm UTC](https://discuss.elastic.co/t/how-to-capture-docker-logs/107228 "2017-11-10T18:29:13Z")
I have Docker containers deployed under Kubernetes. I need to capture the Docker logs which are sent to its console, and I would like to avoid dumping those logs in a file for each container. What is the best way to cap… --- ## [Hourly log rotation?](https://discuss.elastic.co/t/hourly-log-rotation/106958)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 1\ **Last updated:** [November 10, 2017, 3:33pm UTC](https://discuss.elastic.co/t/hourly-log-rotation/106958 "2017-11-10T15:33:58Z")
Hi, Is it possible to create hourly log files? output { if \[type\] == "netflow" { file { path =\> "/home/test/Desktop/test/netflow-%{+YYYY-MM-dd-HH}.gz" gzip =\> true } } } Creates a properly timestamped file whe… --- ## [Logstash output zabbix gemspec](https://discuss.elastic.co/t/logstash-output-zabbix-gemspec/106352)
**Author:** [@vardhasu](https://discuss.elastic.co/u/vardhasu)\ **Replies:** 1\ **Last updated:** [November 10, 2017, 3:21pm UTC](https://discuss.elastic.co/t/logstash-output-zabbix-gemspec/106352 "2017-11-10T15:21:35Z")
Hi Installed logstash 5.5.1, downloaded logstash-output-zabbix-3.0.3.gemspec. When I run gem install logstash-output-zabbix-3.0.3.gemspec, I get ERROR: While executing gem ... (Gem::DependencyError) Unable to resolve… --- ## [CSV Filter](https://discuss.elastic.co/t/csv-filter/107199)
**Author:** [@emre141](https://discuss.elastic.co/u/emre141)\ **Replies:** 0\ **Last updated:** [November 10, 2017, 2:24pm UTC](https://discuss.elastic.co/t/csv-filter/107199 "2017-11-10T14:24:53Z")
Hello, I have a very customizable log lines.They are several lines consist of different column count.My log lines are below BDTz;i68;SiEQ\_WR;s1;TSz0.01;PFr0.01;PTo999999.999; BDTz;i70;SiEQ\_FN;s1;TSz0.01;PFr0.01;PTo49.… --- ## [Update search results with new inserted data](https://discuss.elastic.co/t/update-search-results-with-new-inserted-data/106437)
**Author:** [@parsa](https://discuss.elastic.co/u/parsa)\ **Replies:** 12\ **Last updated:** [November 10, 2017, 2:02pm UTC](https://discuss.elastic.co/t/update-search-results-with-new-inserted-data/106437 "2017-11-10T14:02:51Z")
How can I update search results for new inserted data? What can I do for it update automatically? --- ## [Default schedule logstash (issue with indexing to Elasticsearch)](https://discuss.elastic.co/t/default-schedule-logstash-issue-with-indexing-to-elasticsearch/107196)
**Author:** [@malhotras](https://discuss.elastic.co/u/malhotras)\ **Replies:** 0\ **Last updated:** [November 10, 2017, 1:59pm UTC](https://discuss.elastic.co/t/default-schedule-logstash-issue-with-indexing-to-elasticsearch/107196 "2017-11-10T13:59:51Z")
I am facing problem with Indexing data to Elasticsearch using logstash. I noticed that Logstash is scheduled to restart changing process id after certain point of time. This is creating problem with indexing data to the … --- ## [Unable to install logstash-filter-elapsed logstash 5.5.0](https://discuss.elastic.co/t/unable-to-install-logstash-filter-elapsed-logstash-5-5-0/107164)
**Author:** [@shubham\_kumar1](https://discuss.elastic.co/u/shubham_kumar1)\ **Replies:** 0\ **Last updated:** [November 10, 2017, 10:20am UTC](https://discuss.elastic.co/t/unable-to-install-logstash-filter-elapsed-logstash-5-5-0/107164 "2017-11-10T10:20:50Z")
Hi all, i was tryining to install logstash-filter-\* in my logstash-5.5.0, its throwing this error Error:- Validating logstash-filter-elapsed Installing logstash-filter-elapsed Error Bundler::InstallError, retrying 1… --- ## [Have to run logstash config file automatically](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195)
**Author:** [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Replies:** 16\ **Last updated:** [November 10, 2017, 9:27am UTC](https://discuss.elastic.co/t/have-to-run-logstash-config-file-automatically/106195 "2017-11-10T09:27:52Z")
Hi, I'm running logstash on linux machine as a service. I have 2 config files through which data is indexing. Now, I want this logstash configs to run as a service ( so that indexing will not stop after restarting the… --- ## [Remove duplicate field value](https://discuss.elastic.co/t/remove-duplicate-field-value/107125)
**Author:** [@tharu85](https://discuss.elastic.co/u/tharu85)\ **Replies:** 3\ **Last updated:** [November 10, 2017, 9:14am UTC](https://discuss.elastic.co/t/remove-duplicate-field-value/107125 "2017-11-10T09:14:13Z")
In my logstash output I received following output. "log\_name" =\> \[ \[0\] "Authentication", \[1\] "Authentication" \], "log\_level" =\> "info", "session\_id" =\> \[ \[0\] "4280340685", \[1… --- ## [Field overwrite is not working](https://discuss.elastic.co/t/field-overwrite-is-not-working/107158)
**Author:** [@ou\_snaaksie](https://discuss.elastic.co/u/ou_snaaksie)\ **Replies:** 1\ **Last updated:** [November 10, 2017, 9:13am UTC](https://discuss.elastic.co/t/field-overwrite-is-not-working/107158 "2017-11-10T09:13:32Z")
I want to only keep a certain part of a syslog message as the message. input { udp { port =\> "514" type =\> "syslog-cisco" } tcp { port =\> "514" type =\> "syslog-cisco" } }… --- ## [How to send logstash exec cmd output to elasticsearch](https://discuss.elastic.co/t/how-to-send-logstash-exec-cmd-output-to-elasticsearch/107093)
**Author:** [@RohanKumbhar](https://discuss.elastic.co/u/RohanKumbhar)\ **Replies:** 6\ **Last updated:** [November 10, 2017, 8:31am UTC](https://discuss.elastic.co/t/how-to-send-logstash-exec-cmd-output-to-elasticsearch/107093 "2017-11-10T08:31:49Z")
Hi , I 'm using exec output plugin and i want to send output of exec cmd to elasticsearch. My logstash conf as shown below input { file { path =\> "/log/sample.log" start\_position =\> "beginning" } } filter… --- ## [Log Annotation](https://discuss.elastic.co/t/log-annotation/107138)
**Author:** [@Sanjeev\_Routray](https://discuss.elastic.co/u/Sanjeev_Routray)\ **Replies:** 2\ **Last updated:** [November 10, 2017, 6:45am UTC](https://discuss.elastic.co/t/log-annotation/107138 "2017-11-10T06:45:58Z")
So we have log files from SQL and Oracle logs. I want to know what the typical fields are that we need to extract (annotate) through logstash before sending it for indexing. Not sure if Filebeat has that domain intellige… --- ## [How to index json data into elasticsearch](https://discuss.elastic.co/t/how-to-index-json-data-into-elasticsearch/107035)
**Author:** [@dhanunjayavarma](https://discuss.elastic.co/u/dhanunjayavarma)\ **Replies:** 3\ **Last updated:** [November 10, 2017, 5:54am UTC](https://discuss.elastic.co/t/how-to-index-json-data-into-elasticsearch/107035 "2017-11-10T05:54:34Z")
let say I have json file with this type of data { "\_id":1, "cid":3067, "name":"Tech Mahindra", "type":\[ "Information Technology and Services" \], "empSize":99704, "empSizeInfo":\[ { "on":1507141800, "size":9970… --- ## [If condition not working in Output](https://discuss.elastic.co/t/if-condition-not-working-in-output/106849)
**Author:** [@Telstra\_Aus](https://discuss.elastic.co/u/Telstra_Aus)\ **Replies:** 2\ **Last updated:** [November 10, 2017, 3:02am UTC](https://discuss.elastic.co/t/if-condition-not-working-in-output/106849 "2017-11-10T03:02:08Z")
Hi, Below is my logstash.conf file content and if condition in output and is not working, input { beats { type =\> beats port =\> 5044 host =\> "XX.XX.XX.XX" } } filter { multiline { pattern =\> "^####" negate =\>… --- ## [Imap input plugin error: Error: syntax: expecting '(', found '\\' on win2012](https://discuss.elastic.co/t/imap-input-plugin-error-error-syntax-expecting-found-on-win2012/106901)
**Author:** [@lintungwei](https://discuss.elastic.co/u/lintungwei)\ **Replies:** 0\ **Last updated:** [November 8, 2017, 3:41pm UTC](https://discuss.elastic.co/t/imap-input-plugin-error-error-syntax-expecting-found-on-win2012/106901 "2017-11-08T15:41:18Z")
Hi, I am setting up the imap input plugin. It works but after read several emails (fetch\_count). It give me the error "Error: syntax: expecting '(', found ''". Anyone have the same problem ? Thanks in advance. My env … --- ## [Netscaler appflow](https://discuss.elastic.co/t/netscaler-appflow/107117)
**Author:** [@scherma](https://discuss.elastic.co/u/scherma)\ **Replies:** 0\ **Last updated:** [November 9, 2017, 11:33pm UTC](https://discuss.elastic.co/t/netscaler-appflow/107117 "2017-11-09T23:33:37Z")
Is it possible to reassemble Netscaler appflow http transactions within Logstash? I've worked out a way to do it outside of the application but Ruby isn't my strong point (understatement. I do not know ruby at all) so I … --- ## [Logstash-plugin installs as root without global read](https://discuss.elastic.co/t/logstash-plugin-installs-as-root-without-global-read/107116)
**Author:** [@lookcrabs](https://discuss.elastic.co/u/lookcrabs)\ **Replies:** 0\ **Last updated:** [November 9, 2017, 11:15pm UTC](https://discuss.elastic.co/t/logstash-plugin-installs-as-root-without-global-read/107116 "2017-11-09T23:15:33Z")
I'm using 2 fresh ubuntu 16.04 vms with 16G of ram. I just installed logstash (5.6.4) and java (1.8) and setup the following filter:: filter { if "oslofmt" in \[tags\] { multiline { negate =\> true patter… --- ## [After upgrade to 5.6.2 Logstash wont replace the environment variables](https://discuss.elastic.co/t/after-upgrade-to-5-6-2-logstash-wont-replace-the-environment-variables/106919)
**Author:** [@Deepakv26](https://discuss.elastic.co/u/Deepakv26)\ **Replies:** 1\ **Last updated:** [November 9, 2017, 10:02pm UTC](https://discuss.elastic.co/t/after-upgrade-to-5-6-2-logstash-wont-replace-the-environment-variables/106919 "2017-11-09T22:02:23Z")
Hi, We were using Logstash 1 and now we are upgrading to Logstash 5.6.2. As part of upgrade, we are doing the sanity check and we found that the environment variables are not getting replaced and we are getting the bel… --- ## [Date parse failure and date formatter issue](https://discuss.elastic.co/t/date-parse-failure-and-date-formatter-issue/106925)
**Author:** [@guruprasadh4](https://discuss.elastic.co/u/guruprasadh4)\ **Replies:** 10\ **Last updated:** [November 9, 2017, 10:01pm UTC](https://discuss.elastic.co/t/date-parse-failure-and-date-formatter-issue/106925 "2017-11-09T22:01:02Z")
Sample log - IDENTIFIER TIMESTAMP T C RESOURCE\_NAME DESCRIPTION C6E26F3B 1002042417 I H hdisk111 BACK-UP PATH STATUS CHANGE C6E26F3B 1002042417 I H hdisk103 BACK-UP PATH STATUS CHANGE Config File - … --- ## [Logstash filter for akamai logs](https://discuss.elastic.co/t/logstash-filter-for-akamai-logs/106677)
**Author:** [@giangra](https://discuss.elastic.co/u/giangra)\ **Replies:** 1\ **Last updated:** [November 9, 2017, 9:58pm UTC](https://discuss.elastic.co/t/logstash-filter-for-akamai-logs/106677 "2017-11-09T21:58:46Z")
Hi everyone, I have some problem creating a logstash configuration file to parse akamai logs. In my case, Akamai logs are in a FTP server and they don't have the typical .log extension, because they're all zipped (.gz). … --- ## [Custom timestamp field off by timezone offset](https://discuss.elastic.co/t/custom-timestamp-field-off-by-timezone-offset/107077)
**Author:** [@rsaeks](https://discuss.elastic.co/u/rsaeks)\ **Replies:** 1\ **Last updated:** [November 9, 2017, 9:46pm UTC](https://discuss.elastic.co/t/custom-timestamp-field-off-by-timezone-offset/107077 "2017-11-09T21:46:57Z")
I'm working on revamping some indexes for data and am running into an issue with times. I'm currently ingesting entries like this one below: \<164\>Nov 09 2017 11:26:09: %ASA-4-106023: Deny tcp src outside:191.101.167.235… --- ## [Logstash 2.4.0: Ruby exception occurred: undefined method \`/'](https://discuss.elastic.co/t/logstash-2-4-0-ruby-exception-occurred-undefined-method/107070)
**Author:** [@AlessandroC985](https://discuss.elastic.co/u/AlessandroC985)\ **Replies:** 3\ **Last updated:** [November 9, 2017, 9:42pm UTC](https://discuss.elastic.co/t/logstash-2-4-0-ruby-exception-occurred-undefined-method/107070 "2017-11-09T21:42:07Z")
I'm trying to modify the value of a numeric field (type long) with logstash 2.4.0 using the ruby filter. This is my filter configuration: ruby { code =\> "event.set('myField', event.get('myField') / 1000)" } I have… --- ## [Logstash won't start: permission problem?](https://discuss.elastic.co/t/logstash-wont-start-permission-problem/106886)
**Author:** [@hispeed](https://discuss.elastic.co/u/hispeed)\ **Replies:** 5\ **Last updated:** [November 9, 2017, 9:40pm UTC](https://discuss.elastic.co/t/logstash-wont-start-permission-problem/106886 "2017-11-09T21:40:40Z")
I have updated to ELK 5.4.6 now I get the following error when starting Logstash? Is this a permission problem? Any ideas where I have to search? \[2017-11-08T15:44:24,340\]\[FATAL\]\[logstash.runner \] An unexpecte… --- ## [Field conversion](https://discuss.elastic.co/t/field-conversion/107086)
**Author:** [@jspeer](https://discuss.elastic.co/u/jspeer)\ **Replies:** 1\ **Last updated:** [November 9, 2017, 8:59pm UTC](https://discuss.elastic.co/t/field-conversion/107086 "2017-11-09T20:59:10Z")
I'm trying to convert from strings to floats, in logstash, and push this to elasticsearch, I've deleted the index, however once it's created the field is still of type string. This is the field I'm splitting, and conver… --- ## [Tracking\_column not found in dataset](https://discuss.elastic.co/t/tracking-column-not-found-in-dataset/99044)
**Author:** [@aliyesami](https://discuss.elastic.co/u/aliyesami)\ **Replies:** 2\ **Last updated:** [November 9, 2017, 4:26pm UTC](https://discuss.elastic.co/t/tracking-column-not-found-in-dataset/99044 "2017-11-09T16:26:51Z")
what does this error means? my lostash load is continuing but keep getting this intermittent messages . I checked the tracking column "TXN\_PROCESS\_DATE" for nulls but it doesn't have any null values and its of date typ… --- ## [Collect logs from two NIC cards on same server](https://discuss.elastic.co/t/collect-logs-from-two-nic-cards-on-same-server/106732)
**Author:** [@bambam](https://discuss.elastic.co/u/bambam)\ **Replies:** 11\ **Last updated:** [November 9, 2017, 4:17pm UTC](https://discuss.elastic.co/t/collect-logs-from-two-nic-cards-on-same-server/106732 "2017-11-09T16:17:13Z")
I have Kibana (Version 4.3.1) installed on Red Hat release 7.1 and it's working for the most part. I have two network cards on the computer and logstash is only collecting logs from one network card. The network cards ar… --- ## [Lumberjack input/output plugin AWS LoadBalancer](https://discuss.elastic.co/t/lumberjack-input-output-plugin-aws-loadbalancer/107015)
**Author:** [@Jerry1](https://discuss.elastic.co/u/Jerry1)\ **Replies:** 2\ **Last updated:** [November 9, 2017, 2:50pm UTC](https://discuss.elastic.co/t/lumberjack-input-output-plugin-aws-loadbalancer/107015 "2017-11-09T14:50:48Z")
Hi, I have running two logstash server on different kubernetes cluster with the lumberjack input/output plugin. My config. Client: output { lumberjack { hosts =\> \[ "XXXXXXXXXXXXXXXXXXXXXXXXXXXX-CCCCCCCCC.us-east-1.… --- ## [Logstash logs two filebeat inputs](https://discuss.elastic.co/t/logstash-logs-two-filebeat-inputs/107039)
**Author:** [@rromanovic](https://discuss.elastic.co/u/rromanovic)\ **Replies:** 2\ **Last updated:** [November 9, 2017, 2:07pm UTC](https://discuss.elastic.co/t/logstash-logs-two-filebeat-inputs/107039 "2017-11-09T14:07:25Z")
Hi, I've noticed that I have duplicate data in Elasticsearch. After some investigation I found following repeating error in my logstash log: \[ERROR\]\[logstash.pipeline \] A plugin had an unrecoverable error. Will … --- ## [Message has no content with ActiveMQ as source](https://discuss.elastic.co/t/message-has-no-content-with-activemq-as-source/107006)
**Author:** [@moriol](https://discuss.elastic.co/u/moriol)\ **Replies:** 1\ **Last updated:** [November 9, 2017, 1:00pm UTC](https://discuss.elastic.co/t/message-has-no-content-with-activemq-as-source/107006 "2017-11-09T13:00:31Z")
i am using Logstash, with stomp as an input plugin configured, which should collect messages from ActiveMQ. In ActiveMq, I see the following as the payload: {xyz:null=\[\[ Date: 2017/11/09 09:24:24.687 - Value: test=123.… --- ## [Using LogStash JDBC Input plugin to Fetch CLOB data from Oracle Database](https://discuss.elastic.co/t/using-logstash-jdbc-input-plugin-to-fetch-clob-data-from-oracle-database/107041)
**Author:** [@Aryan\_Dogra](https://discuss.elastic.co/u/Aryan_Dogra)\ **Replies:** 0\ **Last updated:** [November 9, 2017, 12:14pm UTC](https://discuss.elastic.co/t/using-logstash-jdbc-input-plugin-to-fetch-clob-data-from-oracle-database/107041 "2017-11-09T12:14:49Z")
Whether this is doable or achievable : To Fetch CLOB data from Oracle using LogStash JDBC Input Plugin We are storing log Files in Oracle Table where Column is of type CLOB How can i retrieve the CLOB data and how can … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=876) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=878)