# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=878 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 879 --- ## [Occasional duplicate entries going in to different indices](https://discuss.elastic.co/t/occasional-duplicate-entries-going-in-to-different-indices/107040)
**Author:** [@paulglavin](https://discuss.elastic.co/u/paulglavin)\ **Replies:** 0\ **Last updated:** [November 9, 2017, 12:12pm UTC](https://discuss.elastic.co/t/occasional-duplicate-entries-going-in-to-different-indices/107040 "2017-11-09T12:12:39Z")
I have a syslog node that is outputting to logstash on TCP. I seem to occasionally get duplicate log entries, one going to where I expect, the other goes to a new index. My logstash syslog config is this: input { tcp… --- ## [What is the date format to be used with Logstash to match 9 digits millisecond](https://discuss.elastic.co/t/what-is-the-date-format-to-be-used-with-logstash-to-match-9-digits-millisecond/107032)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 1\ **Last updated:** [November 9, 2017, 11:40am UTC](https://discuss.elastic.co/t/what-is-the-date-format-to-be-used-with-logstash-to-match-9-digits-millisecond/107032 "2017-11-09T11:40:04Z")
Hi, I am using Logstash 5.6.3. I have a date time field in my log entry like 07-11-17 13:35:35.656000000 . I would like to use it as my @timestamp, so that I am using date filter to match it and target to @timestamp. A… --- ## [Hosting Logstash tips and tricks](https://discuss.elastic.co/t/hosting-logstash-tips-and-tricks/107026)
**Author:** [@pj.vandendriessche](https://discuss.elastic.co/u/pj.vandendriessche)\ **Replies:** 0\ **Last updated:** [November 9, 2017, 10:56am UTC](https://discuss.elastic.co/t/hosting-logstash-tips-and-tricks/107026 "2017-11-09T10:56:47Z")
Totally new to the ELK stack and a beginner on programming level speaking here. I currently have a Django Python application running on Heroku. It works with Papertrail logging. Papertrail receives the logs through a so… --- ## [GeoLite2 and ASN in v.5.6](https://discuss.elastic.co/t/geolite2-and-asn-in-v-5-6/106713)
**Author:** [@stefws](https://discuss.elastic.co/u/stefws)\ **Replies:** 1\ **Last updated:** [November 9, 2017, 10:52am UTC](https://discuss.elastic.co/t/geolite2-and-asn-in-v-5-6/106713 "2017-11-09T10:52:22Z")
I'm trying to get our Geo location filter to work again after patching from v.2.4 to v.5.6. First I found that I needed manually to convert our geopoint data to floats where previous they were float like their sources (… --- ## [Reading from jms with input-jms](https://discuss.elastic.co/t/reading-from-jms-with-input-jms/106968)
**Author:** [@nugusbayevkk](https://discuss.elastic.co/u/nugusbayevkk)\ **Replies:** 3\ **Last updated:** [November 9, 2017, 9:57am UTC](https://discuss.elastic.co/t/reading-from-jms-with-input-jms/106968 "2017-11-09T09:57:33Z")
guys hello, i test to read message from websphere mq, logstash 5.5.2 - windows 8 elasticsearch 5.5.2 - linux websphere mq 8.0.0.4 i have 2 test case: #1) with filter in logstash.conf #2) without filter #1) averag… --- ## [Aggregation error:](https://discuss.elastic.co/t/aggregation-error/106992)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 0\ **Last updated:** [November 9, 2017, 8:55am UTC](https://discuss.elastic.co/t/aggregation-error/106992 "2017-11-09T08:55:40Z")
Exception: String can't be coerced into Fixnum \[2017-11-09T10:39:40,294\]\[ERROR\]\[logstash.filters.aggregate\] Aggregate exception occurred {:error=\>#\, :code=\>" map\['proces… --- ## [Syslog enquiry](https://discuss.elastic.co/t/syslog-enquiry/106988)
**Author:** [@peterch](https://discuss.elastic.co/u/peterch)\ **Replies:** 3\ **Last updated:** [November 9, 2017, 8:26am UTC](https://discuss.elastic.co/t/syslog-enquiry/106988 "2017-11-09T08:26:44Z")
Dear all, I have a network device and would like to send syslog to logstash. My current logstash.json as below. May i know how to amended. I tried on. Please correct me if I wrong. Thanks logstash.json (before) input { … --- ## [LOGSTASH GETTING STOPPED AFTER USING output-rabbitmq AS OUTPUT](https://discuss.elastic.co/t/logstash-getting-stopped-after-using-output-rabbitmq-as-output/106982)
**Author:** [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\ **Replies:** 1\ **Last updated:** [November 9, 2017, 8:20am UTC](https://discuss.elastic.co/t/logstash-getting-stopped-after-using-output-rabbitmq-as-output/106982 "2017-11-09T08:20:57Z")
MY OUTPUT FILTER: output{ rabbitmq { key =\> "logstashkey" exchange =\> "logstash-rabbitmq" exchange\_type =\> "direct" user =\> "\*\*\*" password =\> "\*\*\*\*" host =\> "\*\*.\*\*.\*.\*\*" port =\> \*\*\*\* durable =\> true persistent =\> true }… --- ## [Why is my timestamp in Logstash output appear different from value in Oracle column?](https://discuss.elastic.co/t/why-is-my-timestamp-in-logstash-output-appear-different-from-value-in-oracle-column/106869)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 7\ **Last updated:** [November 9, 2017, 8:14am UTC](https://discuss.elastic.co/t/why-is-my-timestamp-in-logstash-output-appear-different-from-value-in-oracle-column/106869 "2017-11-09T08:14:23Z")
Hi, I am using Logstash 5.6.3. I am fetching some rows from oracle db. In SQL Developer, my timestamp is displayed like 07-11-17 01:35:35.461000000 PM . When I stdout{ codec =\> json } in Logstash, it is displayed like 2… --- ## [Logstash and Kibana for App Log files recommendations](https://discuss.elastic.co/t/logstash-and-kibana-for-app-log-files-recommendations/106769)
**Author:** [@VIJAY\_JOHN\_BRITTO](https://discuss.elastic.co/u/VIJAY_JOHN_BRITTO)\ **Replies:** 5\ **Last updated:** [November 9, 2017, 6:59am UTC](https://discuss.elastic.co/t/logstash-and-kibana-for-app-log-files-recommendations/106769 "2017-11-09T06:59:53Z")
Hello All We are planning to implement Logstash with kibana for our application logs(log4 j - weblogic..). We have about 10 applications running in 10 instances, we would have about 100 logs files. Total size would be… --- ## [Update data from mysql to elasticsearch](https://discuss.elastic.co/t/update-data-from-mysql-to-elasticsearch/106817)
**Author:** [@Shekhar\_Pandey](https://discuss.elastic.co/u/Shekhar_Pandey)\ **Replies:** 4\ **Last updated:** [November 9, 2017, 6:51am UTC](https://discuss.elastic.co/t/update-data-from-mysql-to-elasticsearch/106817 "2017-11-09T06:51:54Z")
There is any way to fetch only update data from mysql to elasticsearch. I am using sql\_last\_value...but it will fetch only new row which comes in mysql.... Any way if I did change any previous index row in mysql...logst… --- ## [Timestamp to epoch in microseconds - am I doing this right?](https://discuss.elastic.co/t/timestamp-to-epoch-in-microseconds-am-i-doing-this-right/105838)
**Author:** [@Rubyan](https://discuss.elastic.co/u/Rubyan)\ **Replies:** 3\ **Last updated:** [November 9, 2017, 6:18am UTC](https://discuss.elastic.co/t/timestamp-to-epoch-in-microseconds-am-i-doing-this-right/105838 "2017-11-09T06:18:54Z")
I'm reading events from an API that contains a field 'Timestamp' with a timestamp in the following format. The milliseconds are not always present or occur in 1, 2 or 3 digit resolution: 2017-07-11T11:55:05Z 2017-07-11T… --- ## [Logstash port 5044 is not listening](https://discuss.elastic.co/t/logstash-port-5044-is-not-listening/106784)
**Author:** [@peterch](https://discuss.elastic.co/u/peterch)\ **Replies:** 3\ **Last updated:** [November 9, 2017, 3:20am UTC](https://discuss.elastic.co/t/logstash-port-5044-is-not-listening/106784 "2017-11-09T03:20:57Z")
Dear all, I would like to receive log from remote server through beats. However I find the port 5044 in ELK server is not listening. The ELK server is window platform. Any idea? Thanks. ELK server - logstash\\config.jso… --- ## [When to use codecs in ouput?](https://discuss.elastic.co/t/when-to-use-codecs-in-ouput/106956)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 0\ **Last updated:** [November 9, 2017, 2:49am UTC](https://discuss.elastic.co/t/when-to-use-codecs-in-ouput/106956 "2017-11-09T02:49:51Z")
Hi, When would you use a codec in your output? I have a remote logstash instance outputting data to a file with gzip compression. The compressed file will be sent to a storage server from which the main logstash is rea… --- ## [Logstash Filter - GeoIP2-ISP](https://discuss.elastic.co/t/logstash-filter-geoip2-isp/105446)
**Author:** [@rsmoorthy](https://discuss.elastic.co/u/rsmoorthy)\ **Replies:** 1\ **Last updated:** [November 9, 2017, 1:28am UTC](https://discuss.elastic.co/t/logstash-filter-geoip2-isp/105446 "2017-11-09T01:28:51Z")
Hello, Does the logstash-filter-geoip support GeoIP2-ISP database (commercial database from Maxmind)? Their documentation at https://github.com/logstash-plugins/logstash-filter-geoip/blob/master/docs/index.asciidoc sug… --- ## [\[netfow codec\] Can't (yet) decode flowset](https://discuss.elastic.co/t/netfow-codec-cant-yet-decode-flowset/102655)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 5\ **Last updated:** [November 8, 2017, 11:47pm UTC](https://discuss.elastic.co/t/netfow-codec-cant-yet-decode-flowset/102655 "2017-11-08T23:47:56Z")
Another day another something strange happening. Copied a VM running 5.4, upgraded to 5.6.1 =\> 5.6.2. No problem. Same VM, upgraded 5.4 directly to 5.6.2. Now netflow can't decode the flowset and no the problem is not … --- ## [Cannot install multiline filter on FreeBSD 11](https://discuss.elastic.co/t/cannot-install-multiline-filter-on-freebsd-11/106929)
**Author:** [@pdna](https://discuss.elastic.co/u/pdna)\ **Replies:** 2\ **Last updated:** [November 8, 2017, 9:42pm UTC](https://discuss.elastic.co/t/cannot-install-multiline-filter-on-freebsd-11/106929 "2017-11-08T21:42:43Z")
I installed ELK stack from the FreeBSD package manager. sudo /usr/local/logstash/bin/logstash-plugin install logstash-filter-multiline sudo: /usr/local/logstash/bin/logstash-plugin: command not found After making the f… --- ## [Syslog not being picked up even though it's tagged 'syslog'](https://discuss.elastic.co/t/syslog-not-being-picked-up-even-though-its-tagged-syslog/106847)
**Author:** [@sfchrisgleason](https://discuss.elastic.co/u/sfchrisgleason)\ **Replies:** 1\ **Last updated:** [November 8, 2017, 9:34pm UTC](https://discuss.elastic.co/t/syslog-not-being-picked-up-even-though-its-tagged-syslog/106847 "2017-11-08T21:34:17Z")
Good morning all. I am still pretty new to the elastic stack. I've used it on small scales and it worked great, but I'm getting ready to roll out a much larger aggregation model and wanted to be sure I'm understanding w… --- ## [Sincedb\_write\_interval not honoured during quiet period](https://discuss.elastic.co/t/sincedb-write-interval-not-honoured-during-quiet-period/106932)
**Author:** [@guido4f](https://discuss.elastic.co/u/guido4f)\ **Replies:** 0\ **Last updated:** [November 8, 2017, 8:54pm UTC](https://discuss.elastic.co/t/sincedb-write-interval-not-honoured-during-quiet-period/106932 "2017-11-08T20:54:56Z")
There is an issue with the updating of sincedb, if the files have received and completely order partially processed within a single sincedb\_write\_interval; and no subsequent records are processed the changes and read s… --- ## [ES Aggregation as Logstash Input](https://discuss.elastic.co/t/es-aggregation-as-logstash-input/106927)
**Author:** [@chetan\_vartak](https://discuss.elastic.co/u/chetan_vartak)\ **Replies:** 0\ **Last updated:** [November 8, 2017, 8:04pm UTC](https://discuss.elastic.co/t/es-aggregation-as-logstash-input/106927 "2017-11-08T20:04:41Z")
I found 2 topics on this topic with no clear solution. I eventually got it working. How to get ES aggregation data as Logstash input? Aggregation Query possible input ES plugin I basically used the curl to output ES a… --- ## [Another EPOCH time dateparse failure](https://discuss.elastic.co/t/another-epoch-time-dateparse-failure/106893)
**Author:** [@guruprasadh4](https://discuss.elastic.co/u/guruprasadh4)\ **Replies:** 1\ **Last updated:** [November 8, 2017, 6:48pm UTC](https://discuss.elastic.co/t/another-epoch-time-dateparse-failure/106893 "2017-11-08T18:48:35Z")
Sample Log file- IDENTIFIER TIMESTAMP T C RESOURCE\_NAME DESCRIPTION C6E26F3B 1002042417 I H hdisk111 BACK-UP PATH STATUS CHANGE C6E26F3B 1002042417 I H hdisk103 BACK-UP PATH STATUS CHANGE Logstash c… --- ## [Connection Refused to ElasticSearch](https://discuss.elastic.co/t/connection-refused-to-elasticsearch/106601)
**Author:** [@ayayalar](https://discuss.elastic.co/u/ayayalar)\ **Replies:** 7\ **Last updated:** [November 8, 2017, 6:48pm UTC](https://discuss.elastic.co/t/connection-refused-to-elasticsearch/106601 "2017-11-08T18:48:31Z")
I am running Elk stack on docker. All services starts perfectly fine. However I am getting connection refused in logstash logs. Here is my conf file for logstash: input { file { path =\> "/tmp/\*\_log" start\_pos… --- ## [Logstash group authorization error in Kafka Input](https://discuss.elastic.co/t/logstash-group-authorization-error-in-kafka-input/106669)
**Author:** [@Suresh\_Nataraj](https://discuss.elastic.co/u/Suresh_Nataraj)\ **Replies:** 4\ **Last updated:** [November 8, 2017, 6:14pm UTC](https://discuss.elastic.co/t/logstash-group-authorization-error-in-kafka-input/106669 "2017-11-08T18:14:50Z")
Hi All, I am getting the below error while trying to connect to Kafka from Logstash . Exception in thread "Ruby-0-Thread-20: /Users/sn2/Desktop/logstash-5.6.1/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/li… --- ## [Logstash hangs after changing the authentication credentials in Pipeline out of Kibana](https://discuss.elastic.co/t/logstash-hangs-after-changing-the-authentication-credentials-in-pipeline-out-of-kibana/106904)
**Author:** [@rolyn](https://discuss.elastic.co/u/rolyn)\ **Replies:** 0\ **Last updated:** [November 8, 2017, 4:18pm UTC](https://discuss.elastic.co/t/logstash-hangs-after-changing-the-authentication-credentials-in-pipeline-out-of-kibana/106904 "2017-11-08T16:18:30Z")
We just experienced an issue in Logstash after a change of the Pipeline out of Kibana in Version 6.0rc-2. We use full X-Pack Security options and therefore had TLS encryption enabled. In Kibana Pipeline, we unintention… --- ## [Logstash filter to drop data](https://discuss.elastic.co/t/logstash-filter-to-drop-data/106060)
**Author:** [@larryf](https://discuss.elastic.co/u/larryf)\ **Replies:** 20\ **Last updated:** [November 8, 2017, 4:12pm UTC](https://discuss.elastic.co/t/logstash-filter-to-drop-data/106060 "2017-11-08T16:12:09Z")
good morning...new user to be gentle if I don't quite say thing correctly :slight\_smile: My logstash is V 2.2.2, running on CentOS 7.3. Part of the data logstash handles is from a Progress RDBMS database log file. In … --- ## [Logstash if-condition doesn't match](https://discuss.elastic.co/t/logstash-if-condition-doesnt-match/106892)
**Author:** [@Mugen](https://discuss.elastic.co/u/Mugen)\ **Replies:** 0\ **Last updated:** [November 8, 2017, 3:14pm UTC](https://discuss.elastic.co/t/logstash-if-condition-doesnt-match/106892 "2017-11-08T15:14:07Z")
I've setup a filter in my logstash configuration and it doesn't match everytime the field has the correct value. I am using Logstash 5.6.3 ,Elasticsearch 5.6.3 and Kibana 5.6.3 Filter: filter { if \[program\] == "RT\_… --- ## [EPOC time issue](https://discuss.elastic.co/t/epoc-time-issue/106715)
**Author:** [@guruprasadh4](https://discuss.elastic.co/u/guruprasadh4)\ **Replies:** 4\ **Last updated:** [November 8, 2017, 3:06pm UTC](https://discuss.elastic.co/t/epoc-time-issue/106715 "2017-11-08T15:06:59Z")
Sample Data -- (Data is in CSV file) and provided samples. 25200,"1171007000040000","nzapap176:KUX","1171007000040000",40960,24618,16342,+000000000000000000000000000060.2,+000000000000000000000000000039.8,16384,52,163… --- ## [IF condition with XML filter](https://discuss.elastic.co/t/if-condition-with-xml-filter/106877)
**Author:** [@rm1890](https://discuss.elastic.co/u/rm1890)\ **Replies:** 4\ **Last updated:** [November 8, 2017, 2:55pm UTC](https://discuss.elastic.co/t/if-condition-with-xml-filter/106877 "2017-11-08T14:55:15Z")
hi all, I'm trying to apply an XML filter based on the value of one of the fields. So, the config for that part looks like this: filter{ if \[my\_field\] == "x.zhh"{ xml{ … --- ## [Logstash retry with exponential backoff](https://discuss.elastic.co/t/logstash-retry-with-exponential-backoff/106744)
**Author:** [@mathurin68](https://discuss.elastic.co/u/mathurin68)\ **Replies:** 1\ **Last updated:** [November 8, 2017, 1:45pm UTC](https://discuss.elastic.co/t/logstash-retry-with-exponential-backoff/106744 "2017-11-08T13:45:22Z")
OK I installed x-pack and finally got all the credentials in the correct places(including winlogbeats on WEC) but now getting new error in logstash logs - \[2017-11-07T14:29:34,577\]\[ERROR\]\[logstash.outputs.elasticsearch… --- ## [Ruby Map Counter not working fine](https://discuss.elastic.co/t/ruby-map-counter-not-working-fine/106801)
**Author:** [@MariumHassan](https://discuss.elastic.co/u/MariumHassan)\ **Replies:** 1\ **Last updated:** [November 8, 2017, 12:06pm UTC](https://discuss.elastic.co/t/ruby-map-counter-not-working-fine/106801 "2017-11-08T12:06:10Z")
Hi all, I have the following piece of code where I am just using an orderCounter to count the orders but I end up getting wrong count in my FILE\_SAVED logs. I have checked my log file. There isn't any issue in it, and a… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=877) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=879)