# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=704 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 705 --- ## [Logstash can't read some files](https://discuss.elastic.co/t/logstash-cant-read-some-files/143847)
**Author:** [@alicia1](https://discuss.elastic.co/u/alicia1)\ **Replies:** 1\ **Last updated:** [August 10, 2018, 12:49pm UTC](https://discuss.elastic.co/t/logstash-cant-read-some-files/143847 "2018-08-10T12:49:11Z")
Hello, We have 3 files log csv in a repository (windows), but 2 of them can't be read by logstash (6.2.4), the msg logstash is like this: \[WARN \]\[filewatch.tailmode.processor\] \>\>\> Rotation In Progress - inode change d… --- ## [Adding extra data](https://discuss.elastic.co/t/adding-extra-data/143854)
**Author:** [@Olha1](https://discuss.elastic.co/u/Olha1)\ **Replies:** 7\ **Last updated:** [August 10, 2018, 12:48pm UTC](https://discuss.elastic.co/t/adding-extra-data/143854 "2018-08-10T12:48:46Z")
Help me! Why my logstash with this config file adding the same data repeatedly input { jdbc { jdbc\_connection\_string =\> "connection\_string" jdbc\_user =\> "user" jdbc\_password =\> "password" jdbc\_driver\_library =\> "dr… --- ## [Logstash: Output fails](https://discuss.elastic.co/t/logstash-output-fails/143818)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 7\ **Last updated:** [August 10, 2018, 11:26am UTC](https://discuss.elastic.co/t/logstash-output-fails/143818 "2018-08-10T11:26:41Z")
I have two output, kafka and elasticsearch. I want to have a fail safe with my setup. If one of them stops, the other one can still index a data. I tried to stop my elasticsearch while the kafka is still running and I … --- ## [Exctracting JSON format of data from the Input logs containing both JSON and Plain Text](https://discuss.elastic.co/t/exctracting-json-format-of-data-from-the-input-logs-containing-both-json-and-plain-text/143806)
**Author:** [@Anshika](https://discuss.elastic.co/u/Anshika)\ **Replies:** 3\ **Last updated:** [August 10, 2018, 11:06am UTC](https://discuss.elastic.co/t/exctracting-json-format-of-data-from-the-input-logs-containing-both-json-and-plain-text/143806 "2018-08-10T11:06:52Z")
Hi, I am trying to send the JSON format of a log from the logs containing both JSON and non-JSON formats to a Kafka server.I have tried the below and very close to the solution what I need. My logstash config looks lik… --- ## [Unable to send winlogbeat data to logstash](https://discuss.elastic.co/t/unable-to-send-winlogbeat-data-to-logstash/143404)
**Author:** [@ravikt](https://discuss.elastic.co/u/ravikt)\ **Replies:** 6\ **Last updated:** [August 10, 2018, 10:05am UTC](https://discuss.elastic.co/t/unable-to-send-winlogbeat-data-to-logstash/143404 "2018-08-10T10:05:10Z")
I have Both metricbeat and winlogbeat agents installed on same AWS EC2 windows instance. I am trying to push the data to logstash. Logstash is just processing metricbeat data but not for winlogbeat. I am using metricb… --- ## [Custom grok\_pattern on syslog input](https://discuss.elastic.co/t/custom-grok-pattern-on-syslog-input/143669)
**Author:** [@A\_B](https://discuss.elastic.co/u/A_B)\ **Replies:** 3\ **Last updated:** [August 10, 2018, 8:23am UTC](https://discuss.elastic.co/t/custom-grok-pattern-on-syslog-input/143669 "2018-08-10T08:23:54Z")
Hi, I am trying to ingest logs from our Juniper switshes and I'm trying to configure a syslog input. Manyally editing the Logstash input config I got this to work. # Juniper syslog input syslog { port =\> 5518 … --- ## [Duplicated data in kafka using logstash](https://discuss.elastic.co/t/duplicated-data-in-kafka-using-logstash/143677)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 1\ **Last updated:** [August 10, 2018, 6:35am UTC](https://discuss.elastic.co/t/duplicated-data-in-kafka-using-logstash/143677 "2018-08-10T06:35:57Z")
I am using logstash pipeline data from elasticsearch to kafka. I have an index(asd\_logstash-2018.07.15) with around 1000 docs, but upon checking in kafka it went around 22k messages and continuously increasing. I came… --- ## [Exclude keys](https://discuss.elastic.co/t/exclude-keys/142352)
**Author:** [@Arun\_Murugappan](https://discuss.elastic.co/u/Arun_Murugappan)\ **Replies:** 4\ **Last updated:** [August 10, 2018, 4:32am UTC](https://discuss.elastic.co/t/exclude-keys/142352 "2018-08-10T04:32:35Z")
Input file:kvoptions.log {"Name":"admin","ConfigurationObjectType":"ExecutionGroup"} Configuration file: input { file { path =\> "/home/murugar/Documents/kvoptions.log" type =\> "file" start\_position =\> "beginning… --- ## [How to use dynamic string in outout](https://discuss.elastic.co/t/how-to-use-dynamic-string-in-outout/143661)
**Author:** [@Uncel\_wang\_shushu](https://discuss.elastic.co/u/Uncel_wang_shushu)\ **Replies:** 4\ **Last updated:** [August 10, 2018, 4:06am UTC](https://discuss.elastic.co/t/how-to-use-dynamic-string-in-outout/143661 "2018-08-10T04:06:07Z")
I want to use a dynamic string to replace the result as follow : { "DISMAN-EXPRESSION-MIB::sysUpTimeInstance" =\> "71 days, 04:40:14.91", "@timestamp" =\> 2018-08-09T09:02:37.423Z, "type" =\> "snmptrap", "host" =\> "10… --- ## [Use Logstash-input-snmp，cannot receive DELL TEST TRAP](https://discuss.elastic.co/t/use-logstash-input-snmp-cannot-receive-dell-test-trap/143656)
**Author:** [@Uncel\_wang\_shushu](https://discuss.elastic.co/u/Uncel_wang_shushu)\ **Replies:** 3\ **Last updated:** [August 10, 2018, 2:37am UTC](https://discuss.elastic.co/t/use-logstash-input-snmp-cannot-receive-dell-test-trap/143656 "2018-08-10T02:37:10Z")
Error handling trap: Expected snmpTrapOID.0, found \[name=DELL-RAC-MIB::server3.5.3.1.1.0, value=TST001 (OCTET STRING)\] /data/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/snmp-1.2.0/lib/snmp/pdu.rb:293:in trap\_oid' /dat… --- ## [S3 input plugin errors](https://discuss.elastic.co/t/s3-input-plugin-errors/143779)
**Author:** [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\ **Replies:** 0\ **Last updated:** [August 9, 2018, 11:42pm UTC](https://discuss.elastic.co/t/s3-input-plugin-errors/143779 "2018-08-09T23:42:55Z")
Hello there, I started a new s3 input plugin on logstash and noticed the following error in the logstash.log: ========================== \[2018-08-09T19:46:26,790\]\[INFO \]\[logstash.agent \] Successfully started… --- ## [Logstash unable to map field to es](https://discuss.elastic.co/t/logstash-unable-to-map-field-to-es/143599)
**Author:** [@pooch](https://discuss.elastic.co/u/pooch)\ **Replies:** 5\ **Last updated:** [August 9, 2018, 9:10pm UTC](https://discuss.elastic.co/t/logstash-unable-to-map-field-to-es/143599 "2018-08-09T21:10:19Z")
I am using a multiline config in filebeat to send logs as one line, not sure if that is the cause of this. I am currently getting this error when indexing certain log files. response=\>{"index"=\>{"\_index"=\>"filebeat-201… --- ## [Not able to parse large XML nmap scan with logstash-nmap-codec](https://discuss.elastic.co/t/not-able-to-parse-large-xml-nmap-scan-with-logstash-nmap-codec/143722)
**Author:** [@rkokkelk](https://discuss.elastic.co/u/rkokkelk)\ **Replies:** 8\ **Last updated:** [August 9, 2018, 7:45pm UTC](https://discuss.elastic.co/t/not-able-to-parse-large-xml-nmap-scan-with-logstash-nmap-codec/143722 "2018-08-09T19:45:36Z")
Hi, I've got some serieus issues with parsing relativly large nmap scan using the logstash-nmap-codec. In my situation I've got a ~1.8MB XML file with the resutls of the nmap scan which I want to import but I've got iss… --- ## [Logstash YAML Error](https://discuss.elastic.co/t/logstash-yaml-error/143693)
**Author:** [@bigdamhero](https://discuss.elastic.co/u/bigdamhero)\ **Replies:** 2\ **Last updated:** [August 9, 2018, 7:07pm UTC](https://discuss.elastic.co/t/logstash-yaml-error/143693 "2018-08-09T19:07:02Z")
Yet Another Newby Question I am not sure what file to look in I have looked over logstash.yml several times and do not see were the error may be. I might need a better guide to yml files. Here is the error: Aug 09 0… --- ## [Ruby exception occurred: undefined method \`\[\]' for nil:NilClas](https://discuss.elastic.co/t/ruby-exception-occurred-undefined-method-for-nil-nilclas/143750)
**Author:** [@MikaelLindstrom](https://discuss.elastic.co/u/MikaelLindstrom)\ **Replies:** 1\ **Last updated:** [August 9, 2018, 7:01pm UTC](https://discuss.elastic.co/t/ruby-exception-occurred-undefined-method-for-nil-nilclas/143750 "2018-08-09T19:01:08Z")
Hi all, We're seeing these two messages in the logs: /var/log/logstash/logstash-plain.log: \[2018-08-09T05:13:41,528\]\[ERROR\]\[logstash.filters.ruby \] Ruby exception occurred: undefined method \`\[\]' for nil:NilClass W… --- ## [Logstash CISCO ASA logs not parsing](https://discuss.elastic.co/t/logstash-cisco-asa-logs-not-parsing/143736)
**Author:** [@Samvid\_Kulkarni](https://discuss.elastic.co/u/Samvid_Kulkarni)\ **Replies:** 4\ **Last updated:** [August 9, 2018, 6:31pm UTC](https://discuss.elastic.co/t/logstash-cisco-asa-logs-not-parsing/143736 "2018-08-09T18:31:32Z")
I have log file which has entries like this 2018-07-30 09:50:30||20||6||172.16.1.4||%ASA-6-302014: Teardown TCP connection 597505728 for outside:40.97.142.8/443 to inside:192.168.5.120/50041 duration 0:00:00 bytes 0 TCP… --- ## [Performance issues while importing CSV files into Elasticsearch](https://discuss.elastic.co/t/performance-issues-while-importing-csv-files-into-elasticsearch/143723)
**Author:** [@jnci](https://discuss.elastic.co/u/jnci)\ **Replies:** 1\ **Last updated:** [August 9, 2018, 5:28pm UTC](https://discuss.elastic.co/t/performance-issues-while-importing-csv-files-into-elasticsearch/143723 "2018-08-09T17:28:21Z")
I'm trying to import some gigabytes of CSV files (approximately 100+ million rows with 5 columns) but the throughput is very low (~1mb/s). I'm not quite sure yet what the issue may be, but maybe someone here has some lea… --- ## [Is it possible to aggregate logs from cef input](https://discuss.elastic.co/t/is-it-possible-to-aggregate-logs-from-cef-input/139976)
**Author:** [@saad](https://discuss.elastic.co/u/saad)\ **Replies:** 23\ **Last updated:** [August 9, 2018, 4:39pm UTC](https://discuss.elastic.co/t/is-it-possible-to-aggregate-logs-from-cef-input/139976 "2018-08-09T16:39:41Z")
Hi, Is it possible to apply aggregate filter on cef logs. Would you please check my conf file if there is something wrong. it should aggregating when name and sourceAddress are common and increase field aggregatedEvent… --- ## [Setting the correct timezone in date filter plugin](https://discuss.elastic.co/t/setting-the-correct-timezone-in-date-filter-plugin/143457)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 3\ **Last updated:** [August 8, 2018, 2:12pm UTC](https://discuss.elastic.co/t/setting-the-correct-timezone-in-date-filter-plugin/143457 "2018-08-08T14:12:57Z")
Hello, I'm using a date filter plugin to match a date field, which is in the format yyyy-MM-dd (so the field does not provide the time, for example 2018-07-03). My desire would be to obtain Logstash parsing the date @00:… --- ## [How to split linux and windows data in two different indexes](https://discuss.elastic.co/t/how-to-split-linux-and-windows-data-in-two-different-indexes/143697)
**Author:** [@gabberoid](https://discuss.elastic.co/u/gabberoid)\ **Replies:** 9\ **Last updated:** [August 9, 2018, 2:49pm UTC](https://discuss.elastic.co/t/how-to-split-linux-and-windows-data-in-two-different-indexes/143697 "2018-08-09T14:49:50Z")
Hi. I created two conf file in /etc/logstash/conf.d for two hosts: Linux hosts with filebeat and windows hosts with winlogbeat. In output part I wrote two dofferent indexes in pattern: "linux-%{+YYYY.MM.dd}" and "windo… --- ## [How to get host from websocket input plugin in logstash](https://discuss.elastic.co/t/how-to-get-host-from-websocket-input-plugin-in-logstash/143720)
**Author:** [@sreeja](https://discuss.elastic.co/u/sreeja)\ **Replies:** 0\ **Last updated:** [August 9, 2018, 2:29pm UTC](https://discuss.elastic.co/t/how-to-get-host-from-websocket-input-plugin-in-logstash/143720 "2018-08-09T14:29:58Z")
Could you please let me know is it possible to get the host from websocket input plugin. Right now my output has @timestamp, @version and type. I need to use multiple socket connection and it would be good to have a host… --- ## [Multiple Filebeat prospectors, one filebeat instance, multiple logstash .conf files](https://discuss.elastic.co/t/multiple-filebeat-prospectors-one-filebeat-instance-multiple-logstash-conf-files/143714)
**Author:** [@Georgios\_Gkinis](https://discuss.elastic.co/u/Georgios_Gkinis)\ **Replies:** 1\ **Last updated:** [August 9, 2018, 2:21pm UTC](https://discuss.elastic.co/t/multiple-filebeat-prospectors-one-filebeat-instance-multiple-logstash-conf-files/143714 "2018-08-09T14:21:12Z")
Hello Everyone, I have been using filebeat in conjunction with Logstash for some time. Up untill recently i had no issue with using multiple input { beats { port=5044 } } statements in separate logstash config fi… --- ## [Filter event from Winlogbeat](https://discuss.elastic.co/t/filter-event-from-winlogbeat/143684)
**Author:** [@gabberoid](https://discuss.elastic.co/u/gabberoid)\ **Replies:** 5\ **Last updated:** [August 9, 2018, 2:11pm UTC](https://discuss.elastic.co/t/filter-event-from-winlogbeat/143684 "2018-08-09T14:11:37Z")
Hi community. Somebody has any filter example for conf.d in order to filter Eventlog of Windows (Security)? I'm using Winlogbeat and didnt find any usable example in documentation. Need your help! --- ## [How to access json objects](https://discuss.elastic.co/t/how-to-access-json-objects/143703)
**Author:** [@Gegenki](https://discuss.elastic.co/u/Gegenki)\ **Replies:** 8\ **Last updated:** [August 9, 2018, 1:53pm UTC](https://discuss.elastic.co/t/how-to-access-json-objects/143703 "2018-08-09T13:53:32Z")
I cannot figure out how to access an array or flatten it. I get input from an SQL database. I use jdbc\_streaming to get more data jdbc\_streaming { jdbc\_connection\_string =\> "jdbc:postgresql://localhost:5432/db" sta… --- ## [Unix socket input with stdout](https://discuss.elastic.co/t/unix-socket-input-with-stdout/143525)
**Author:** [@Lubomir\_Bielik](https://discuss.elastic.co/u/Lubomir_Bielik)\ **Replies:** 3\ **Last updated:** [August 9, 2018, 1:42pm UTC](https://discuss.elastic.co/t/unix-socket-input-with-stdout/143525 "2018-08-09T13:42:44Z")
Hello, I am new to Logstash, but I would like to use its capability of unix socket input, which comes very handy. I have a program sending data to unix socket, i have that socket tested and i am 100% sure, that data ar… --- ## [Date filter failure](https://discuss.elastic.co/t/date-filter-failure/143706)
**Author:** [@JPelastic](https://discuss.elastic.co/u/JPelastic)\ **Replies:** 1\ **Last updated:** [August 9, 2018, 1:27pm UTC](https://discuss.elastic.co/t/date-filter-failure/143706 "2018-08-09T13:27:13Z")
I am trying to parse "datum" into "@timestamp". This is the output { "tags" =\> \[ \[0\] "\_dateparsefailure" \], "@version" =\> "1", "datum" =\> 2018-03-01T01:24:31.037Z, "@timestamp" =\>… --- ## [Mutate filter replace - please clarify the documentation](https://discuss.elastic.co/t/mutate-filter-replace-please-clarify-the-documentation/143700)
**Author:** [@JPelastic](https://discuss.elastic.co/u/JPelastic)\ **Replies:** 2\ **Last updated:** [August 9, 2018, 1:18pm UTC](https://discuss.elastic.co/t/mutate-filter-replace-please-clarify-the-documentation/143700 "2018-08-09T13:18:19Z")
This is the info given in the docs: Replace a field with a new value. The new value can include %{foo} strings to help you build a new value from other parts of the event. This first sentence is highly ambiguous. D… --- ## [Logstash collecting logs from multiple servers](https://discuss.elastic.co/t/logstash-collecting-logs-from-multiple-servers/143682)
**Author:** [@Priyaranjan\_Mudliar](https://discuss.elastic.co/u/Priyaranjan_Mudliar)\ **Replies:** 3\ **Last updated:** [August 9, 2018, 1:09pm UTC](https://discuss.elastic.co/t/logstash-collecting-logs-from-multiple-servers/143682 "2018-08-09T13:09:18Z")
Is there a way so that i can use logstash to input from multiple servers without me having to install filebeat on multiple servers? Please Help ... Any leads would be appreciated. :slightly\_smiling\_face: PS:- I am worki… --- ## [Forward log to siem](https://discuss.elastic.co/t/forward-log-to-siem/143369)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 3\ **Last updated:** [August 9, 2018, 12:46pm UTC](https://discuss.elastic.co/t/forward-log-to-siem/143369 "2018-08-09T12:46:23Z")
I would like to forward syslog from many device cisco to a siem using logstash. I tried to use syslog plugin but logstash insert a message into log and don't forward only the syslog message. this is the part of logstas… --- ## [Plugin test failing](https://discuss.elastic.co/t/plugin-test-failing/143612)
**Author:** [@mandi](https://discuss.elastic.co/u/mandi)\ **Replies:** 2\ **Last updated:** [August 9, 2018, 12:21pm UTC](https://discuss.elastic.co/t/plugin-test-failing/143612 "2018-08-09T12:21:19Z")
This might be a stupid question, but I can't seem to figure it out. I'm trying to write a filter plugin for logstash and started by generating the default "Hello, world!" plugin and going through the basic development p… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=703) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=705)