# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=673 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 674 --- ## [Load csv file in logstash](https://discuss.elastic.co/t/load-csv-file-in-logstash/151319)
**Author:** [@balumurari1](https://discuss.elastic.co/u/balumurari1)\ **Replies:** 1\ **Last updated:** [October 8, 2018, 12:50pm UTC](https://discuss.elastic.co/t/load-csv-file-in-logstash/151319 "2018-10-08T12:50:17Z")
Please provide sample examples to upload csv, pdf files in logstash --- ## [Logstash x ActiveMQ](https://discuss.elastic.co/t/logstash-x-activemq/151455)
**Author:** [@denisgalvao](https://discuss.elastic.co/u/denisgalvao)\ **Replies:** 0\ **Last updated:** [October 8, 2018, 1:05pm UTC](https://discuss.elastic.co/t/logstash-x-activemq/151455 "2018-10-08T13:05:34Z")
Hello. Im trying to figure out how to set up logstash as an input for ActiveMQ. The documentation about it is not clear at all. Do I need to configure the MQ on logstash conf files(conf.d on Debian) only? Is there any o… --- ## [Match different logs with minimal grok](https://discuss.elastic.co/t/match-different-logs-with-minimal-grok/151395)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 0\ **Last updated:** [October 8, 2018, 7:43am UTC](https://discuss.elastic.co/t/match-different-logs-with-minimal-grok/151395 "2018-10-08T07:43:17Z")
Hi, Is there a way to match certain parts of a log without having to write a grok regex that matches the whole log? If so, an example would be appreciated :slight\_smile: EDIT: Solved with break\_on\_match =\> false --- ## [Reparsing same files](https://discuss.elastic.co/t/reparsing-same-files/151410)
**Author:** [@richagautam](https://discuss.elastic.co/u/richagautam)\ **Replies:** 1\ **Last updated:** [October 8, 2018, 12:21pm UTC](https://discuss.elastic.co/t/reparsing-same-files/151410 "2018-10-08T12:21:41Z")
Hi Team, I have one doubt: Does reparsing of same log files having with same index name will save new records in elasticsearch every time we execute the logstash pipeline ? If yes, then will these entry will reflect i… --- ## [Logstash is not breaking the logs](https://discuss.elastic.co/t/logstash-is-not-breaking-the-logs/151341)
**Author:** [@shruti51](https://discuss.elastic.co/u/shruti51)\ **Replies:** 2\ **Last updated:** [October 8, 2018, 11:50am UTC](https://discuss.elastic.co/t/logstash-is-not-breaking-the-logs/151341 "2018-10-08T11:50:57Z")
Hi, Logstash is not breaking the logs It is throwing the error: \[2018-10-07T13:57:46,327\]\[WARN \]\[org.logstash.dissect.Dissector\] Dissector mapping, pattern not found {"field"=\>"message ", "pattern"=\>"%{Timestamp} ##%{… --- ## [Grok Pattern for logstash](https://discuss.elastic.co/t/grok-pattern-for-logstash/149783)
**Author:** [@muthu\_kumar1](https://discuss.elastic.co/u/muthu_kumar1)\ **Replies:** 13\ **Last updated:** [October 8, 2018, 10:32am UTC](https://discuss.elastic.co/t/grok-pattern-for-logstash/149783 "2018-10-08T10:32:28Z")
Hi , Can someone help on grok pattern for the below logs message#IE-7.0||trident/4.0#message tried with below pattern %{WORD:THREADNAME}#%{HOSTNAME:IE}||trident/%{JAVACLASS}#{%WORD:MESSAGE} but not worked..please he… --- ## [Elasticsearch Filter 404 not found](https://discuss.elastic.co/t/elasticsearch-filter-404-not-found/151423)
**Author:** [@Pierre2](https://discuss.elastic.co/u/Pierre2)\ **Replies:** 0\ **Last updated:** [October 8, 2018, 9:49am UTC](https://discuss.elastic.co/t/elasticsearch-filter-404-not-found/151423 "2018-10-08T09:49:30Z")
Hello, I try to use the Elasticsearch filter Elasticsearch filter plugin | Logstash Reference \[6.3\] | Elastic There is the situation. I have two indexes : "svi" and "cdr\_sbc" The is an example of one document in the "… --- ## [Logstash taking more time to start](https://discuss.elastic.co/t/logstash-taking-more-time-to-start/151409)
**Author:** [@ramanna\_hk](https://discuss.elastic.co/u/ramanna_hk)\ **Replies:** 0\ **Last updated:** [October 8, 2018, 9:08am UTC](https://discuss.elastic.co/t/logstash-taking-more-time-to-start/151409 "2018-10-08T09:08:01Z")
Hello guys, I am using logstash 6.4.1 and logstash-output-kafka plugin 7.2.0 to publish data to kafka 1.0.0 OS: CentOS Linux release 7.5.1804 (Core) RAM : available RAM is 6 GB Currently i have only one .conf file. … --- ## [Correlation of IPs within ELK](https://discuss.elastic.co/t/correlation-of-ips-within-elk/151189)
**Author:** [@Alessandro\_89](https://discuss.elastic.co/u/Alessandro_89)\ **Replies:** 6\ **Last updated:** [October 8, 2018, 8:38am UTC](https://discuss.elastic.co/t/correlation-of-ips-within-elk/151189 "2018-10-08T08:38:35Z")
Greetings, I am trying to automate/ease a procedure to review firewall rules within ELK (ElasticSearch, Logstash, Kibana). I have some data obtained from a CSV, which is structured like this: Source;Destination;Service… --- ## [6.4.1 invalid\_index\_name\_exception](https://discuss.elastic.co/t/6-4-1-invalid-index-name-exception/151205)
**Author:** [@sles](https://discuss.elastic.co/u/sles)\ **Replies:** 2\ **Last updated:** [October 8, 2018, 8:10am UTC](https://discuss.elastic.co/t/6-4-1-invalid-index-name-exception/151205 "2018-10-08T08:10:47Z")
Hello! I'm newbie, just installed logstash 6.4.1 and see errors in log like: \[2018-10-05T11:45:49,445\]\[ERROR\]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:… --- ## [SNMP Bind Failed](https://discuss.elastic.co/t/snmp-bind-failed/151014)
**Author:** [@paul1243](https://discuss.elastic.co/u/paul1243)\ **Replies:** 11\ **Last updated:** [October 8, 2018, 8:04am UTC](https://discuss.elastic.co/t/snmp-bind-failed/151014 "2018-10-08T08:04:02Z")
When I run the SNMP input filter, I get the SNMP trap listener died; can not assign requested address . Here is my config: input { snmptrap { host =\> '\*\*.\*\*.\*\*.\*\*' port =\> 161 community =\> "public" yamlmi… --- ## [Mapping json in http plugin](https://discuss.elastic.co/t/mapping-json-in-http-plugin/151384)
**Author:** [@robindeshwal](https://discuss.elastic.co/u/robindeshwal)\ **Replies:** 0\ **Last updated:** [October 8, 2018, 4:55am UTC](https://discuss.elastic.co/t/mapping-json-in-http-plugin/151384 "2018-10-08T04:55:36Z")
Hi, Ii am using http output plugin to send data to my api's. i am using mapping for map data and post data, here: mapping =\> {"foo" =\> "%{host}" "bar" =\> "%{type}"} But i want to send below type of data: mapping =\> … --- ## [BadResponseCodeError error=\>"Got response code '401'](https://discuss.elastic.co/t/badresponsecodeerror-error-got-response-code-401/151342)
**Author:** [@af615dbd55cac2cacf32](https://discuss.elastic.co/u/af615dbd55cac2cacf32)\ **Replies:** 0\ **Last updated:** [October 7, 2018, 9:42am UTC](https://discuss.elastic.co/t/badresponsecodeerror-error-got-response-code-401/151342 "2018-10-07T09:42:16Z")
hello, I posted a topic about Kibana disabled to login after x-pack security turned on. i resolved that issue and the topic url is below. after that, elasticsearch and kibana look fine, but below errors were started o… --- ## [CEF plugin losing events](https://discuss.elastic.co/t/cef-plugin-losing-events/151373)
**Author:** [@hybrid](https://discuss.elastic.co/u/hybrid)\ **Replies:** 1\ **Last updated:** [October 8, 2018, 1:11am UTC](https://discuss.elastic.co/t/cef-plugin-losing-events/151373 "2018-10-08T01:11:01Z")
We're using rsyslog to round-robin TCP syslog events to multiple logstash syslog listeners with the CEF plugin. Using tcpdump on a logstash host, I find a random eventid and search for it in elastic. Sometimes it's ther… --- ## [XML filter not parsing](https://discuss.elastic.co/t/xml-filter-not-parsing/151078)
**Author:** [@tropas](https://discuss.elastic.co/u/tropas)\ **Replies:** 5\ **Last updated:** [October 7, 2018, 9:07am UTC](https://discuss.elastic.co/t/xml-filter-not-parsing/151078 "2018-10-07T09:07:56Z")
I'm using packetbeat to sniff network traffic (only http), which I'm then trying to parse (as it's all XML) with logstash. my logstash file looks like: filter { if \[method\] == "PUT" { xml { source =\> "%{\[http\]\[reque… --- ## [Cannot access logstash environmental variables](https://discuss.elastic.co/t/cannot-access-logstash-environmental-variables/151294)
**Author:** [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Replies:** 2\ **Last updated:** [October 7, 2018, 5:32am UTC](https://discuss.elastic.co/t/cannot-access-logstash-environmental-variables/151294 "2018-10-07T05:32:37Z")
I installed logstash via Docker and I keep getting the error when I am trying to access an environmental variable locally on the machine. I've tried adding it in /etc/environment and /etc/default/logstash both getting th… --- ## [Backslash in the if statement - LogStash](https://discuss.elastic.co/t/backslash-in-the-if-statement-logstash/151292)
**Author:** [@VietCong](https://discuss.elastic.co/u/VietCong)\ **Replies:** 3\ **Last updated:** [October 6, 2018, 9:44am UTC](https://discuss.elastic.co/t/backslash-in-the-if-statement-logstash/151292 "2018-10-06T09:44:50Z")
Hi, I would like to know how to write an if statement with the condition of a (backslash) being in the string like below: if ( \\backslash\\ in \[field1\] ) { Do\_something {} } I have tried: if ( "\\" in \[field1\] ) {} i… --- ## [Flatten/convert Nested Array Object to string](https://discuss.elastic.co/t/flatten-convert-nested-array-object-to-string/151301)
**Author:** [@VietCong](https://discuss.elastic.co/u/VietCong)\ **Replies:** 0\ **Last updated:** [October 6, 2018, 8:23am UTC](https://discuss.elastic.co/t/flatten-convert-nested-array-object-to-string/151301 "2018-10-06T08:23:04Z")
I am struggling with formatting nested array object fields where I have data structure like below: "Target": \[ { "ID": "Unknown", "Type": 0 }, { "ID": "known", "Type": 1 }, { "ID": "Maybe",… --- ## [Custome Logstash Filter Plugin Parameter Problem](https://discuss.elastic.co/t/custome-logstash-filter-plugin-parameter-problem/151166)
**Author:** [@Nww\_Pot\_Fung\_Nng](https://discuss.elastic.co/u/Nww_Pot_Fung_Nng)\ **Replies:** 5\ **Last updated:** [October 6, 2018, 1:28am UTC](https://discuss.elastic.co/t/custome-logstash-filter-plugin-parameter-problem/151166 "2018-10-06T01:28:48Z")
I am writing a logstash filter plugin. It takes a payload and generates a jwt token, and looks like this. gentoken { payload =\> { "userid" =\> '%{userid}' "role" =\> '%{role}' } secret =\> "305356… --- ## [A single backslash becomes double backslash](https://discuss.elastic.co/t/a-single-backslash-becomes-double-backslash/151202)
**Author:** [@Nww\_Pot\_Fung\_Nng](https://discuss.elastic.co/u/Nww_Pot_Fung_Nng)\ **Replies:** 4\ **Last updated:** [October 6, 2018, 1:24am UTC](https://discuss.elastic.co/t/a-single-backslash-becomes-double-backslash/151202 "2018-10-06T01:24:31Z")
I have messages which contain computer name under Windows domain. The format will be domain\\computername. However, the message ends up domain\\\\computername in elasticsearch indice. When I do a search, I need to put domai… --- ## [Shibboleth Auth Logs \_grokparsefailure](https://discuss.elastic.co/t/shibboleth-auth-logs-grokparsefailure/151280)
**Author:** [@Dr\_Natas](https://discuss.elastic.co/u/Dr_Natas)\ **Replies:** 0\ **Last updated:** [October 5, 2018, 8:32pm UTC](https://discuss.elastic.co/t/shibboleth-auth-logs-grokparsefailure/151280 "2018-10-05T20:32:00Z")
How filebeat sends my shibboleth logs. /etc/filebeat/filebeat.yml Note: Code is truncated. #--------------------------- Prospectors Path --------------------------------- #filebeat.config.inputs: # enabled: true # p… --- ## [Errors with journald plugin](https://discuss.elastic.co/t/errors-with-journald-plugin/151286)
**Author:** [@opoplawski](https://discuss.elastic.co/u/opoplawski)\ **Replies:** 0\ **Last updated:** [October 5, 2018, 10:35pm UTC](https://discuss.elastic.co/t/errors-with-journald-plugin/151286 "2018-10-05T22:35:25Z")
Trying to use the journald plugin with logstash/elastic 6.4.2 and I'm getting: \[2018-10-05T14:14:17,662\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {… --- ## [Logstash 2.4 to logstash 6.2](https://discuss.elastic.co/t/logstash-2-4-to-logstash-6-2/151237)
**Author:** [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\ **Replies:** 1\ **Last updated:** [October 5, 2018, 4:50pm UTC](https://discuss.elastic.co/t/logstash-2-4-to-logstash-6-2/151237 "2018-10-05T16:50:40Z")
Hi I have elk6.2 already running. I want to read logs from aix system. For Lsf to send logs to logstash, I need to use logstash 2.4. Can I run one instance of LS2.4 for logstash-forwarder and send it's output to LS6.… --- ## [How to push .log file to elasticsearch](https://discuss.elastic.co/t/how-to-push-log-file-to-elasticsearch/151246)
**Author:** [@sundarcse1216](https://discuss.elastic.co/u/sundarcse1216)\ **Replies:** 3\ **Last updated:** [October 5, 2018, 9:18pm UTC](https://discuss.elastic.co/t/how-to-push-log-file-to-elasticsearch/151246 "2018-10-05T21:18:51Z")
Hi all, I have .log file with following content. values separated by \[tab\]. |2018-10-04-00-15-17|10.1.1.100|8080|10.1.1.105|1| |2018-10-04-00-15-20|10.1.1.10|80|10.2.1.15|1| |2018-10-04-00-15-31|10.4.1.20|50560|10.1.… --- ## [Aggregate: end\_of\_task in a conditional block?](https://discuss.elastic.co/t/aggregate-end-of-task-in-a-conditional-block/150191)
**Author:** [@gryvia](https://discuss.elastic.co/u/gryvia)\ **Replies:** 3\ **Last updated:** [October 4, 2018, 5:40pm UTC](https://discuss.elastic.co/t/aggregate-end-of-task-in-a-conditional-block/150191 "2018-10-04T17:40:33Z")
Hello everyone, I'm trying to aggregate log lines that monitor file access into basically two lines per file. A quick example will make it clear. The input looks like this: op:open user:userx path:pathx op:open user:… --- ## [Storing event data as global variable in key value pair manner](https://discuss.elastic.co/t/storing-event-data-as-global-variable-in-key-value-pair-manner/151279)
**Author:** [@xyz2](https://discuss.elastic.co/u/xyz2)\ **Replies:** 0\ **Last updated:** [October 5, 2018, 8:20pm UTC](https://discuss.elastic.co/t/storing-event-data-as-global-variable-in-key-value-pair-manner/151279 "2018-10-05T20:20:57Z")
Hi, I have 2 input files . I read a file , then I want to store 2 values from each event as key value pair in global variable something like map in java. Then I have another file events. As part of this , I want to sea… --- ## [Kafka output failed in 6.4.0](https://discuss.elastic.co/t/kafka-output-failed-in-6-4-0/145868)
**Author:** [@shiloong](https://discuss.elastic.co/u/shiloong)\ **Replies:** 4\ **Last updated:** [October 5, 2018, 7:29pm UTC](https://discuss.elastic.co/t/kafka-output-failed-in-6-4-0/145868 "2018-10-05T19:29:34Z")
In logstash 6.4.0 on MacOS and oracle jdk 1.8.0\_181-b13，use the default setting for kafka output plugin start failed \[logstash.outputs.kafka \] Unable to create Kafka producer from given configuration {:kafka\_error\_messa… --- ## [Elapsed time filter - not matching first event!](https://discuss.elastic.co/t/elapsed-time-filter-not-matching-first-event/150370)
**Author:** [@jonathan.keenan](https://discuss.elastic.co/u/jonathan.keenan)\ **Replies:** 1\ **Last updated:** [October 5, 2018, 7:10pm UTC](https://discuss.elastic.co/t/elapsed-time-filter-not-matching-first-event/150370 "2018-10-05T19:10:51Z")
Hi there, I've searched the forum but it seems to be such a fundamental part of the elapsed time filter that no one else seems to have had the problem, but..... has the keep\_start\_event tag in this filter ever not been … --- ## [Sql\_last\_value Start from First Execution](https://discuss.elastic.co/t/sql-last-value-start-from-first-execution/151269)
**Author:** [@stevenmarrocco](https://discuss.elastic.co/u/stevenmarrocco)\ **Replies:** 0\ **Last updated:** [October 5, 2018, 6:16pm UTC](https://discuss.elastic.co/t/sql-last-value-start-from-first-execution/151269 "2018-10-05T18:16:11Z")
Hello, Is there a way to set the value of sql\_last\_value to when the logstash service starts? We have a very large database and when we run our logstash service it affects the application the database is connected to. … --- ## [Help with Logstash Grok](https://discuss.elastic.co/t/help-with-logstash-grok/151261)
**Author:** [@ndmoe](https://discuss.elastic.co/u/ndmoe)\ **Replies:** 0\ **Last updated:** [October 5, 2018, 5:32pm UTC](https://discuss.elastic.co/t/help-with-logstash-grok/151261 "2018-10-05T17:32:36Z")
I'm sure its something I'm missing but I can't seem to figure this out. I'm trying to build a grok filter for some syslog input from a system. Grokdebugger validates, and so do the other 3 online grok validators I used b… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=672) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=674)