# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=389 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 390 --- ## [@timestamp field is missing in dead letter queue](https://discuss.elastic.co/t/timestamp-field-is-missing-in-dead-letter-queue/223749)
**Author:** [@darkwarrior](https://discuss.elastic.co/u/darkwarrior)\ **Replies:** 0\ **Last updated:** [March 16, 2020, 11:56am UTC](https://discuss.elastic.co/t/timestamp-field-is-missing-in-dead-letter-queue/223749 "2020-03-16T11:56:05Z")
Hello everyone, I am using dead letter queue and based on some filters I want to index all the processed events to ES. In output section of dlq I am using, index =\> "index-prod-l%{+YYYYMMdd}" to create dynamic indices… --- ## [Vibrant slack or IRC needed at least Mailing list](https://discuss.elastic.co/t/vibrant-slack-or-irc-needed-at-least-mailing-list/223727)
**Author:** [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Replies:** 0\ **Last updated:** [March 16, 2020, 10:12am UTC](https://discuss.elastic.co/t/vibrant-slack-or-irc-needed-at-least-mailing-list/223727 "2020-03-16T10:12:57Z")
Hi all , i know there is an IRC for logstash and 2 channles for elastic and 1 for kibana in freenode. but none of them are vibrant. when i say vibrant i dont mean that , as soon as you post a question some one should he… --- ## [Help with parse logs not standardized](https://discuss.elastic.co/t/help-with-parse-logs-not-standardized/223618)
**Author:** [@Ernest\_Lachowicz](https://discuss.elastic.co/u/Ernest_Lachowicz)\ **Replies:** 3\ **Last updated:** [March 16, 2020, 9:28am UTC](https://discuss.elastic.co/t/help-with-parse-logs-not-standardized/223618 "2020-03-16T09:28:06Z")
Hi, I'm starting my adventure with Logstash (ELK). I would like parse logs: 2020-02-11 15:10:17.435 DEV\_CTRL UPPER OUTPUT/ue/ue1/dev\_ctrl/logger IMSI=01723600 Sending message BearerContextReleased: BearerContextRelease… --- ## [Sprintf reference in grok pattern (help me~!)](https://discuss.elastic.co/t/sprintf-reference-in-grok-pattern-help-me/223694)
**Author:** [@judragon\_dark](https://discuss.elastic.co/u/judragon_dark)\ **Replies:** 4\ **Last updated:** [March 16, 2020, 8:57am UTC](https://discuss.elastic.co/t/sprintf-reference-in-grok-pattern-help-me/223694 "2020-03-16T08:57:47Z")
logstash filter code filter{ grok { patterns\_dir =\> \["/etc/logstash/patterns/ssk-patterns"\] match =\> {"message" =\> "%{\[fields\]\[log\_type\]}"} } } \[fields… --- ## [How can parse log data into individual key=val pair dynamically in logstash?](https://discuss.elastic.co/t/how-can-parse-log-data-into-individual-key-val-pair-dynamically-in-logstash/223317)
**Author:** [@Pathshala\_Gopi](https://discuss.elastic.co/u/Pathshala_Gopi)\ **Replies:** 8\ **Last updated:** [March 16, 2020, 7:14am UTC](https://discuss.elastic.co/t/how-can-parse-log-data-into-individual-key-val-pair-dynamically-in-logstash/223317 "2020-03-16T07:14:26Z")
My question is how can i separate the log data into individual key=val pair in logstash.This means when i get the log data from beats or file sources it will be in string form so i need that string must be divided separa… --- ## [Elapsed filter plugin issues](https://discuss.elastic.co/t/elapsed-filter-plugin-issues/223246)
**Author:** [@Nicolas\_Aizier](https://discuss.elastic.co/u/Nicolas_Aizier)\ **Replies:** 3\ **Last updated:** [March 16, 2020, 6:47am UTC](https://discuss.elastic.co/t/elapsed-filter-plugin-issues/223246 "2020-03-16T06:47:52Z")
Hi all, I'll try to keep it short and technical enough to not be boring (I too hate to read long drifting problems) I've got some issues with the elapsed plugin. Here is my setup in a nutshell. Just know that this set… --- ## [How can i send duplicate lines (from logstash to elasticsearch) in log file](https://discuss.elastic.co/t/how-can-i-send-duplicate-lines-from-logstash-to-elasticsearch-in-log-file/222064)
**Author:** [@chandanachk](https://discuss.elastic.co/u/chandanachk)\ **Replies:** 5\ **Last updated:** [March 16, 2020, 4:38am UTC](https://discuss.elastic.co/t/how-can-i-send-duplicate-lines-from-logstash-to-elasticsearch-in-log-file/222064 "2020-03-16T04:38:42Z")
I'm a beginner of this area and i want to send duplicate lines of a log file to elastic search. currently it is identifying duplicates and send only one line, sample log file 00:00:59,62.10.38.47,39473311,"POST /v0.1/te… --- ## [Pipeline virtual address from Filbeat event field](https://discuss.elastic.co/t/pipeline-virtual-address-from-filbeat-event-field/223665)
**Author:** [@sjne](https://discuss.elastic.co/u/sjne)\ **Replies:** 1\ **Last updated:** [March 15, 2020, 11:16pm UTC](https://discuss.elastic.co/t/pipeline-virtual-address-from-filbeat-event-field/223665 "2020-03-15T23:16:44Z")
Is it possible to get the pipeline-to-pipeline virtual address by accessing event fields? Sort of like this: Filebeat config filebeat.inputs: - type: log enabled: true fields: { instance\_id: "first" } paths: … --- ## [Filebeat doesnt send event.dataset and event.module](https://discuss.elastic.co/t/filebeat-doesnt-send-event-dataset-and-event-module/223662)
**Author:** [@Burga](https://discuss.elastic.co/u/Burga)\ **Replies:** 0\ **Last updated:** [March 15, 2020, 7:26pm UTC](https://discuss.elastic.co/t/filebeat-doesnt-send-event-dataset-and-event-module/223662 "2020-03-15T19:26:55Z")
Hi , I'm new with logstash and filebeat so I was loosing my mind trying to understand why logstash doesn't use filter and I figured out that filebeat doesn't send event.dataset and event.module fields , how can I f… --- ## [Log-stash not showing output in stdout](https://discuss.elastic.co/t/log-stash-not-showing-output-in-stdout/223610)
**Author:** [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Replies:** 7\ **Last updated:** [March 15, 2020, 7:17pm UTC](https://discuss.elastic.co/t/log-stash-not-showing-output-in-stdout/223610 "2020-03-15T19:17:49Z")
config -\> my config file logstash -f "C:\\busapps\\drsb\\gbl0\\logstash\\7.0.0\\bin\\pipelines\\logstash.conf" above is my command below is the output : \[2020-03-14T10:43:12,670\]\[WARN \]\[logstash.config.source.multilocal… --- ## [Passing to output index field but getting \[@metadata\]\[index\]](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650)
**Author:** [@Burga](https://discuss.elastic.co/u/Burga)\ **Replies:** 9\ **Last updated:** [March 15, 2020, 5:50pm UTC](https://discuss.elastic.co/t/passing-to-output-index-field-but-getting-metadata-index/223650 "2020-03-15T17:50:12Z")
Hi , i'm creating index name dynamically but when I'm passing the field to output i'm getting the variable name (\[@metadata\]\[index\]): mutate { add\_field =\> { "\[@metadata\]\[index\_prefix\]" =\> "system-secure" } } outpu… --- ## [Pipeline-to-pipeline with single input and output](https://discuss.elastic.co/t/pipeline-to-pipeline-with-single-input-and-output/223631)
**Author:** [@Burga](https://discuss.elastic.co/u/Burga)\ **Replies:** 0\ **Last updated:** [March 14, 2020, 6:45pm UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-with-single-input-and-output/223631 "2020-03-14T18:45:25Z")
Hi , I'm trying to configure logstash with pipelines configuration logstash pipelines: input --\> conf files with different log types (system,security, auditing ...) --\> output but it doesn't work(without errors ) ,… --- ## [How to set path.data and pidfile per conf file in Logstash Service](https://discuss.elastic.co/t/how-to-set-path-data-and-pidfile-per-conf-file-in-logstash-service/223630)
**Author:** [@Yungyoung\_Ok](https://discuss.elastic.co/u/Yungyoung_Ok)\ **Replies:** 0\ **Last updated:** [March 14, 2020, 6:24pm UTC](https://discuss.elastic.co/t/how-to-set-path-data-and-pidfile-per-conf-file-in-logstash-service/223630 "2020-03-14T18:24:02Z")
I am running logstash as a service using systemctl. Running logstash with systemctl runs the conf files in conf.d directory. First, How do I set path.data for each conf file in conf.d directory? Second, how to set pid… --- ## [Aggregate only working when timeout = 0](https://discuss.elastic.co/t/aggregate-only-working-when-timeout-0/223617)
**Author:** [@karlm](https://discuss.elastic.co/u/karlm)\ **Replies:** 0\ **Last updated:** [March 14, 2020, 1:41pm UTC](https://discuss.elastic.co/t/aggregate-only-working-when-timeout-0/223617 "2020-03-14T13:41:21Z")
I have an Elasticsearch input that collects list of fails that have not been handled. Then aggregate the false count and output the count to an index. It only works when the timeout = 0, if i change the timeout to anyth… --- ## [Push json key value with ELK](https://discuss.elastic.co/t/push-json-key-value-with-elk/223112)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 13\ **Last updated:** [March 14, 2020, 12:33pm UTC](https://discuss.elastic.co/t/push-json-key-value-with-elk/223112 "2020-03-14T12:33:33Z")
I have a lot of json like this that every day have different data: \[{ "date": "2020-02-24 21:00:00", "country": "US", "men": 101, "women": 26, "childreni": 127, }\] I tried to push this into elasticsearch but n… --- ## [Log stash filters log message - need help](https://discuss.elastic.co/t/log-stash-filters-log-message-need-help/222076)
**Author:** [@Seetharaman\_K](https://discuss.elastic.co/u/Seetharaman_K)\ **Replies:** 5\ **Last updated:** [March 14, 2020, 11:10am UTC](https://discuss.elastic.co/t/log-stash-filters-log-message-need-help/222076 "2020-03-14T11:10:21Z")
Below are the log stash fields Log stash error : \[2020-03-02T16:48:48,354\]\[WARN \]\[logstash.filters.json \] Error parsing json {:source=\>"message", :raw=\>"WITH CORRESPONDING LINE(S)", "HoldReason": "Invoice pric… --- ## [Logstash pipeline actively closes connection](https://discuss.elastic.co/t/logstash-pipeline-actively-closes-connection/223569)
**Author:** [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\ **Replies:** 0\ **Last updated:** [March 13, 2020, 8:05pm UTC](https://discuss.elastic.co/t/logstash-pipeline-actively-closes-connection/223569 "2020-03-13T20:05:25Z")
Greetings, I am running logstash with the following configuration change in the logstash.yml file: http.host: "192.168.0.250" Here is the pipeline configuration: input { beats { port =\> 5044 } } output { e… --- ## [Error: no such instance currently exists at this OID](https://discuss.elastic.co/t/error-no-such-instance-currently-exists-at-this-oid/223575)
**Author:** [@fabian1](https://discuss.elastic.co/u/fabian1)\ **Replies:** 0\ **Last updated:** [March 13, 2020, 8:39pm UTC](https://discuss.elastic.co/t/error-no-such-instance-currently-exists-at-this-oid/223575 "2020-03-13T20:39:14Z")
I am working it with logstash version 7.5.2 with snmp input, but I get the following error: error: no such instance currently exists at this OID Below I attached the configuration on the Cisco sg-500 switch and the pipel… --- ## [How can i configure logstash to read every new csv files automatically?](https://discuss.elastic.co/t/how-can-i-configure-logstash-to-read-every-new-csv-files-automatically/223394)
**Author:** [@Emna1](https://discuss.elastic.co/u/Emna1)\ **Replies:** 4\ **Last updated:** [March 13, 2020, 8:11pm UTC](https://discuss.elastic.co/t/how-can-i-configure-logstash-to-read-every-new-csv-files-automatically/223394 "2020-03-13T20:11:41Z")
How can i use logstash to read csv files in real time ? How can i configure logstash to read evey new csv files automatically ? I have an empty folder,I want each time by adding a file in this folder, logstash automati… --- ## [Convert String to IP Address - Hexidecimal](https://discuss.elastic.co/t/convert-string-to-ip-address-hexidecimal/223192)
**Author:** [@Jim\_Thunder](https://discuss.elastic.co/u/Jim_Thunder)\ **Replies:** 4\ **Last updated:** [March 13, 2020, 6:26pm UTC](https://discuss.elastic.co/t/convert-string-to-ip-address-hexidecimal/223192 "2020-03-13T18:26:56Z")
Hello! I have fields that contain long Hexidecimal numbers 000000000FFFF24CA110B The prefix is the 0's and the four F's. So the only needed characters are the last 8 (2 for each octet of the IP address). This number n… --- ## [Parsing dynamically using Logstash grok pattern(custom pattern)](https://discuss.elastic.co/t/parsing-dynamically-using-logstash-grok-pattern-custom-pattern/223450)
**Author:** [@judragon\_dark](https://discuss.elastic.co/u/judragon_dark)\ **Replies:** 1\ **Last updated:** [March 13, 2020, 6:00pm UTC](https://discuss.elastic.co/t/parsing-dynamically-using-logstash-grok-pattern-custom-pattern/223450 "2020-03-13T18:00:11Z")
Below is the code for the logstash filter. MYCUSTOMPATTERN is a custom pattern defined in ssk-patterns. filter { grok { patterns\_dir =\> \["/etc/logstash/patterns/ssk-patterns"\] match =\> {"mes… --- ## [Unable to see the logs in kibana dashboard . But Filebeat is working fine](https://discuss.elastic.co/t/unable-to-see-the-logs-in-kibana-dashboard-but-filebeat-is-working-fine/223459)
**Author:** [@jathev\_GE](https://discuss.elastic.co/u/jathev_GE)\ **Replies:** 1\ **Last updated:** [March 13, 2020, 5:59pm UTC](https://discuss.elastic.co/t/unable-to-see-the-logs-in-kibana-dashboard-but-filebeat-is-working-fine/223459 "2020-03-13T17:59:05Z")
I'm unable to see the logs in kibana(kibana 6.3.1) dashboard. But Filebeat is working fine. When I checked it in logstash, I got some errors. Please find the error message below & give some advice about this. ==========… --- ## [Getting Json parsing exception in logstash.filters.json](https://discuss.elastic.co/t/getting-json-parsing-exception-in-logstash-filters-json/223489)
**Author:** [@Arpit\_Pruthi](https://discuss.elastic.co/u/Arpit_Pruthi)\ **Replies:** 1\ **Last updated:** [March 13, 2020, 5:49pm UTC](https://discuss.elastic.co/t/getting-json-parsing-exception-in-logstash-filters-json/223489 "2020-03-13T17:49:05Z")
This is my logstash config :- input { stdin { } } filter { grok{ patterns\_dir =\> \["./patterns"\] match =\> {"message" =\> "^abc-logs:%{TEMP:msg}" }} json{source =\> "msg"} kv{ source =\> … --- ## [Emailing in Logstash](https://discuss.elastic.co/t/emailing-in-logstash/223493)
**Author:** [@Sagar\_Mandal](https://discuss.elastic.co/u/Sagar_Mandal)\ **Replies:** 1\ **Last updated:** [March 13, 2020, 5:48pm UTC](https://discuss.elastic.co/t/emailing-in-logstash/223493 "2020-03-13T17:48:10Z")
Hi Team, How to send Email through logstash-output section ?? Thanks --- ## [Logstash date field problem](https://discuss.elastic.co/t/logstash-date-field-problem/223388)
**Author:** [@tomba2k](https://discuss.elastic.co/u/tomba2k)\ **Replies:** 1\ **Last updated:** [March 13, 2020, 4:53pm UTC](https://discuss.elastic.co/t/logstash-date-field-problem/223388 "2020-03-13T16:53:14Z")
I cannot get logstash to format custom field to be suitable for sort in kibana. pipeline filter config is: filter { if (\[fields\]\[log\_type\] == "zira\_prod\_log") { grok { match =\> { "message" =\> "\[%{TIMESTAMP\_ISO8601:… --- ## [Filters Not Working At All](https://discuss.elastic.co/t/filters-not-working-at-all/222150)
**Author:** [@ck\_montco](https://discuss.elastic.co/u/ck_montco)\ **Replies:** 10\ **Last updated:** [March 13, 2020, 1:55pm UTC](https://discuss.elastic.co/t/filters-not-working-at-all/222150 "2020-03-13T13:55:16Z")
I have the following filter set up to drop certain logs and add tags to syslog events from my Sophos XG550 firewalls: input { tcp { port =\> 6000 type =\> syslog } udp { port =\> 6000 type =\> syslog … --- ## [Failed to excute logstash](https://discuss.elastic.co/t/failed-to-excute-logstash/223341)
**Author:** [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\ **Replies:** 6\ **Last updated:** [March 13, 2020, 1:27pm UTC](https://discuss.elastic.co/t/failed-to-excute-logstash/223341 "2020-03-13T13:27:56Z")
Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::PluginLoadingError", :message=\>"Couldn't find any input plugin named 'filebeat'. Are you sure this is corr… --- ## [Index name is not read from my logstash configuration \[SOLVED\]](https://discuss.elastic.co/t/index-name-is-not-read-from-my-logstash-configuration-solved/223512)
**Author:** [@Axl](https://discuss.elastic.co/u/Axl)\ **Replies:** 1\ **Last updated:** [March 13, 2020, 1:25pm UTC](https://discuss.elastic.co/t/index-name-is-not-read-from-my-logstash-configuration-solved/223512 "2020-03-13T13:25:02Z")
I am a newbie to ELK, and this is very probably a n00b question. I have set up ELK, and those are currently the indices that I have on my host: curl 'http://localhost:9200/\_cat/indices?v' health status index … --- ## [Multiple pipelines configuration while using the distributor pattern](https://discuss.elastic.co/t/multiple-pipelines-configuration-while-using-the-distributor-pattern/223100)
**Author:** [@moscha](https://discuss.elastic.co/u/moscha)\ **Replies:** 1\ **Last updated:** [March 13, 2020, 12:57pm UTC](https://discuss.elastic.co/t/multiple-pipelines-configuration-while-using-the-distributor-pattern/223100 "2020-03-13T12:57:05Z")
Hello all. I have a quick question about using multiple pipelines along with the distributor pattern. So basically, as per the given examples on the official doc, I have set up the following piece of code, so depending … --- ## [Logstash read the entire file again whenever pipeline reload](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247)
**Author:** [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Replies:** 12\ **Last updated:** [March 13, 2020, 10:34am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247 "2020-03-13T10:34:31Z")
Hi all, What I want is read all the file in a directory and tail for new change and discover new files Btw, the sincedb file I specified is always 0byte, nothing ever write My config: input { file { path =\> "/home… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=388) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=390)