# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=510 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 511 --- ## [Merge 2 fields from seperate events into one field](https://discuss.elastic.co/t/merge-2-fields-from-seperate-events-into-one-field/190587)
**Author:** [@swiftingdarkness](https://discuss.elastic.co/u/swiftingdarkness)\ **Replies:** 1\ **Last updated:** [July 15, 2019, 5:09pm UTC](https://discuss.elastic.co/t/merge-2-fields-from-seperate-events-into-one-field/190587 "2019-07-15T17:09:39Z")
I am trying to merge the field "username" from one source (xx.xxx.xx.xx) with the field "user" from a different source (yy.yyy.yyy.yyy). I want to merge them into a new field called "User". Example Logs: \[USERNAME= joh… --- ## [Bizarre log structure for parsing in Logstash. Opinions appreciated](https://discuss.elastic.co/t/bizarre-log-structure-for-parsing-in-logstash-opinions-appreciated/190509)
**Author:** [@hongphuc95](https://discuss.elastic.co/u/hongphuc95)\ **Replies:** 1\ **Last updated:** [July 15, 2019, 5:05pm UTC](https://discuss.elastic.co/t/bizarre-log-structure-for-parsing-in-logstash-opinions-appreciated/190509 "2019-07-15T17:05:27Z")
Hello everyone, I have a log file which basically doesn’t follow the format of common log files. Normally in the known log like Apache, you have an event logged in one line. Mine, however, spread out through multiline, k… --- ## [Logstash 7.1 not indexing an array of objects](https://discuss.elastic.co/t/logstash-7-1-not-indexing-an-array-of-objects/190586)
**Author:** [@az123](https://discuss.elastic.co/u/az123)\ **Replies:** 0\ **Last updated:** [July 15, 2019, 4:54pm UTC](https://discuss.elastic.co/t/logstash-7-1-not-indexing-an-array-of-objects/190586 "2019-07-15T16:54:46Z")
Hi, I am receiving nested json objects into an array in logstash. Logstash does not index fields within the array (I see unindexed fields cannot be searched message on kibana). How can I change the object mapping from a… --- ## [Logstash json ouput choose keys and values](https://discuss.elastic.co/t/logstash-json-ouput-choose-keys-and-values/190546)
**Author:** [@Sous\_Lesquels](https://discuss.elastic.co/u/Sous_Lesquels)\ **Replies:** 4\ **Last updated:** [July 15, 2019, 4:37pm UTC](https://discuss.elastic.co/t/logstash-json-ouput-choose-keys-and-values/190546 "2019-07-15T16:37:56Z")
Say I have a pipeline that writes events to two files. Both files should be in json format, but they should have different fields from the event and in a different format. For example, say the event is like this: {"a":… --- ## [Question about logstash snmp input engine-id option](https://discuss.elastic.co/t/question-about-logstash-snmp-input-engine-id-option/189617)
**Author:** [@az123](https://discuss.elastic.co/u/az123)\ **Replies:** 1\ **Last updated:** [July 15, 2019, 4:29pm UTC](https://discuss.elastic.co/t/question-about-logstash-snmp-input-engine-id-option/189617 "2019-07-15T16:29:59Z")
Hi, I am trying to get some data from a node via snmpv3 into logstash. I need to provide engine-id for the snmpwalk to get meaningful data from the node. How do I do that in the logstash config file? --- ## [Is GROK parsing automatically assigning a \_grokparsefailure to tags field if it does not match a pattern?](https://discuss.elastic.co/t/is-grok-parsing-automatically-assigning-a-grokparsefailure-to-tags-field-if-it-does-not-match-a-pattern/190566)
**Author:** [@skydancer](https://discuss.elastic.co/u/skydancer)\ **Replies:** 2\ **Last updated:** [July 15, 2019, 3:38pm UTC](https://discuss.elastic.co/t/is-grok-parsing-automatically-assigning-a-grokparsefailure-to-tags-field-if-it-does-not-match-a-pattern/190566 "2019-07-15T15:38:11Z")
I imagine this may be a trivial question but since it is my early days with GROK patterns I want to makes sure. I created a custom GROK pattern for my environment and it works fine: my filter file looks for a presence of… --- ## [Non-Production Logstash Instance Accepts Filebeat Connection, but not Production](https://discuss.elastic.co/t/non-production-logstash-instance-accepts-filebeat-connection-but-not-production/190106)
**Author:** [@DougR](https://discuss.elastic.co/u/DougR)\ **Replies:** 5\ **Last updated:** [July 15, 2019, 3:02pm UTC](https://discuss.elastic.co/t/non-production-logstash-instance-accepts-filebeat-connection-but-not-production/190106 "2019-07-15T15:02:24Z")
I'm stumped. I have two ELK stacks running - one in production, one in non-production. Each uses the identical config. I have numerous beats reporting into each. When attempting to connect a new filebeat to the product… --- ## [Conditional is not working to find nested JSON field](https://discuss.elastic.co/t/conditional-is-not-working-to-find-nested-json-field/190430)
**Author:** [@76a99de1e92a429af5f9](https://discuss.elastic.co/u/76a99de1e92a429af5f9)\ **Replies:** 1\ **Last updated:** [July 15, 2019, 2:59pm UTC](https://discuss.elastic.co/t/conditional-is-not-working-to-find-nested-json-field/190430 "2019-07-15T14:59:24Z")
Hi, I used logstash filter to avoid mapper-parsing-exception in this case. JSON Data {"request\_body":{ "enveloped\_message":{"X":"x", "Y":"y"} } OR {"request\_body":"ABC"} logstash filter config if "enveloped\_m… --- ## [Can't merge a non object mapping \[EventId\] with an object mapping \[EventId\]](https://discuss.elastic.co/t/cant-merge-a-non-object-mapping-eventid-with-an-object-mapping-eventid/190529)
**Author:** [@minder](https://discuss.elastic.co/u/minder)\ **Replies:** 1\ **Last updated:** [July 15, 2019, 2:20pm UTC](https://discuss.elastic.co/t/cant-merge-a-non-object-mapping-eventid-with-an-object-mapping-eventid/190529 "2019-07-15T14:20:56Z")
have got a log: { "time": "2019-07-12T16:16:27.5682286+03:00", "level": "ERROR", "message": "{"AddedTime":"2019-07-12 16:16:23.8962","RequestedTime":"2019-07-12 16:16:23.8962","WebSso":"AQIC5wM2LY4Sfcw7nbDeQ9IKsDkdlSwuQ… --- ## [Dead Letter Queue and its usage](https://discuss.elastic.co/t/dead-letter-queue-and-its-usage/189984)
**Author:** [@dhruvsingh](https://discuss.elastic.co/u/dhruvsingh)\ **Replies:** 6\ **Last updated:** [July 15, 2019, 1:58pm UTC](https://discuss.elastic.co/t/dead-letter-queue-and-its-usage/189984 "2019-07-15T13:58:22Z")
There have already been a lot of discussions on dead\_letter\_queue but FWIW I could not find answers to all my questions, and thus posting them out here. My setup (local machine): logstash - 6.8.1 ES - Version: 6.8.1, … --- ## [Logstash-input-jdbc raise error\_message=\>"\\"\\\\x89\\" from ASCII-8BIT to UTF-8"](https://discuss.elastic.co/t/logstash-input-jdbc-raise-error-message-x89-from-ascii-8bit-to-utf-8/190540)
**Author:** [@muzijiang](https://discuss.elastic.co/u/muzijiang)\ **Replies:** 0\ **Last updated:** [July 15, 2019, 1:14pm UTC](https://discuss.elastic.co/t/logstash-input-jdbc-raise-error-message-x89-from-ascii-8bit-to-utf-8/190540 "2019-07-15T13:14:48Z")
When I use logstash-input-jdbc plugin collect infor from Oracle database, it raised the following error messages. input { jdbc { jdbc\_validate\_connection =\> true jdbc\_connection\_string =\> "jdbc:oracle:thin:@xx.xx.xx.… --- ## [Logstash going to failed state after restart](https://discuss.elastic.co/t/logstash-going-to-failed-state-after-restart/189373)
**Author:** [@soumya2](https://discuss.elastic.co/u/soumya2)\ **Replies:** 10\ **Last updated:** [July 15, 2019, 11:47am UTC](https://discuss.elastic.co/t/logstash-going-to-failed-state-after-restart/189373 "2019-07-15T11:47:17Z")
Hi Team, i am configured ELK stack for visualizing Squid logs in Kibana. i use filebeat in squid server to send the log to logstash. In logstash i filtered the squid logs using grok . but after restart logstash it goe… --- ## [Aws elasticsearch input plugin](https://discuss.elastic.co/t/aws-elasticsearch-input-plugin/190500)
**Author:** [@Uchiha\_Kakashi](https://discuss.elastic.co/u/Uchiha_Kakashi)\ **Replies:** 0\ **Last updated:** [July 15, 2019, 10:29am UTC](https://discuss.elastic.co/t/aws-elasticsearch-input-plugin/190500 "2019-07-15T10:29:32Z")
I am trying to import data from amazon elasticsearch service but, unlike amazon\_es output plugin, there is no input plugin for amazon\_es. Using elasticsearch input plugin did not work ..... \< input { amazon\_es { hos… --- ## [Error parsing JSON content from Http](https://discuss.elastic.co/t/error-parsing-json-content-from-http/190069)
**Author:** [@SamDi](https://discuss.elastic.co/u/SamDi)\ **Replies:** 6\ **Last updated:** [July 15, 2019, 10:14am UTC](https://discuss.elastic.co/t/error-parsing-json-content-from-http/190069 "2019-07-15T10:14:08Z")
Hello to everybody. I am really noob on elastic stack and I am trying to import some json information with the HTTP input to Elasticsarch, but I recive a parse error: exception=\>#\ **Author:** [@floflo](https://discuss.elastic.co/u/floflo)\ **Replies:** 0\ **Last updated:** [July 15, 2019, 8:29am UTC](https://discuss.elastic.co/t/nullpointerexception-in-logstash-filter-throttle/190464 "2019-07-15T08:29:40Z") Hi there, I am currently trying to make the Throttle plugin work in logstash (7.2.0) but some nasty NullPointerException occurs and I can't manage to understand why from the stack trace. Any idea ? Filter config: filt… --- ## [Way to identify matching grok pattern](https://discuss.elastic.co/t/way-to-identify-matching-grok-pattern/190082)
**Author:** [@AmolB](https://discuss.elastic.co/u/AmolB)\ **Replies:** 6\ **Last updated:** [July 15, 2019, 7:52am UTC](https://discuss.elastic.co/t/way-to-identify-matching-grok-pattern/190082 "2019-07-15T07:52:26Z")
Hello Everyone, I am using Grok filter plugin for Logstash (Logstash version 6.7.1) . My log line has several formats, so I have written more than 10 grok patterns for it. Now, I am interested in understanding the grok … --- ## [Sniffing recover after all instances are down](https://discuss.elastic.co/t/sniffing-recover-after-all-instances-are-down/190445)
**Author:** [@alonana](https://discuss.elastic.co/u/alonana)\ **Replies:** 0\ **Last updated:** [July 15, 2019, 7:31am UTC](https://discuss.elastic.co/t/sniffing-recover-after-all-instances-are-down/190445 "2019-07-15T07:31:05Z")
We are using Logstash pipeline with Elasticsearch output plugin in a kubernetes environment. We've added sniffing to the plugin: ... output { elasticsearch { ... hosts =\> \["elasticsearch-service"\] sniffin… --- ## [Warning Could not find logstash.yml](https://discuss.elastic.co/t/warning-could-not-find-logstash-yml/189952)
**Author:** [@Andreasky](https://discuss.elastic.co/u/Andreasky)\ **Replies:** 5\ **Last updated:** [July 15, 2019, 6:35am UTC](https://discuss.elastic.co/t/warning-could-not-find-logstash-yml/189952 "2019-07-15T06:35:54Z")
WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults I see that there are others that have th… --- ## [How to read custom logs from azure blob storage](https://discuss.elastic.co/t/how-to-read-custom-logs-from-azure-blob-storage/190416)
**Author:** [@Altiano\_Gerung](https://discuss.elastic.co/u/Altiano_Gerung)\ **Replies:** 0\ **Last updated:** [July 15, 2019, 3:51am UTC](https://discuss.elastic.co/t/how-to-read-custom-logs-from-azure-blob-storage/190416 "2019-07-15T03:51:28Z")
I have setup a logging in my API to a blob storage. I have no idea how to use logstash azure event hub plugin to get notified every log that is written the file. Any idea? --- ## [Logstash-output-amazon\_es](https://discuss.elastic.co/t/logstash-output-amazon-es/190398)
**Author:** [@amandrell1](https://discuss.elastic.co/u/amandrell1)\ **Replies:** 1\ **Last updated:** [July 14, 2019, 11:09pm UTC](https://discuss.elastic.co/t/logstash-output-amazon-es/190398 "2019-07-14T23:09:20Z")
Hi! I am having trouble understanding how to download the amazon\_es plugin for logstash. Current documentation says that we should be able to install with the command '/bin/logstash install logstash-output-amazone\_es'. M… --- ## [Date in the name of an index](https://discuss.elastic.co/t/date-in-the-name-of-an-index/190396)
**Author:** [@GGUERRA](https://discuss.elastic.co/u/GGUERRA)\ **Replies:** 2\ **Last updated:** [July 14, 2019, 10:09pm UTC](https://discuss.elastic.co/t/date-in-the-name-of-an-index/190396 "2019-07-14T22:09:43Z")
Hello everyone, I would like to ask about the generation of indices. I have noticed that in the logstash output in many example cases it is usually written: output { elasticsearch { hosts =\> \["x.x.x.x"\] … --- ## [\[SOLVED\] Grok - multiple match lines - set tag based on which grok match was used](https://discuss.elastic.co/t/solved-grok-multiple-match-lines-set-tag-based-on-which-grok-match-was-used/190368)
**Author:** [@nnet](https://discuss.elastic.co/u/nnet)\ **Replies:** 1\ **Last updated:** [July 14, 2019, 2:44pm UTC](https://discuss.elastic.co/t/solved-grok-multiple-match-lines-set-tag-based-on-which-grok-match-was-used/190368 "2019-07-14T14:44:34Z")
I have a grok match statement with multiple matches: grok { match =\> { "message" =\> \[ "\<%{NONNEGINT:syslog\_pri}\>%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{INT:cisco\_eventnum:int}: %{DATA:cis… --- ## [Some logs to Elastic, some to SIEM](https://discuss.elastic.co/t/some-logs-to-elastic-some-to-siem/186307)
**Author:** [@Vladx](https://discuss.elastic.co/u/Vladx)\ **Replies:** 11\ **Last updated:** [July 14, 2019, 11:59am UTC](https://discuss.elastic.co/t/some-logs-to-elastic-some-to-siem/186307 "2019-07-14T11:59:14Z")
Hi, I'm quite new here, so please forgive my dumbness on this. But I've spent the whole afternoon to figure out how can I achieve what I want. Digged into docs, forums and so on... Maybe it is because I'm coming from th… --- ## [Logstash config works but unable to see logs in Kibana](https://discuss.elastic.co/t/logstash-config-works-but-unable-to-see-logs-in-kibana/190374)
**Author:** [@Mohammad\_hossein\_Taj](https://discuss.elastic.co/u/Mohammad_hossein_Taj)\ **Replies:** 1\ **Last updated:** [July 14, 2019, 11:36am UTC](https://discuss.elastic.co/t/logstash-config-works-but-unable-to-see-logs-in-kibana/190374 "2019-07-14T11:36:34Z")
Hi. I'm trying to read two types of log files that one of them has different logs that one of them is in this type: 2019-06-26 01:00:31,069 INFO ir.ac.ut.sdrwebservice.SDRWebService @ batchAddStdDoc, System:G, User:2511… --- ## [How to handle https end point monitoring using certs](https://discuss.elastic.co/t/how-to-handle-https-end-point-monitoring-using-certs/189187)
**Author:** [@sukku77](https://discuss.elastic.co/u/sukku77)\ **Replies:** 4\ **Last updated:** [July 13, 2019, 6:19am UTC](https://discuss.elastic.co/t/how-to-handle-https-end-point-monitoring-using-certs/189187 "2019-07-13T06:19:52Z")
Hi, I would like to invoke below https url for health check every 30s. I am using http\_poller inut plugin. But getting below handshake issue Host name 'xxxxxxx' does not match the certificate subject provided by the pe… --- ## [Trouble Ingesting JSON](https://discuss.elastic.co/t/trouble-ingesting-json/190315)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 3\ **Last updated:** [July 13, 2019, 1:08pm UTC](https://discuss.elastic.co/t/trouble-ingesting-json/190315 "2019-07-13T13:08:07Z")
I am trying to ingest a JSON document. Unfortunately, there seems to be a problem with null values on an object field. There are multiple fields like this, how can I resolve it? When the field is null, it looks like t… --- ## [Display text file in Kibana with filebeat and logstash](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692)
**Author:** [@Vladpov](https://discuss.elastic.co/u/Vladpov)\ **Replies:** 11\ **Last updated:** [July 13, 2019, 10:29am UTC](https://discuss.elastic.co/t/display-text-file-in-kibana-with-filebeat-and-logstash/189692 "2019-07-13T10:29:44Z")
Hi guys, I run ELK Stack (one machine) with filebeat (second machine) as log shipper. I've got txt file with 4228 rows where each row is one log in follow form: Jul 4 13:56:17 vMMR mmr-core\[29839\]: GtsAwegAOMTbez\_15… --- ## [How to send LogStash parse errors to ElasticSearch](https://discuss.elastic.co/t/how-to-send-logstash-parse-errors-to-elasticsearch/190292)
**Author:** [@Joe\_Still](https://discuss.elastic.co/u/Joe_Still)\ **Replies:** 4\ **Last updated:** [July 12, 2019, 7:24pm UTC](https://discuss.elastic.co/t/how-to-send-logstash-parse-errors-to-elasticsearch/190292 "2019-07-12T19:24:56Z")
Hello, I am running into issues where my JSON logs being sent to Logstash are sometimes failing to be parsed. The parse failure is just me sending bad data. However, Logstash does log that it failed to parse it: "error"… --- ## [Logstash filter to remove a parent in a big array](https://discuss.elastic.co/t/logstash-filter-to-remove-a-parent-in-a-big-array/190289)
**Author:** [@tellus83](https://discuss.elastic.co/u/tellus83)\ **Replies:** 1\ **Last updated:** [July 12, 2019, 7:07pm UTC](https://discuss.elastic.co/t/logstash-filter-to-remove-a-parent-in-a-big-array/190289 "2019-07-12T19:07:03Z")
I simply want to remove the parent in a big array whitout creating mute rname rules for all felds. This is what i have now. "timestamp": "2019-07-12T15:37:06.516Z", "foo": { "bar": { "number": "one" } }, "value… --- ## [How to handle XML file "Last ## unconsumed characters" error](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385)
**Author:** [@edster](https://discuss.elastic.co/u/edster)\ **Replies:** 10\ **Last updated:** [July 12, 2019, 7:04pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385 "2019-07-12T19:04:23Z")
Elastic has an issue parsing xml. Returns error "Error parsing xml with XmlSimple...." Also "Last ## unconsumed characters". Logstash config file is as follows input { file { path =\> "filepath" start\_position =\> "b… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=509) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=511)