# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=519 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 520 --- ## [Nested Json throws error "can't merge a non object mapping \[field\] with an object mapping \[field\]"](https://discuss.elastic.co/t/nested-json-throws-error-cant-merge-a-non-object-mapping-field-with-an-object-mapping-field/188123)
**Author:** [@Suresh2011](https://discuss.elastic.co/u/Suresh2011)\ **Replies:** 0\ **Last updated:** [June 29, 2019, 4:24am UTC](https://discuss.elastic.co/t/nested-json-throws-error-cant-merge-a-non-object-mapping-field-with-an-object-mapping-field/188123 "2019-06-29T04:24:05Z")
Hi, I have a nested json dataset of 2 layers which consist of around 1300 fields in the second layer. I am getting the below error for some fields ERROR: type =\> illegal\_argument\_exception reason =\> can't merge a non… --- ## [Regarding automapping for json file](https://discuss.elastic.co/t/regarding-automapping-for-json-file/188109)
**Author:** [@leeyu](https://discuss.elastic.co/u/leeyu)\ **Replies:** 5\ **Last updated:** [June 28, 2019, 11:52pm UTC](https://discuss.elastic.co/t/regarding-automapping-for-json-file/188109 "2019-06-28T23:52:22Z")
Hi, all Is there any way to disable json's automapping? I tried to use the template to disable the numeric detection but it does not work. Since my data has many fields, it does not make sense to change type for each … --- ## [Persistent Queues with multiple logstash processes](https://discuss.elastic.co/t/persistent-queues-with-multiple-logstash-processes/188099)
**Author:** [@aedades](https://discuss.elastic.co/u/aedades)\ **Replies:** 1\ **Last updated:** [June 28, 2019, 8:34pm UTC](https://discuss.elastic.co/t/persistent-queues-with-multiple-logstash-processes/188099 "2019-06-28T20:34:38Z")
We are using the official logstash docker image and want to use the persistent queue feature (using the default path.queue, LOGSTASH\_HOME/data/queue, and mounting it to something like /etc/logstash/queue/ on the host). I… --- ## [Fetching data from a csv file and updating another file accordingly](https://discuss.elastic.co/t/fetching-data-from-a-csv-file-and-updating-another-file-accordingly/187281)
**Author:** [@jatin6418](https://discuss.elastic.co/u/jatin6418)\ **Replies:** 4\ **Last updated:** [June 28, 2019, 6:20pm UTC](https://discuss.elastic.co/t/fetching-data-from-a-csv-file-and-updating-another-file-accordingly/187281 "2019-06-28T18:20:23Z")
hey I am having two files one is a log file and one is a csv file. From log file I am reading the itemid and I want to add the name of this item which is stored in the csv file can someone help me . input{ file{ … --- ## [Constructing snmp input for logstash](https://discuss.elastic.co/t/constructing-snmp-input-for-logstash/187663)
**Author:** [@ksarpong](https://discuss.elastic.co/u/ksarpong)\ **Replies:** 5\ **Last updated:** [June 28, 2019, 4:34pm UTC](https://discuss.elastic.co/t/constructing-snmp-input-for-logstash/187663 "2019-06-28T16:34:42Z")
Newbie here, can someone outline how I'd parse construct the inputs for log example below in Logstash? I replaced some sensitive info like real ip address etc Sat Jun 22 03:43:35 2019 - SNMPV1TRAP sent - 10.10.10.10:162… --- ## [Logstash only working for One filter while for other its not](https://discuss.elastic.co/t/logstash-only-working-for-one-filter-while-for-other-its-not/188089)
**Author:** [@Karn\_Gusain](https://discuss.elastic.co/u/Karn_Gusain)\ **Replies:** 0\ **Last updated:** [June 28, 2019, 4:23pm UTC](https://discuss.elastic.co/t/logstash-only-working-for-one-filter-while-for-other-its-not/188089 "2019-06-28T16:23:50Z")
I have below config almost running for 1 year and suddenly i got some open file issues on one of the cluster which i fixed after setting on the limits.conf in linux. I have below logstash pipeline which is working fine … --- ## [My logs are missing in elasticsearch and kibana](https://discuss.elastic.co/t/my-logs-are-missing-in-elasticsearch-and-kibana/187865)
**Author:** [@irobot678](https://discuss.elastic.co/u/irobot678)\ **Replies:** 3\ **Last updated:** [June 28, 2019, 4:18pm UTC](https://discuss.elastic.co/t/my-logs-are-missing-in-elasticsearch-and-kibana/187865 "2019-06-28T16:18:26Z")
Hi , my log file had exactly 1,573,249 lines/logs/documents, when i run logstash i can see only 1,541,787 documents in kibana . Around 31k logs/documents are missing and this is too bad . Can anyone have any solution t… --- ## [Logstash doesn't parse milliseconds with more than 4 digits](https://discuss.elastic.co/t/logstash-doesnt-parse-milliseconds-with-more-than-4-digits/188027)
**Author:** [@111167](https://discuss.elastic.co/u/111167)\ **Replies:** 2\ **Last updated:** [June 28, 2019, 4:05pm UTC](https://discuss.elastic.co/t/logstash-doesnt-parse-milliseconds-with-more-than-4-digits/188027 "2019-06-28T16:05:42Z")
Hello. I have logstash 6.6.1 version and conf file described below. input { file { start\_position =\> "beginning" path =\> "/logs/log.json" type =\> json } } filter { mutate { rename… --- ## [Flatten recursively a JSON to Logstash in Ruby](https://discuss.elastic.co/t/flatten-recursively-a-json-to-logstash-in-ruby/188076)
**Author:** [@Ayzoh](https://discuss.elastic.co/u/Ayzoh)\ **Replies:** 1\ **Last updated:** [June 28, 2019, 4:03pm UTC](https://discuss.elastic.co/t/flatten-recursively-a-json-to-logstash-in-ruby/188076 "2019-06-28T16:03:40Z")
Hello, I have the following JSON to parse in Logstash : "payload": \[ { "route":"/v1/things", "method":"PUT", "status":200, "content-type":"json", "body":{ "licencePlate"… --- ## [Logs on two lines](https://discuss.elastic.co/t/logs-on-two-lines/187915)
**Author:** [@kentatoi](https://discuss.elastic.co/u/kentatoi)\ **Replies:** 6\ **Last updated:** [June 28, 2019, 3:21pm UTC](https://discuss.elastic.co/t/logs-on-two-lines/187915 "2019-06-28T15:21:29Z")
Hi, I am trying to get logs whose are on two different lines : \[2019-06-18T06:57:32.032Z\] \[org.kie.api.internal.utils.tydhiks\] \[main\] \[141\] \[INFO \] Adding Service org.drools.core.concurrent.EEEeeF In my logstash I … --- ## [Rename or Change Available Fields](https://discuss.elastic.co/t/rename-or-change-available-fields/188035)
**Author:** [@Wilks](https://discuss.elastic.co/u/Wilks)\ **Replies:** 2\ **Last updated:** [June 28, 2019, 2:50pm UTC](https://discuss.elastic.co/t/rename-or-change-available-fields/188035 "2019-06-28T14:50:38Z")
Hi, How do I rename an available field? So for example we pull logs from GCP and use a json filter before outputting it to elasticsearch. The message field shows an event named "bytes\_sent" but in Kibana under Availabl… --- ## [Parsing message using Grok filter](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424)
**Author:** [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\ **Replies:** 12\ **Last updated:** [June 28, 2019, 2:21pm UTC](https://discuss.elastic.co/t/parsing-message-using-grok-filter/187424 "2019-06-28T14:21:26Z")
Hi, I am trying to drop a couple of words in a message. /test/data/user/log1/xyz How can I drop the first two words "test" and "data" only when the message starts with "/test" and store username=user and file=log1/xyz… --- ## [Usage of Nagios Logstash Plugins](https://discuss.elastic.co/t/usage-of-nagios-logstash-plugins/188051)
**Author:** [@anshul27jain](https://discuss.elastic.co/u/anshul27jain)\ **Replies:** 0\ **Last updated:** [June 28, 2019, 1:16pm UTC](https://discuss.elastic.co/t/usage-of-nagios-logstash-plugins/188051 "2019-06-28T13:16:09Z")
I checked multiple threads but was unable to found any good documentation of the usage of Logstash Plugins for Nagios. logstash-input-nagioscheck logstash-output-nagios logstash-output-nagiosnsca If any one can help … --- ## [Logstash Uknown error while parsing user agent data](https://discuss.elastic.co/t/logstash-uknown-error-while-parsing-user-agent-data/185334)
**Author:** [@ElkNovice](https://discuss.elastic.co/u/ElkNovice)\ **Replies:** 1\ **Last updated:** [June 28, 2019, 1:12pm UTC](https://discuss.elastic.co/t/logstash-uknown-error-while-parsing-user-agent-data/185334 "2019-06-28T13:12:49Z")
Using logstash 7.1, in logs I repeated see the below mentioned error messages. What can I do to get more information on the reason of these errors? Also, if it is known problem, can you guide on what could be the issue h… --- ## [Grokparsefailue on haproxy logs (but grok debugger result is OK in kibana...)](https://discuss.elastic.co/t/grokparsefailue-on-haproxy-logs-but-grok-debugger-result-is-ok-in-kibana/187854)
**Author:** [@zebu14](https://discuss.elastic.co/u/zebu14)\ **Replies:** 3\ **Last updated:** [June 28, 2019, 12:56pm UTC](https://discuss.elastic.co/t/grokparsefailue-on-haproxy-logs-but-grok-debugger-result-is-ok-in-kibana/187854 "2019-06-28T12:56:37Z")
Hello, I have read various similar topics about haproxy logs in logstash but I am still stuck with a grokparsefailure error on my Logstahs output. My configuration is : Logstash 7.1.1 HA-Proxy version 1.5.18 Haproxy… --- ## [Logstash with xpack and filebeat](https://discuss.elastic.co/t/logstash-with-xpack-and-filebeat/187533)
**Author:** [@markov](https://discuss.elastic.co/u/markov)\ **Replies:** 16\ **Last updated:** [June 28, 2019, 10:21am UTC](https://discuss.elastic.co/t/logstash-with-xpack-and-filebeat/187533 "2019-06-28T10:21:58Z")
hi, i am sending logs with filebeat to logstash and also i've installed xpack, i made the necessary configurations creation of the built in user editing the logstash.yml, uncomment the lines xpack.monitoring.enabled:… --- ## [Partial logs ingested by logstash causing \_jsonparsefailure](https://discuss.elastic.co/t/partial-logs-ingested-by-logstash-causing-jsonparsefailure/187999)
**Author:** [@nikhilo](https://discuss.elastic.co/u/nikhilo)\ **Replies:** 0\ **Last updated:** [June 28, 2019, 9:14am UTC](https://discuss.elastic.co/t/partial-logs-ingested-by-logstash-causing-jsonparsefailure/187999 "2019-06-28T09:14:50Z")
We're ingesting logs from a file in JSON format. Here's the config, input { file { 'codec' =\> 'json' 'path' =\> '/mnt/log/app/json.log\*' 'type' =\> 'erlang' 'sincedb\_path' =\> '/usr/share/logstash/dat… --- ## [Extract parameters from a message](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797)
**Author:** [@WarriorHarb](https://discuss.elastic.co/u/WarriorHarb)\ **Replies:** 11\ **Last updated:** [June 28, 2019, 9:13am UTC](https://discuss.elastic.co/t/extract-parameters-from-a-message/187797 "2019-06-28T09:13:49Z")
Hello, So i have a message in my log that contains parameters that i need to extract. This is an log line exemple : AgentID=50119&Tenant=2003&GroupID=15043&CountryCode=00&AreaCode=&DialedNumber=000000000 I need to … --- ## [Grok failure grok debugger ok](https://discuss.elastic.co/t/grok-failure-grok-debugger-ok/187885)
**Author:** [@Antoine.E](https://discuss.elastic.co/u/Antoine.E)\ **Replies:** 2\ **Last updated:** [June 27, 2019, 5:21pm UTC](https://discuss.elastic.co/t/grok-failure-grok-debugger-ok/187885 "2019-06-27T17:21:08Z")
Hello everybody, I'm currently parsing Cisco logs and I don't understand why my logs are marked as parsed failure but when I go and test it on https://grokdebug.herokuapp.com/ everything is parsing perfectly. My Logsta… --- ## [Logstash 7.2 JVM critical state management](https://discuss.elastic.co/t/logstash-7-2-jvm-critical-state-management/187938)
**Author:** [@Alexl](https://discuss.elastic.co/u/Alexl)\ **Replies:** 1\ **Last updated:** [June 28, 2019, 5:32am UTC](https://discuss.elastic.co/t/logstash-7-2-jvm-critical-state-management/187938 "2019-06-28T05:32:40Z")
Hello all! I have a lot of resources on powered machine but JVM is still running with high cpu and ram usage. Is there any hack like workers count, batch amount manipulation formula, etc to drop cpu and ram temperature … --- ## [Logstash hanging while split array (consume too much memory)](https://discuss.elastic.co/t/logstash-hanging-while-split-array-consume-too-much-memory/187726)
**Author:** [@taewoo\_Roh](https://discuss.elastic.co/u/taewoo_Roh)\ **Replies:** 2\ **Last updated:** [June 28, 2019, 4:43am UTC](https://discuss.elastic.co/t/logstash-hanging-while-split-array-consume-too-much-memory/187726 "2019-06-28T04:43:43Z")
Hello all. I have a problem while using logstash split filter. Logstash extract data from API using http\_polled input plug-in. and returned data contain array (format is json) I split array data using split filter p… --- ## [Remove extra double quote for a json object](https://discuss.elastic.co/t/remove-extra-double-quote-for-a-json-object/187942)
**Author:** [@singham](https://discuss.elastic.co/u/singham)\ **Replies:** 1\ **Last updated:** [June 27, 2019, 11:29pm UTC](https://discuss.elastic.co/t/remove-extra-double-quote-for-a-json-object/187942 "2019-06-27T23:29:11Z")
My JSON object coming from Kafka stream is as below """{"attack\_type\_etd" : ""DDOS"","business\_system\_id" : ""34521"","card\_number" : ""6937-8119-1476-3355"","client\_id" : "22","edge\_id" : 4,"email" : ""user7@client.com"… --- ## [Java Version 8 Update 211 installation with ELK 7.2 Logstash closes some warnings](https://discuss.elastic.co/t/java-version-8-update-211-installation-with-elk-7-2-logstash-closes-some-warnings/187939)
**Author:** [@Alexl](https://discuss.elastic.co/u/Alexl)\ **Replies:** 0\ **Last updated:** [June 27, 2019, 10:37pm UTC](https://discuss.elastic.co/t/java-version-8-update-211-installation-with-elk-7-2-logstash-closes-some-warnings/187939 "2019-06-27T22:37:04Z")
Hello all! I'm using elasticsearch/jdk/bin as system path component on Windows but Logstash puts several warnings to console on startup. Then, I have installed latest Java SE and run Logstash again and many warnings lef… --- ## [Logstash 7.2 JVM heap size limit restricted to 1gb](https://discuss.elastic.co/t/logstash-7-2-jvm-heap-size-limit-restricted-to-1gb/187937)
**Author:** [@Alexl](https://discuss.elastic.co/u/Alexl)\ **Replies:** 0\ **Last updated:** [June 27, 2019, 10:08pm UTC](https://discuss.elastic.co/t/logstash-7-2-jvm-heap-size-limit-restricted-to-1gb/187937 "2019-06-27T22:08:25Z")
Hello all! Cann't set both jvm.options -Xms and -Xmx limit to the value that higher than 1g. I can erase this settings. But with higher values service doesn't start on Windows 12gb RAM machine. Any suggestions? Thank y… --- ## [Logstash 7.2 input file plugin multiline codec without auto\_flush\_interval](https://discuss.elastic.co/t/logstash-7-2-input-file-plugin-multiline-codec-without-auto-flush-interval/187923)
**Author:** [@Alexl](https://discuss.elastic.co/u/Alexl)\ **Replies:** 0\ **Last updated:** [June 27, 2019, 8:43pm UTC](https://discuss.elastic.co/t/logstash-7-2-input-file-plugin-multiline-codec-without-auto-flush-interval/187923 "2019-06-27T20:43:35Z")
Hello all! Let's produce answers. What kind of behavior has Logstash got when auto\_flush\_interval not set? Any magic around it? Thank you. --- ## [Logstash 7.2 grok match filter plugin using multiple patterns](https://discuss.elastic.co/t/logstash-7-2-grok-match-filter-plugin-using-multiple-patterns/187922)
**Author:** [@Alexl](https://discuss.elastic.co/u/Alexl)\ **Replies:** 0\ **Last updated:** [June 27, 2019, 8:37pm UTC](https://discuss.elastic.co/t/logstash-7-2-grok-match-filter-plugin-using-multiple-patterns/187922 "2019-06-27T20:37:47Z")
Hello all! How does match grok work when we have \["message", "pattern1", "pattern2", ..., "patternN"\] compliance with the same names variables in patterns? Thank you. --- ## [Logstash 7.2 Filters and filter plugins execution order](https://discuss.elastic.co/t/logstash-7-2-filters-and-filter-plugins-execution-order/187908)
**Author:** [@Alexl](https://discuss.elastic.co/u/Alexl)\ **Replies:** 1\ **Last updated:** [June 27, 2019, 8:26pm UTC](https://discuss.elastic.co/t/logstash-7-2-filters-and-filter-plugins-execution-order/187908 "2019-06-27T20:26:56Z")
Hello all! Please, expand the concise documentation. What is the execution order which Logstash applies to many filters and their inner plugins? For example, I have this in logstash.conf: filter{ mutate1... grok1.… --- ## [Using split filter to break multi-line messages into separate (single-line) events](https://discuss.elastic.co/t/using-split-filter-to-break-multi-line-messages-into-separate-single-line-events/187894)
**Author:** [@aedades](https://discuss.elastic.co/u/aedades)\ **Replies:** 2\ **Last updated:** [June 27, 2019, 5:40pm UTC](https://discuss.elastic.co/t/using-split-filter-to-break-multi-line-messages-into-separate-single-line-events/187894 "2019-06-27T17:40:44Z")
An example message field I'm trying to split: "337 \<190\>1 2019-06-27T17:05:15.711968+00:00 host app worker.1 - request\_id=no\_request\_id organization\_id=no\_organization\_id user\_id=no\_user\_id hostname=no\_hostname 13988540… --- ## [Output only part of the message field using CSV filter](https://discuss.elastic.co/t/output-only-part-of-the-message-field-using-csv-filter/187863)
**Author:** [@Merril](https://discuss.elastic.co/u/Merril)\ **Replies:** 2\ **Last updated:** [June 27, 2019, 3:21pm UTC](https://discuss.elastic.co/t/output-only-part-of-the-message-field-using-csv-filter/187863 "2019-06-27T15:21:16Z")
Hi all, I am trying to index only certain columns from a Zeek(Bro) log. I managed to send all the columns within a message to ES and Kibana. here is an example line of the log file: 1561640400.525924 CGtlxS2Yk3ztKrfMK… --- ## [Create a document id using more than 2 fields](https://discuss.elastic.co/t/create-a-document-id-using-more-than-2-fields/187867)
**Author:** [@edster](https://discuss.elastic.co/u/edster)\ **Replies:** 3\ **Last updated:** [June 27, 2019, 3:09pm UTC](https://discuss.elastic.co/t/create-a-document-id-using-more-than-2-fields/187867 "2019-06-27T15:09:49Z")
How can one create a document id using multiple fields, more than just two? I'm trying to use more than 3 fields but it doesn't seem to accept it unless there is another way to do it. Right now i'm using this format: d… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=518) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=520)