# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=193 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 194 --- ## [Logstash tags entries as\_grokparsefailures but multiple online debuggers says otherwise](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753)
**Author:** [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Replies:** 11\ **Last updated:** [September 23, 2021, 2:56pm UTC](https://discuss.elastic.co/t/logstash-tags-entries-as-grokparsefailures-but-multiple-online-debuggers-says-otherwise/283753 "2021-09-23T14:56:19Z")
Hi All, I've a very strange question that I hope to explain eloquently. I'm dealing with FW logs. Already wrote a grok parser that works like a charm (prefer not to provided it here, let me know if it's going to be need… --- ## [How can I parse nested JSON object with arrays and dict into Elasticsearch using Logstash](https://discuss.elastic.co/t/how-can-i-parse-nested-json-object-with-arrays-and-dict-into-elasticsearch-using-logstash/284971)
**Author:** [@shaikadil95](https://discuss.elastic.co/u/shaikadil95)\ **Replies:** 1\ **Last updated:** [September 23, 2021, 2:43pm UTC](https://discuss.elastic.co/t/how-can-i-parse-nested-json-object-with-arrays-and-dict-into-elasticsearch-using-logstash/284971 "2021-09-23T14:43:14Z")
I have a nested JSON object below. { "data": { "testChannels": \[ { "name": "TC1", "Time": "2021-08-31 12:02:51.103188", "info": { "user": "johan", "epic": "Epic" … --- ## [Wrong events order if time difference is less than 1ms](https://discuss.elastic.co/t/wrong-events-order-if-time-difference-is-less-than-1ms/284983)
**Author:** [@alexandrpaliy](https://discuss.elastic.co/u/alexandrpaliy)\ **Replies:** 1\ **Last updated:** [September 23, 2021, 2:10pm UTC](https://discuss.elastic.co/t/wrong-events-order-if-time-difference-is-less-than-1ms/284983 "2021-09-23T14:10:47Z")
Hi. I use version 7.13.4 of all stack (ELK + filebeats). Filebeat grabs events (auth.log entries in this case) on remote host, then passes them to logstash to filter/modify data, then events go to Elasticsearch. Talkin… --- ## [Logstash for z/OS](https://discuss.elastic.co/t/logstash-for-z-os/284916)
**Author:** [@Akhil2](https://discuss.elastic.co/u/Akhil2)\ **Replies:** 2\ **Last updated:** [September 23, 2021, 1:57pm UTC](https://discuss.elastic.co/t/logstash-for-z-os/284916 "2021-09-23T13:57:44Z")
Hello there, Hope you are doing well! I have a few questions regarding Log monitoring on Mainframe through Logstash. My understanding here is extract and transform logs into JSON through JCL workbench and send it to th… --- ## [Logstash SNMP input plugin: no data extracted](https://discuss.elastic.co/t/logstash-snmp-input-plugin-no-data-extracted/284982)
**Author:** [@daniele.saccon](https://discuss.elastic.co/u/daniele.saccon)\ **Replies:** 0\ **Last updated:** [September 23, 2021, 9:31am UTC](https://discuss.elastic.co/t/logstash-snmp-input-plugin-no-data-extracted/284982 "2021-09-23T09:31:06Z")
Hi, I am using the SNMP input plugin with the following configurations: input { snmp { walk =\> \["1.3.6.1.2.1.1"\] hosts =\> \[{host =\> "udp:XXXXXXXX/161" community =\> "public" version =\> "2c" retries =\> 2 time… --- ## [Logstash 7.13 and Elasticsearch 7.9.2 compatibility](https://discuss.elastic.co/t/logstash-7-13-and-elasticsearch-7-9-2-compatibility/284962)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 2\ **Last updated:** [September 23, 2021, 9:15am UTC](https://discuss.elastic.co/t/logstash-7-13-and-elasticsearch-7-9-2-compatibility/284962 "2021-09-23T09:15:15Z")
I do understand that we need to have the same version of components for elastic stack, but would like to know whether 7.13 version of logstash can run with 7.9.2 version of Elasticsearch. --- ## [Grok parsing and skipping lines](https://discuss.elastic.co/t/grok-parsing-and-skipping-lines/284864)
**Author:** [@Mira\_9](https://discuss.elastic.co/u/Mira_9)\ **Replies:** 2\ **Last updated:** [September 23, 2021, 8:38am UTC](https://discuss.elastic.co/t/grok-parsing-and-skipping-lines/284864 "2021-09-23T08:38:42Z")
Hey! i have to parse the output of a command linux,extract specific fields and skip some lines.I'm trying but its not working. SSID: "#SOMB-875-LY5C" Mode: Managed RSSI: 1 dBm SNR: 0 dB noise: -89 dBm Chann… --- ## [Elastic cluster is unreachable - Logstash with docker](https://discuss.elastic.co/t/elastic-cluster-is-unreachable-logstash-with-docker/284964)
**Author:** [@Rashmika\_Gamage](https://discuss.elastic.co/u/Rashmika_Gamage)\ **Replies:** 0\ **Last updated:** [September 23, 2021, 7:02am UTC](https://discuss.elastic.co/t/elastic-cluster-is-unreachable-logstash-with-docker/284964 "2021-09-23T07:02:25Z")
I have created a ELK cluster with 3 elastic nodes with docker compose. I am getting an error from logstash. I have been trying with several ways. Appreciate you help. My docker-compose.yml version: '2.2' services: es… --- ## [Field reference from \_source for conditional output](https://discuss.elastic.co/t/field-reference-from-source-for-conditional-output/284956)
**Author:** [@alfianaf](https://discuss.elastic.co/u/alfianaf)\ **Replies:** 4\ **Last updated:** [September 23, 2021, 4:32am UTC](https://discuss.elastic.co/t/field-reference-from-source-for-conditional-output/284956 "2021-09-23T04:32:31Z")
hello, I'm new to logstash conditional, I want to make different index output based on some field reference on my logstash here is my index example : { "\_index": "iris-new-2021.09", "\_type": "\_doc", "\_id": "EKS5E… --- ## [How to read each array item into a map of key value pair?](https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769)
**Author:** [@Indigo\_Star](https://discuss.elastic.co/u/Indigo_Star)\ **Replies:** 7\ **Last updated:** [September 23, 2021, 3:34am UTC](https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769 "2021-09-23T03:34:45Z")
I have a multiline message field, msg= "CPU=9% TotalMem=12288 MB FreeMem=3895 MB AMQMemoryPercentUsage=0% n0-kkk size=0 inFlight=0 enqRate=0 deqRate=0 n1-xxx size=1 inFlight=1 enqRate=13 deqRate=2 n2-yyyy-oo si… --- ## [How to write multiline codec for python stacktrace](https://discuss.elastic.co/t/how-to-write-multiline-codec-for-python-stacktrace/284936)
**Author:** [@Chandrakant\_Naik](https://discuss.elastic.co/u/Chandrakant_Naik)\ **Replies:** 3\ **Last updated:** [September 22, 2021, 8:50pm UTC](https://discuss.elastic.co/t/how-to-write-multiline-codec-for-python-stacktrace/284936 "2021-09-22T20:50:11Z")
below is how my log is captured \[2021-09-17 14:24:16,527\] \[ERROR\] \[app\]\[MainThread\] \[5d9b63a9-dc4d-4756-af87-eba4fb275584\] \[log\_exception:1440\] : Exception on /download\_request\_data \[GET\] Traceback (most recent call las… --- ## [How to use .conut to exclude a character from the .length count?](https://discuss.elastic.co/t/how-to-use-conut-to-exclude-a-character-from-the-length-count/284938)
**Author:** [@Rauly\_Koto](https://discuss.elastic.co/u/Rauly_Koto)\ **Replies:** 1\ **Last updated:** [September 22, 2021, 8:21pm UTC](https://discuss.elastic.co/t/how-to-use-conut-to-exclude-a-character-from-the-length-count/284938 "2021-09-22T20:21:24Z")
Currently it does the account but does not delete the "/". ruby { code =\> ' event.set("execution\_length\_input", event.get("\[execution\]\[input\]").to\_s.length - ("\[execution\]\[input\]… --- ## [Extracting string from log path](https://discuss.elastic.co/t/extracting-string-from-log-path/284748)
**Author:** [@Bkumar](https://discuss.elastic.co/u/Bkumar)\ **Replies:** 9\ **Last updated:** [September 22, 2021, 6:30pm UTC](https://discuss.elastic.co/t/extracting-string-from-log-path/284748 "2021-09-22T18:30:59Z")
Hi, As per requirement, i can't put the tag to filebeat to filter log path at logstash end. So only option left with me to filter the log path on the basis of string. Here is the sample log path- /opt/tomcat/instances… --- ## [Questions about multiple pipelines and whether events are processed by all or only one](https://discuss.elastic.co/t/questions-about-multiple-pipelines-and-whether-events-are-processed-by-all-or-only-one/284907)
**Author:** [@nsouth](https://discuss.elastic.co/u/nsouth)\ **Replies:** 2\ **Last updated:** [September 22, 2021, 5:48pm UTC](https://discuss.elastic.co/t/questions-about-multiple-pipelines-and-whether-events-are-processed-by-all-or-only-one/284907 "2021-09-22T17:48:44Z")
I would like help understanding how multiple pipelines defined in pipelines.yml work together. I originally had a single pipeline defined by commands-pipeline.conf file. I then introduced a new pipeline in commands-raw-p… --- ## [Slow performance on big unstructured file](https://discuss.elastic.co/t/slow-performance-on-big-unstructured-file/284567)
**Author:** [@nestor1](https://discuss.elastic.co/u/nestor1)\ **Replies:** 9\ **Last updated:** [September 22, 2021, 5:31pm UTC](https://discuss.elastic.co/t/slow-performance-on-big-unstructured-file/284567 "2021-09-22T17:31:39Z")
I have couple of millions small files, but few of them are extremely big (comparing to small ones), around 200-400 GB. Pipeline is set to fetch data from each file (1 file -\> 1 event in Elasticsearch). It's working ok, b… --- ## [While reindexing a index, I have empty date fields, date parse error?](https://discuss.elastic.co/t/while-reindexing-a-index-i-have-empty-date-fields-date-parse-error/284868)
**Author:** [@padamrai](https://discuss.elastic.co/u/padamrai)\ **Replies:** 0\ **Last updated:** [September 22, 2021, 12:42pm UTC](https://discuss.elastic.co/t/while-reindexing-a-index-i-have-empty-date-fields-date-parse-error/284868 "2021-09-22T12:42:11Z")
Hello Everyone, While reindexing the index from one server to another, I have three date fields except timestamps. Some fields are empty. Due to empty fields I am getting a date parser exception I want it ignore empty d… --- ## [How to logstash if field equals value assign other field value?](https://discuss.elastic.co/t/how-to-logstash-if-field-equals-value-assign-other-field-value/284857)
**Author:** [@Max\_Wang](https://discuss.elastic.co/u/Max_Wang)\ **Replies:** 2\ **Last updated:** [September 22, 2021, 12:05pm UTC](https://discuss.elastic.co/t/how-to-logstash-if-field-equals-value-assign-other-field-value/284857 "2021-09-22T12:05:07Z")
I have a logstash conf field A : Value A field B : Value B ——————————————————————————— I would like to become The value of field A is equal to N/A, i will use the value of field B, as follows: field A : N/A -\> … --- ## [如何去掉kibana显示中json中的\\](https://discuss.elastic.co/t/kibana-json/284820)
**Author:** [@chengxiaobai](https://discuss.elastic.co/u/chengxiaobai)\ **Replies:** 0\ **Last updated:** [September 22, 2021, 8:58am UTC](https://discuss.elastic.co/t/kibana-json/284820 "2021-09-22T08:58:36Z")
图1 为kibana中json显示内容。 图2 为日志原始格式。 目前想在kibana中查看message的信息（需要去掉\\）,请问怎么处理。 我尝试过json反序列化，并没有作用。请指教。。 --- ## [error\_type=\>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=\>"Elasticsearch Unreachable:](https://discuss.elastic.co/t/error-type-logstash-hostunreachableerror-error-elasticsearch-unreachable/284803)
**Author:** [@padamrai](https://discuss.elastic.co/u/padamrai)\ **Replies:** 2\ **Last updated:** [September 22, 2021, 7:05am UTC](https://discuss.elastic.co/t/error-type-logstash-hostunreachableerror-error-elasticsearch-unreachable/284803 "2021-09-22T07:05:29Z")
I am using the docker configuration on the server. I am not that much familiar with docker. Before docker, it is working fine but now it is not working and throwing an error. elasticsearch - Attempted to resurrect conne… --- ## [Translate doesnt work when key in yml dictionary is a number](https://discuss.elastic.co/t/translate-doesnt-work-when-key-in-yml-dictionary-is-a-number/284767)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 5\ **Last updated:** [September 21, 2021, 10:53pm UTC](https://discuss.elastic.co/t/translate-doesnt-work-when-key-in-yml-dictionary-is-a-number/284767 "2021-09-21T22:53:09Z")
Hi, I have this yml dictionary 1967345056 : "VISACI nodata IIS" and this is the translate: translate { field =\> "\[monitorgroup\]" destination =\> "\[nuevo\_campo\]" dictionary\_path =\> "/etc/logs… --- ## [Extract a substring and assign to new field](https://discuss.elastic.co/t/extract-a-substring-and-assign-to-new-field/284692)
**Author:** [@Andrea\_Bozzano](https://discuss.elastic.co/u/Andrea_Bozzano)\ **Replies:** 2\ **Last updated:** [September 21, 2021, 3:10pm UTC](https://discuss.elastic.co/t/extract-a-substring-and-assign-to-new-field/284692 "2021-09-21T15:10:18Z")
Hi everyone! Hope everyone's good. in my application, i'm sending a JSON wich has a string field always made of 16 char For example: XXXYYY12Y00K111H Is there a fine way, or do you have any particular hint to extract… --- ## [Needs to add few field from one log to another log](https://discuss.elastic.co/t/needs-to-add-few-field-from-one-log-to-another-log/284600)
**Author:** [@rajvel](https://discuss.elastic.co/u/rajvel)\ **Replies:** 2\ **Last updated:** [September 21, 2021, 2:57pm UTC](https://discuss.elastic.co/t/needs-to-add-few-field-from-one-log-to-another-log/284600 "2021-09-21T14:57:29Z")
Hi Team, We have a ansible log, One is stream and another one is job status. Now i want to add few field from one log to another, then only i can able to create a table for my requirements. --- ## [Keep a concrete character regular expressioon](https://discuss.elastic.co/t/keep-a-concrete-character-regular-expressioon/284696)
**Author:** [@rodri.gz](https://discuss.elastic.co/u/rodri.gz)\ **Replies:** 2\ **Last updated:** [September 21, 2021, 1:44pm UTC](https://discuss.elastic.co/t/keep-a-concrete-character-regular-expressioon/284696 "2021-09-21T13:44:54Z")
i need to keep only two characters of a field but i dont know how to do it. I would like to keep the second and third character of the server name, which indicates the country code, for example T52EDFS011 (52). i'm new… --- ## [Kv filter usage](https://discuss.elastic.co/t/kv-filter-usage/284718)
**Author:** [@PaoloZhao](https://discuss.elastic.co/u/PaoloZhao)\ **Replies:** 2\ **Last updated:** [September 21, 2021, 1:40pm UTC](https://discuss.elastic.co/t/kv-filter-usage/284718 "2021-09-21T13:40:27Z")
Hi guys! I am a fresh fish. I have some logs like blow. 15:08:16.2104 Info {"message":"BlankProcess execution started","level":"Information","logType":"Default","timeStamp":"2021-09-21T15:08:16.2015207+08:00","fingerp… --- ## [Add existing fields to my logstash](https://discuss.elastic.co/t/add-existing-fields-to-my-logstash/284626)
**Author:** [@rodri.gz](https://discuss.elastic.co/u/rodri.gz)\ **Replies:** 8\ **Last updated:** [September 21, 2021, 12:24pm UTC](https://discuss.elastic.co/t/add-existing-fields-to-my-logstash/284626 "2021-09-21T12:24:49Z")
hello ! I would like to be able to add existing fields in an already stored index and to be able to work with them in logstash. I understand that there are enrichment policies but I would like not to have to use kibana… --- ## [Logstash changing Object events using ruby not working](https://discuss.elastic.co/t/logstash-changing-object-events-using-ruby-not-working/284722)
**Author:** [@antonisnyc94](https://discuss.elastic.co/u/antonisnyc94)\ **Replies:** 0\ **Last updated:** [September 21, 2021, 12:26pm UTC](https://discuss.elastic.co/t/logstash-changing-object-events-using-ruby-not-working/284722 "2021-09-21T12:26:53Z")
Hello, I am having the below logstash filter ruby code to change the name of an event with Object type but im still getting an error. Am i doing something wrong? if \[body\]\[json\]\[attr\]\[config\] { … --- ## [Docker compose error - cannot connect logstash to elastic search](https://discuss.elastic.co/t/docker-compose-error-cannot-connect-logstash-to-elastic-search/284685)
**Author:** [@Rashmika\_Gamage](https://discuss.elastic.co/u/Rashmika_Gamage)\ **Replies:** 3\ **Last updated:** [September 21, 2021, 12:13pm UTC](https://discuss.elastic.co/t/docker-compose-error-cannot-connect-logstash-to-elastic-search/284685 "2021-09-21T12:13:11Z")
I am trying to create a elastic search cluster with logstash and kibana using docker compose. Have been trying for a long time but i am getting an error. My log lines are kib01 | {"type":"log","@timestamp":"2021-… --- ## [Logstash duration management](https://discuss.elastic.co/t/logstash-duration-management/284716)
**Author:** [@simoamarca](https://discuss.elastic.co/u/simoamarca)\ **Replies:** 0\ **Last updated:** [September 21, 2021, 11:57am UTC](https://discuss.elastic.co/t/logstash-duration-management/284716 "2021-09-21T11:57:54Z")
Halloooo, I would like to ask you how to manage, with rubydebug in logstash, durations with format like 07:22:27. I wrote a script.. ruby { # code =\> " # s = event.get('Logon du… --- ## [Moving fields to root](https://discuss.elastic.co/t/moving-fields-to-root/283663)
**Author:** [@vijay\_kaali](https://discuss.elastic.co/u/vijay_kaali)\ **Replies:** 1\ **Last updated:** [September 21, 2021, 5:14am UTC](https://discuss.elastic.co/t/moving-fields-to-root/283663 "2021-09-21T05:14:33Z")
i have metricbeat installed and for desupported version RHEL 4 we collect it via custom script we ingest them with type name devbeat and in elk it is showing all fields prefixed with devbeat. how to avoid prefixing … --- ## [Understanding HTTP\_Poller](https://discuss.elastic.co/t/understanding-http-poller/284639)
**Author:** [@tofubeats](https://discuss.elastic.co/u/tofubeats)\ **Replies:** 1\ **Last updated:** [September 21, 2021, 1:13am UTC](https://discuss.elastic.co/t/understanding-http-poller/284639 "2021-09-21T01:13:06Z")
Hi, if I were to use HTTP\_Poller plugin on a logstash instance to query an ES API on another machine, would it hit the transport interface or the HTTP interface? Thank you! [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=192) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=194)