# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=272 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 273 --- ## [My ruby filter doesn't work](https://discuss.elastic.co/t/my-ruby-filter-doesnt-work/258439)
**Author:** [@olegans](https://discuss.elastic.co/u/olegans)\ **Replies:** 6\ **Last updated:** [December 11, 2020, 7:01pm UTC](https://discuss.elastic.co/t/my-ruby-filter-doesnt-work/258439 "2020-12-11T19:01:01Z")
I am trying to use a ruby ​​filter to process incoming json that I receive in the logs. At the beginning, using a grok filter, I receive json, then I try to process it in a ruby ​​filter grok { match =\> { "l… --- ## [Cluster\_block\_exception blocks all ingestion](https://discuss.elastic.co/t/cluster-block-exception-blocks-all-ingestion/258338)
**Author:** [@elk\_follower](https://discuss.elastic.co/u/elk_follower)\ **Replies:** 6\ **Last updated:** [December 11, 2020, 5:14pm UTC](https://discuss.elastic.co/t/cluster-block-exception-blocks-all-ingestion/258338 "2020-12-11T17:14:41Z")
Hello, We have a ELK 7.3.2 cluster with 3 nodes and almost 3 TB disk free. Also have Index lifecycle policy which force merge all indices older than 7 days Free Disk: ILP: We deployed filebeat to ingest some ol… --- ## [JDBC Input Error after upgrading to 7.10.0 on Windows](https://discuss.elastic.co/t/jdbc-input-error-after-upgrading-to-7-10-0-on-windows/256902)
**Author:** [@stevedearl](https://discuss.elastic.co/u/stevedearl)\ **Replies:** 1\ **Last updated:** [December 11, 2020, 4:41pm UTC](https://discuss.elastic.co/t/jdbc-input-error-after-upgrading-to-7-10-0-on-windows/256902 "2020-12-11T16:41:19Z")
Hi All, I updated my Windows DEV environment to 7.10.0 from 7.9.2 earlier today. When I try to run my existing Logstash configurations in 7.10.0 (which worked fine under 7.9.2) I see the following error: \[2020-11-27T12… --- ## [2 or more nested array without duplicates](https://discuss.elastic.co/t/2-or-more-nested-array-without-duplicates/258391)
**Author:** [@pyrovoice](https://discuss.elastic.co/u/pyrovoice)\ **Replies:** 1\ **Last updated:** [December 11, 2020, 9:45am UTC](https://discuss.elastic.co/t/2-or-more-nested-array-without-duplicates/258391 "2020-12-11T09:45:07Z")
I am building my ES indexes from data from our MySQL databases, using a jdbc plugin. One of my columns has multiple many-to-many parameters, that I need to import into the index (as ES does not manage many-to-many well)… --- ## [CSV Movement after loading into Elastic Search using Logstash](https://discuss.elastic.co/t/csv-movement-after-loading-into-elastic-search-using-logstash/258159)
**Author:** [@mdwazirhassan](https://discuss.elastic.co/u/mdwazirhassan)\ **Replies:** 4\ **Last updated:** [December 11, 2020, 4:12am UTC](https://discuss.elastic.co/t/csv-movement-after-loading-into-elastic-search-using-logstash/258159 "2020-12-11T04:12:30Z")
Hi, I am new with Elastic and Kibana. Senario is as below: We have a folder called "livedata" In this(livedata) folder on every second or alternate new csv file is arraived from ftp. I have created a config logstash… --- ## [Current Logs Output Is a Single Line - How do I make multi-newlines at specific points](https://discuss.elastic.co/t/current-logs-output-is-a-single-line-how-do-i-make-multi-newlines-at-specific-points/258361)
**Author:** [@test\_tester](https://discuss.elastic.co/u/test_tester)\ **Replies:** 0\ **Last updated:** [December 11, 2020, 3:54am UTC](https://discuss.elastic.co/t/current-logs-output-is-a-single-line-how-do-i-make-multi-newlines-at-specific-points/258361 "2020-12-11T03:54:36Z")
Good Day All, I have a problem whereby my current logs are outputting in a single line. The backstory is that our application is receiving messages from another party and within those messages, we have /n, so when we o… --- ## [How to transfer a dynamic parameter from filter of logstash](https://discuss.elastic.co/t/how-to-transfer-a-dynamic-parameter-from-filter-of-logstash/258358)
**Author:** [@Br1ckman](https://discuss.elastic.co/u/Br1ckman)\ **Replies:** 0\ **Last updated:** [December 11, 2020, 3:26am UTC](https://discuss.elastic.co/t/how-to-transfer-a-dynamic-parameter-from-filter-of-logstash/258358 "2020-12-11T03:26:35Z")
How to transfer a dynamic parameter from the init module of filter to the .rb file of path =\>"XX.rb"？ --- ## [Is there any option or way to turn off stdout related options?](https://discuss.elastic.co/t/is-there-any-option-or-way-to-turn-off-stdout-related-options/258266)
**Author:** [@shcho](https://discuss.elastic.co/u/shcho)\ **Replies:** 2\ **Last updated:** [December 11, 2020, 3:12am UTC](https://discuss.elastic.co/t/is-there-any-option-or-way-to-turn-off-stdout-related-options/258266 "2020-12-11T03:12:45Z")
When using the logstash elasticsearch output , it can be seen that stdout logging is displayed even without additional stdout setting. Is there any option or way to turn off stdout related options? --- ## [How to parse logstash's data to a json array?](https://discuss.elastic.co/t/how-to-parse-logstashs-data-to-a-json-array/258274)
**Author:** [@iooi](https://discuss.elastic.co/u/iooi)\ **Replies:** 2\ **Last updated:** [December 11, 2020, 2:08am UTC](https://discuss.elastic.co/t/how-to-parse-logstashs-data-to-a-json-array/258274 "2020-12-11T02:08:38Z")
How to parse logstash's data to a json array? Using filebeat sending a file to logstash. The file includes 2 line of records: {"request\_id": "m2ee22d045f6c5ce07fe43dbdaea1de0","method": "GET","status": "304","forwarde… --- ## [Error in elasticsearch filter of logstash: Failed to query elasticsearch for previous event](https://discuss.elastic.co/t/error-in-elasticsearch-filter-of-logstash-failed-to-query-elasticsearch-for-previous-event/256686)
**Author:** [@Vita\_Rosenberg](https://discuss.elastic.co/u/Vita_Rosenberg)\ **Replies:** 5\ **Last updated:** [December 11, 2020, 1:11am UTC](https://discuss.elastic.co/t/error-in-elasticsearch-filter-of-logstash-failed-to-query-elasticsearch-for-previous-event/256686 "2020-12-11T01:11:20Z")
Hello. I have an index in kibana called poc1\*, it contains fields of machineId and timestamp I need to get from that index a maximum value of timestamp to a new index for each machineId (which is deviceId in sql). … --- ## [Find time difference between series of events with uniqueid](https://discuss.elastic.co/t/find-time-difference-between-series-of-events-with-uniqueid/257911)
**Author:** [@GokulD](https://discuss.elastic.co/u/GokulD)\ **Replies:** 5\ **Last updated:** [December 10, 2020, 8:45pm UTC](https://discuss.elastic.co/t/find-time-difference-between-series-of-events-with-uniqueid/257911 "2020-12-10T20:45:09Z")
Hi , I have requirement where I have series of events that has a unique trace id. I want to find and take the timestamp from the first and last event of this id and find the difference. Since I dont have any start and … --- ## [Multiple values in a string value in logstash?](https://discuss.elastic.co/t/multiple-values-in-a-string-value-in-logstash/258316)
**Author:** [@GetYourSh1tTogether](https://discuss.elastic.co/u/GetYourSh1tTogether)\ **Replies:** 2\ **Last updated:** [December 10, 2020, 7:46pm UTC](https://discuss.elastic.co/t/multiple-values-in-a-string-value-in-logstash/258316 "2020-12-10T19:46:10Z")
I am trying to set up this plugin by itself it works great and imports everything in the bucket. However, there are tons of subdirectories that i would like to prevent logstash from reading. I found this field "exclude\_p… --- ## [How to send multiple headers in Logstash http output plugin?](https://discuss.elastic.co/t/how-to-send-multiple-headers-in-logstash-http-output-plugin/258318)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 2\ **Last updated:** [December 10, 2020, 6:45pm UTC](https://discuss.elastic.co/t/how-to-send-multiple-headers-in-logstash-http-output-plugin/258318 "2020-12-10T18:45:07Z")
Hi, I am using Logstash 7.8.0. I am trying to send some message to an http endpoint using POST. I would like to add 3 header parameters and I would like to know how to do that. Its mentioned in the docs that its a hash h… --- ## [Adding field creates an array not a static string](https://discuss.elastic.co/t/adding-field-creates-an-array-not-a-static-string/258149)
**Author:** [@jknott](https://discuss.elastic.co/u/jknott)\ **Replies:** 7\ **Last updated:** [December 10, 2020, 6:23pm UTC](https://discuss.elastic.co/t/adding-field-creates-an-array-not-a-static-string/258149 "2020-12-10T18:23:52Z")
When I add this in my filter it adds it but my other pipelines also add a field of the same and the values turn into an array instead of a static value per log. mutate { add\_field =\> { "service\_name"… --- ## [Required logstash patterns to extract JSON format log](https://discuss.elastic.co/t/required-logstash-patterns-to-extract-json-format-log/255516)
**Author:** [@nikhilesh](https://discuss.elastic.co/u/nikhilesh)\ **Replies:** 3\ **Last updated:** [December 10, 2020, 5:31pm UTC](https://discuss.elastic.co/t/required-logstash-patterns-to-extract-json-format-log/255516 "2020-12-10T17:31:40Z")
Hi All, Below is the log message which is in JSON format, would like to extract couple of keys from the below log ( say ex: x-forwarded-for, host, etc..). can any please help me to write logstash filters to extract the … --- ## [Kafka input plugin](https://discuss.elastic.co/t/kafka-input-plugin/258303)
**Author:** [@Samuel\_Delepiere](https://discuss.elastic.co/u/Samuel_Delepiere)\ **Replies:** 1\ **Last updated:** [December 10, 2020, 5:17pm UTC](https://discuss.elastic.co/t/kafka-input-plugin/258303 "2020-12-10T17:17:54Z")
The documentation does not really make sense when it comes to poll\_timeout\_ms and fetch\_max\_wait\_ms. According to the documentation: fetch\_max\_wait\_ms has a default value of 500ms poll\_timeout\_ms has a default value o… --- ## [Why collect the same log when I set different port?](https://discuss.elastic.co/t/why-collect-the-same-log-when-i-set-different-port/258211)
**Author:** [@111418](https://discuss.elastic.co/u/111418)\ **Replies:** 3\ **Last updated:** [December 10, 2020, 4:29pm UTC](https://discuss.elastic.co/t/why-collect-the-same-log-when-i-set-different-port/258211 "2020-12-10T16:29:08Z")
I have two conf file in /etc/logstash/conf.d/. One is named "nas\_ftp.conf" and include below setting:\`\`\` input { tcp { port =\> 10515 } } filter { } output { elasticsearch { hosts … --- ## [Running filebeat modules alongside a custom pipeline](https://discuss.elastic.co/t/running-filebeat-modules-alongside-a-custom-pipeline/258272)
**Author:** [@pbatley](https://discuss.elastic.co/u/pbatley)\ **Replies:** 4\ **Last updated:** [December 10, 2020, 3:16pm UTC](https://discuss.elastic.co/t/running-filebeat-modules-alongside-a-custom-pipeline/258272 "2020-12-10T15:16:39Z")
I am completely new to Elastic so apologies if this is trivial. I have managed to set up the ELK stack (self managed, all on same host machine) and have got filebeat's apache module enable sending logs to logstash. I us… --- ## [No configuration found in the configured sources](https://discuss.elastic.co/t/no-configuration-found-in-the-configured-sources/258029)
**Author:** [@djb000m](https://discuss.elastic.co/u/djb000m)\ **Replies:** 3\ **Last updated:** [December 10, 2020, 2:28pm UTC](https://discuss.elastic.co/t/no-configuration-found-in-the-configured-sources/258029 "2020-12-10T14:28:27Z")
Hi, I am running the standard 7.10.0 Logstash docker container and I want to use it to do its thing and send data on to an ECE elastic deployment. I'm running into an odd error that I cannot find a solution for. When I… --- ## [How to extract field name which has space in them using xpath](https://discuss.elastic.co/t/how-to-extract-field-name-which-has-space-in-them-using-xpath/258201)
**Author:** [@kamalbeg](https://discuss.elastic.co/u/kamalbeg)\ **Replies:** 1\ **Last updated:** [December 10, 2020, 12:53am UTC](https://discuss.elastic.co/t/how-to-extract-field-name-which-has-space-in-them-using-xpath/258201 "2020-12-10T00:53:51Z")
I have following xml code xml { source =\> "message" store\_xml =\> "false" remove\_namespaces =\> "true" xpath =\> \[ '/E2ETraceEvent/System/Computer/text()', "\[grc\]\[System\]\[Computer\]", … --- ## [How to apply if condition on the field of json log](https://discuss.elastic.co/t/how-to-apply-if-condition-on-the-field-of-json-log/257931)
**Author:** [@dharanesh](https://discuss.elastic.co/u/dharanesh)\ **Replies:** 1\ **Last updated:** [December 10, 2020, 12:45am UTC](https://discuss.elastic.co/t/how-to-apply-if-condition-on-the-field-of-json-log/257931 "2020-12-10T00:45:49Z")
I have a log which contains json, i want to apply if condition if log line contains "hai" i need to push data to elasticsearch @REQ.WS: {"message":"hai","Id":"123"} --- ## [Passing parameters to Java filter plugin during runtime](https://discuss.elastic.co/t/passing-parameters-to-java-filter-plugin-during-runtime/258062)
**Author:** [@lakechat](https://discuss.elastic.co/u/lakechat)\ **Replies:** 5\ **Last updated:** [December 9, 2020, 10:53pm UTC](https://discuss.elastic.co/t/passing-parameters-to-java-filter-plugin-during-runtime/258062 "2020-12-09T22:53:34Z")
The Java filter plugin example (https://www.elastic.co/guide/en/logstash/current/java-filter-plugin.html) was called without any inputs because everything is fixed during compilation time. This makes the plugin less gene… --- ## [New to ELK Stack, Logstash Will Not Start - Please Help](https://discuss.elastic.co/t/new-to-elk-stack-logstash-will-not-start-please-help/258172)
**Author:** [@NJMitchell](https://discuss.elastic.co/u/NJMitchell)\ **Replies:** 11\ **Last updated:** [December 9, 2020, 10:52pm UTC](https://discuss.elastic.co/t/new-to-elk-stack-logstash-will-not-start-please-help/258172 "2020-12-09T22:52:47Z")
Hello elastic community, I'm very new to the ELK stack and am working on getting a production environment up and running. So far I have a four node Elasticsearch cluster running with Kibana. Everything there is working … --- ## [Can not create an index](https://discuss.elastic.co/t/can-not-create-an-index/258055)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 4\ **Last updated:** [December 9, 2020, 5:42pm UTC](https://discuss.elastic.co/t/can-not-create-an-index/258055 "2020-12-09T17:42:08Z")
Hi. I am trying to create a new index in Kibana. My logstash conf file looks like below: if \[type\] == "uatelectron\_appLog" { mutate { split =\> \["message", "|"\] … --- ## [Rotate and remove old Logstash output logs](https://discuss.elastic.co/t/rotate-and-remove-old-logstash-output-logs/256803)
**Author:** [@denisdM](https://discuss.elastic.co/u/denisdM)\ **Replies:** 7\ **Last updated:** [December 9, 2020, 5:09pm UTC](https://discuss.elastic.co/t/rotate-and-remove-old-logstash-output-logs/256803 "2020-12-09T17:09:20Z")
Hello, I've got a Logstash installation (centos 7.6, Logstash 7.2.0) which work great but produce logs and never delete them. I've never changed any default configuration so the /etc/logstash/log4j2.properties files co… --- ## [Set a nested field with a variable](https://discuss.elastic.co/t/set-a-nested-field-with-a-variable/258153)
**Author:** [@dwatbmc](https://discuss.elastic.co/u/dwatbmc)\ **Replies:** 1\ **Last updated:** [December 9, 2020, 4:00pm UTC](https://discuss.elastic.co/t/set-a-nested-field-with-a-variable/258153 "2020-12-09T16:00:29Z")
Can't find a way to do this with Mutate, so I am attempting it in Ruby, but still failing: I have two fields: \[log\]\[group\] = "panos" \[log\]\[type\] = "threat" Based on these two field values, I want to copy the JSON Obj… --- ## [Logstash - Output - Http - Message](https://discuss.elastic.co/t/logstash-output-http-message/258150)
**Author:** [@rguerrero](https://discuss.elastic.co/u/rguerrero)\ **Replies:** 0\ **Last updated:** [December 9, 2020, 3:31pm UTC](https://discuss.elastic.co/t/logstash-output-http-message/258150 "2020-12-09T15:31:59Z")
The next pipeline work good. input { jdbc { jdbc\_driver\_library =\> "C:\\\\03\_ELK\\\\logstash\\\\drivers\\\\ojdbc6-11.2.0.3.jar" jdbc\_driver\_class =\> "Java::oracle.jdbc.driver.OracleDriver" jdbc\_connection\_string =\>… --- ## [Unable to install logstash-codec-sflow plugin in logstash 5.3.2 version](https://discuss.elastic.co/t/unable-to-install-logstash-codec-sflow-plugin-in-logstash-5-3-2-version/258148)
**Author:** [@praveen\_cs](https://discuss.elastic.co/u/praveen_cs)\ **Replies:** 0\ **Last updated:** [December 9, 2020, 3:24pm UTC](https://discuss.elastic.co/t/unable-to-install-logstash-codec-sflow-plugin-in-logstash-5-3-2-version/258148 "2020-12-09T15:24:53Z")
Hi, Unable to install logstash-codec-sflow plugin in logstash 5.3.2 version ,Tried with logstash version 7.9.2 it is installing , why it is not allowing to install with logstash version 5.3.2 and am getting the foll… --- ## [Store value of filed as array](https://discuss.elastic.co/t/store-value-of-filed-as-array/258141)
**Author:** [@AayushPatel](https://discuss.elastic.co/u/AayushPatel)\ **Replies:** 0\ **Last updated:** [December 9, 2020, 2:50pm UTC](https://discuss.elastic.co/t/store-value-of-filed-as-array/258141 "2020-12-09T14:50:41Z")
I am trying to store a json response from an API that received by http plugin of Logstash, in Elasticsearch. one of fields called users returns value like this "users" =\> \[ \[0\] { … --- ## [Dateparsefailure](https://discuss.elastic.co/t/dateparsefailure/258133)
**Author:** [@Muhammad\_Faisal](https://discuss.elastic.co/u/Muhammad_Faisal)\ **Replies:** 1\ **Last updated:** [December 9, 2020, 2:15pm UTC](https://discuss.elastic.co/t/dateparsefailure/258133 "2020-12-09T14:15:06Z")
hi all, i have very simple input , why i am getting date parse failure here ? \`input { stdin{} } filter { date { match =\> \[ "message", "MMM dd yyyy HH:mm:ss" \] } } output { stdout { codec =\> "rubyde… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=271) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=273)