# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=181 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 182 --- ## [Working with JSON array of objects via Ruby code block](https://discuss.elastic.co/t/working-with-json-array-of-objects-via-ruby-code-block/289009)
**Author:** [@Colin\_K](https://discuss.elastic.co/u/Colin_K)\ **Replies:** 10\ **Last updated:** [November 15, 2021, 9:14pm UTC](https://discuss.elastic.co/t/working-with-json-array-of-objects-via-ruby-code-block/289009 "2021-11-15T21:14:29Z")
I have a JSON array of nested objects. I wrote a filter json { source =\> \['body'\] target =\> "parsed" } ruby { code =\> "mapped = event.get('parsed').map{ |h| \[h\['k1'\], h\['k2'\],h\['k3'\],h\['k4'\]\] }; event.set(… --- ## [LogStash Error](https://discuss.elastic.co/t/logstash-error/288444)
**Author:** [@John\_snow](https://discuss.elastic.co/u/John_snow)\ **Replies:** 18\ **Last updated:** [November 15, 2021, 7:30pm UTC](https://discuss.elastic.co/t/logstash-error/288444 "2021-11-15T19:30:37Z")
Hi Team, I'm getting exception=\>#\ … --- ## [Cannot access environment variables](https://discuss.elastic.co/t/cannot-access-environment-variables/288928)
**Author:** [@tofubeats](https://discuss.elastic.co/u/tofubeats)\ **Replies:** 7\ **Last updated:** [November 15, 2021, 6:15pm UTC](https://discuss.elastic.co/t/cannot-access-environment-variables/288928 "2021-11-15T18:15:16Z")
Hi, My config is really simple, I'm just trying to get a simple envornment variable to be accessed and used by a Logstash pipeline. It looks like this: input{ file { path =\> "${test}" } } filter {} output {… --- ## [Logstash Date Parse failure](https://discuss.elastic.co/t/logstash-date-parse-failure/289185)
**Author:** [@Stephy\_Jacob](https://discuss.elastic.co/u/Stephy_Jacob)\ **Replies:** 4\ **Last updated:** [November 15, 2021, 5:18pm UTC](https://discuss.elastic.co/t/logstash-date-parse-failure/289185 "2021-11-15T17:18:41Z")
Hello, I am banging my head for quite sometime to understand why am I getting dateparsefailure after trying multiple match combinations. Logstash Version = 7.9.2 sudo echo '2021-11-15T11:17:56.831Z' | /usr/share/logst… --- ## [JDBC input/output with additional data transforming](https://discuss.elastic.co/t/jdbc-input-output-with-additional-data-transforming/289194)
**Author:** [@Stateros](https://discuss.elastic.co/u/Stateros)\ **Replies:** 2\ **Last updated:** [November 15, 2021, 2:16pm UTC](https://discuss.elastic.co/t/jdbc-input-output-with-additional-data-transforming/289194 "2021-11-15T14:16:55Z")
Hello folks. I am pretty new in logstash. I have a task reload lot of data 5B records from mysql to another mysql with some aggregation. I need to load all fields param1, date group them and take count rows in main table… --- ## [Using redis read replica in logstash input configuration](https://discuss.elastic.co/t/using-redis-read-replica-in-logstash-input-configuration/289182)
**Author:** [@vdere1](https://discuss.elastic.co/u/vdere1)\ **Replies:** 0\ **Last updated:** [November 15, 2021, 10:28am UTC](https://discuss.elastic.co/t/using-redis-read-replica-in-logstash-input-configuration/289182 "2021-11-15T10:28:38Z")
Hello, I am having a throttling issue on redis which could be due to the usage of just the primary endpoint to collect the logs from redis using logstash input configuration. Is it possible to use the redis read replic… --- ## [Filter does not work properly](https://discuss.elastic.co/t/filter-does-not-work-properly/289039)
**Author:** [@111435](https://discuss.elastic.co/u/111435)\ **Replies:** 1\ **Last updated:** [November 12, 2021, 5:47pm UTC](https://discuss.elastic.co/t/filter-does-not-work-properly/289039 "2021-11-12T17:47:43Z")
I have this filter: filter { mutate { convert =\> {"ticket\_fields" =\> "string" "tags" =\> "string"} } mutate { rename =\> { "tags" =\> "category" } } if "неактивность" in \[category\] { if "Диалог не состоя… --- ## [Elasticsearch schedule not working](https://discuss.elastic.co/t/elasticsearch-schedule-not-working/289088)
**Author:** [@SANDRA\_ROY\_ARICATT](https://discuss.elastic.co/u/SANDRA_ROY_ARICATT)\ **Replies:** 1\ **Last updated:** [November 12, 2021, 4:57pm UTC](https://discuss.elastic.co/t/elasticsearch-schedule-not-working/289088 "2021-11-12T16:57:57Z")
Even when I provided the schedule option, logstash is getting records from MySQL every second. I wanted logstash to poll MySQL only in every 30 minutes. logstash db.conf file #file : db.conf input{ jdbc { #connecti… --- ## [Logstash connection with oracle wallet](https://discuss.elastic.co/t/logstash-connection-with-oracle-wallet/289086)
**Author:** [@rodolfo\_pedra](https://discuss.elastic.co/u/rodolfo_pedra)\ **Replies:** 0\ **Last updated:** [November 12, 2021, 3:47pm UTC](https://discuss.elastic.co/t/logstash-connection-with-oracle-wallet/289086 "2021-11-12T15:47:55Z")
Hello I'm trying to make a connection to oracle database that uses wallet. I didn't find enough information to do it, so I decided to do it according to my knowledge, but I'm not successful. If anyone can help me. inpu… --- ## [Grok pattern for a unique value inside a field](https://discuss.elastic.co/t/grok-pattern-for-a-unique-value-inside-a-field/288890)
**Author:** [@sidharth\_vijayakumar](https://discuss.elastic.co/u/sidharth_vijayakumar)\ **Replies:** 2\ **Last updated:** [November 12, 2021, 9:35am UTC](https://discuss.elastic.co/t/grok-pattern-for-a-unique-value-inside-a-field/288890 "2021-11-12T09:35:20Z")
need to create a new field status\_code with value-successful by using ingest pipeline when status inside message field has 200 and when status inside message field is 502,404,402 it muse create status\_code with value fai… --- ## [Parsing logs using Regex pattern in logstash filter](https://discuss.elastic.co/t/parsing-logs-using-regex-pattern-in-logstash-filter/288117)
**Author:** [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)\ **Replies:** 2\ **Last updated:** [November 12, 2021, 6:53am UTC](https://discuss.elastic.co/t/parsing-logs-using-regex-pattern-in-logstash-filter/288117 "2021-11-12T06:53:22Z")
Hi, I am using logstash to ingest data into the Elasticsearch using a txt file. I build a regex query to extract msg-id field from the logs, but msg-id field is not getting extracted though the regex query is working f… --- ## [How to Improve throughput of Azure EventHub input plugin](https://discuss.elastic.co/t/how-to-improve-throughput-of-azure-eventhub-input-plugin/288871)
**Author:** [@menha](https://discuss.elastic.co/u/menha)\ **Replies:** 1\ **Last updated:** [November 12, 2021, 1:56am UTC](https://discuss.elastic.co/t/how-to-improve-throughput-of-azure-eventhub-input-plugin/288871 "2021-11-12T01:56:53Z")
Hi Experts, I am using Azure EventHub plugin as input in Logstash, but the event receiving rate is at most around 1.2k/s, it cannot be improved by any methods i tried. Currently the events being ingested to EventHub wou… --- ## [Filter json parse is replacing tags set from the input](https://discuss.elastic.co/t/filter-json-parse-is-replacing-tags-set-from-the-input/289032)
**Author:** [@teebu](https://discuss.elastic.co/u/teebu)\ **Replies:** 2\ **Last updated:** [November 12, 2021, 1:21am UTC](https://discuss.elastic.co/t/filter-json-parse-is-replacing-tags-set-from-the-input/289032 "2021-11-12T01:21:36Z")
I have input that sets tags input { ... tags =\> \["a"\] } But in the filter, I do a json parse: filter { json { source =\> "i" } } And part of that parse, includes tags. However, the tags it has replace the … --- ## [How can I use a proxy as gateway of my logstash to controll my elasticsearch cluster users?](https://discuss.elastic.co/t/how-can-i-use-a-proxy-as-gateway-of-my-logstash-to-controll-my-elasticsearch-cluster-users/288965)
**Author:** [@skylogs](https://discuss.elastic.co/u/skylogs)\ **Replies:** 0\ **Last updated:** [November 11, 2021, 8:16am UTC](https://discuss.elastic.co/t/how-can-i-use-a-proxy-as-gateway-of-my-logstash-to-controll-my-elasticsearch-cluster-users/288965 "2021-11-11T08:16:36Z")
Hi, I'm trying to design Elasticsearch as a service platform and I want to control my user in size of logs that they can send to my cluster. how can drop some user's log in logstash input or using a gateway before logst… --- ## [How toi get Mainfram data to Logstash](https://discuss.elastic.co/t/how-toi-get-mainfram-data-to-logstash/288972)
**Author:** [@Shalinicts](https://discuss.elastic.co/u/Shalinicts)\ **Replies:** 1\ **Last updated:** [November 11, 2021, 4:29pm UTC](https://discuss.elastic.co/t/how-toi-get-mainfram-data-to-logstash/288972 "2021-11-11T16:29:36Z")
Read data files which are delivered in zipped EDCDIC format and get it into Elasticsearch via Logstash --- ## [Rsyslog logstash input](https://discuss.elastic.co/t/rsyslog-logstash-input/288664)
**Author:** [@duscha](https://discuss.elastic.co/u/duscha)\ **Replies:** 9\ **Last updated:** [November 11, 2021, 2:46pm UTC](https://discuss.elastic.co/t/rsyslog-logstash-input/288664 "2021-11-11T14:46:07Z")
I installed on CentOS-7 an ELK stack according to a tutorial. This worked all fine and I can see and search the logs of the localhost server itself in the Kibana interface. The purpose of the server is to act as a centr… --- ## [Logstash not sending data to elastic search](https://discuss.elastic.co/t/logstash-not-sending-data-to-elastic-search/288903)
**Author:** [@Akhil\_Chandran](https://discuss.elastic.co/u/Akhil_Chandran)\ **Replies:** 4\ **Last updated:** [November 11, 2021, 10:02am UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elastic-search/288903 "2021-11-11T10:02:06Z")
Trying to send data from logstash to an ES index, but the index is not created when logstash is run. Tried logginf the input lines and there is rows logged in the logstash log, but data not sent to ES index jdbc { … --- ## [Logstash - MSSQL Events and alerts](https://discuss.elastic.co/t/logstash-mssql-events-and-alerts/288931)
**Author:** [@Trinity\_Rolling](https://discuss.elastic.co/u/Trinity_Rolling)\ **Replies:** 1\ **Last updated:** [November 10, 2021, 10:33pm UTC](https://discuss.elastic.co/t/logstash-mssql-events-and-alerts/288931 "2021-11-10T22:33:39Z")
This is probably a basic question, but I cannot find a good example of where to start. I am currently ingesting MSSQL's ERRORLOG using Filebeat -\> LogStash -\> Elasticsearch. In Elasticsearch, I would like to set an ale… --- ## [Winlogbeat to Logstash Load Balancing and Failover](https://discuss.elastic.co/t/winlogbeat-to-logstash-load-balancing-and-failover/288902)
**Author:** [@6igwig](https://discuss.elastic.co/u/6igwig)\ **Replies:** 3\ **Last updated:** [November 10, 2021, 7:41pm UTC](https://discuss.elastic.co/t/winlogbeat-to-logstash-load-balancing-and-failover/288902 "2021-11-10T19:41:34Z")
I have two logstash servers. Do I have to choose between load balancing between them all the time vs using one as a fail over? For instance, if I am load balancing winlogbeat output and one of the logstash servers goes … --- ## [Attempting to Install template](https://discuss.elastic.co/t/attempting-to-install-template/288899)
**Author:** [@MKirby](https://discuss.elastic.co/u/MKirby)\ **Replies:** 1\ **Last updated:** [November 10, 2021, 6:58pm UTC](https://discuss.elastic.co/t/attempting-to-install-template/288899 "2021-11-10T18:58:54Z")
Good Morning Logstash experts. As most of you know I have been working on injesting and being able to see the syslogs that are being sent directly to my logstash server. When I run a debug on my logstash I see the foll… --- ## [Logstash-keystore.bat not using bundled JDK (Windows)](https://discuss.elastic.co/t/logstash-keystore-bat-not-using-bundled-jdk-windows/288719)
**Author:** [@Ryan\_Clark](https://discuss.elastic.co/u/Ryan_Clark)\ **Replies:** 4\ **Last updated:** [November 10, 2021, 6:05pm UTC](https://discuss.elastic.co/t/logstash-keystore-bat-not-using-bundled-jdk-windows/288719 "2021-11-10T18:05:44Z")
I was trying to setup a keystore and when running .\\logstash-keystore.bat I get a message saying "Using bundled JDK: "" then it says, No 'java.exe' executable found on PATH. I'm running version 7.15.1. The crazy thing is… --- ## [Logstash Filtering logs](https://discuss.elastic.co/t/logstash-filtering-logs/288521)
**Author:** [@John\_snow](https://discuss.elastic.co/u/John_snow)\ **Replies:** 5\ **Last updated:** [November 10, 2021, 5:46pm UTC](https://discuss.elastic.co/t/logstash-filtering-logs/288521 "2021-11-10T17:46:31Z")
I want to filter logs on base of account\_id and send to different output in logstash. Log1: {'account\_id': '1234567890', 'other data': "Some\_data"} Log2: {'account\_id': '0987654321', 'other data': "Some\_data"} i want … --- ## [Csv filter output splitting a row in two because of some special char](https://discuss.elastic.co/t/csv-filter-output-splitting-a-row-in-two-because-of-some-special-char/288710)
**Author:** [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Replies:** 8\ **Last updated:** [November 10, 2021, 2:59pm UTC](https://discuss.elastic.co/t/csv-filter-output-splitting-a-row-in-two-because-of-some-special-char/288710 "2021-11-10T14:59:24Z")
I have a csv file with 7543 entries that I'd like to get to cloud SIEM. My config file is as follows input { file { … --- ## ["bad\_certificate" error on elasticsearch input plugin](https://discuss.elastic.co/t/bad-certificate-error-on-elasticsearch-input-plugin/288812)
**Author:** [@plubbs](https://discuss.elastic.co/u/plubbs)\ **Replies:** 5\ **Last updated:** [November 10, 2021, 2:40pm UTC](https://discuss.elastic.co/t/bad-certificate-error-on-elasticsearch-input-plugin/288812 "2021-11-10T14:40:22Z")
Hello, TLDR - Our certificate we use with the output-Elasticsearch plugin does not work on the input-Elasticsearch plugin despite being a valid certificate We are currently attempting to connect to an existing index in… --- ## [Split a specific csv column into multiple fields](https://discuss.elastic.co/t/split-a-specific-csv-column-into-multiple-fields/288875)
**Author:** [@Greninja\_San](https://discuss.elastic.co/u/Greninja_San)\ **Replies:** 2\ **Last updated:** [November 10, 2021, 2:03pm UTC](https://discuss.elastic.co/t/split-a-specific-csv-column-into-multiple-fields/288875 "2021-11-10T14:03:33Z")
I have a csv file, with data like this : field1;field2;field3,field4\_1::field4\_2::field4\_3;field5;field6 The 4th field is composed of 3 values that I want to divide into 3 different fields in Elasticsearch. Is there a… --- ## [How to read the data displayed inside \[ERROR\]\[org.logstash.Logstash\]?](https://discuss.elastic.co/t/how-to-read-the-data-displayed-inside-error-org-logstash-logstash/288893)
**Author:** [@Robert\_Neagu](https://discuss.elastic.co/u/Robert_Neagu)\ **Replies:** 0\ **Last updated:** [November 10, 2021, 1:45pm UTC](https://discuss.elastic.co/t/how-to-read-the-data-displayed-inside-error-org-logstash-logstash/288893 "2021-11-10T13:45:30Z")
I want to read the data that is displayed inside the \[ERROR\]\[org.logstash.Logstash\] section inside the email plugin. --- ## [Trying to rename all elements inside array](https://discuss.elastic.co/t/trying-to-rename-all-elements-inside-array/288884)
**Author:** [@Ankitkumar\_Ravidas](https://discuss.elastic.co/u/Ankitkumar_Ravidas)\ **Replies:** 1\ **Last updated:** [November 10, 2021, 12:36pm UTC](https://discuss.elastic.co/t/trying-to-rename-all-elements-inside-array/288884 "2021-11-10T12:36:23Z")
Hi team, i have tags array in my mongodb contacts collection, which contain mongodb ids, its store as "tags":\[{"$oid":"61822c908200c6b810bfee4d"},{"$oid":"61822c908200c6b810bfee4e"}\] i want to rename all element… --- ## [Http\_poller plugin and parsing second level of json](https://discuss.elastic.co/t/http-poller-plugin-and-parsing-second-level-of-json/288872)
**Author:** [@111435](https://discuss.elastic.co/u/111435)\ **Replies:** 0\ **Last updated:** [November 10, 2021, 10:09am UTC](https://discuss.elastic.co/t/http-poller-plugin-and-parsing-second-level-of-json/288872 "2021-11-10T10:09:09Z")
Hello, I am using Http\_poller plugin for get api call. My conf.file: input { http\_poller { urls =\> { test2 =\> { method =\> get url =\> "https://somethin.somethin.kz/uapi/tickets?api\_token=fdf… --- ## [How to create Alerting/health checks for ELK services (filebeat/metricbeat/logstash)](https://discuss.elastic.co/t/how-to-create-alerting-health-checks-for-elk-services-filebeat-metricbeat-logstash/288844)
**Author:** [@Bharah01](https://discuss.elastic.co/u/Bharah01)\ **Replies:** 0\ **Last updated:** [November 10, 2021, 6:56am UTC](https://discuss.elastic.co/t/how-to-create-alerting-health-checks-for-elk-services-filebeat-metricbeat-logstash/288844 "2021-11-10T06:56:57Z")
I want to configure an Alerting seup for the ELK services (filebeat/metricbeat/logstash). The kibana version running is 6.5.1. The need is because whenever any of these services goes down there should be a trigger to Mi… --- ## [Logstash showing Document count as 1. only last line of the syslog is displaying in kibana](https://discuss.elastic.co/t/logstash-showing-document-count-as-1-only-last-line-of-the-syslog-is-displaying-in-kibana/288787)
**Author:** [@kalyan\_M](https://discuss.elastic.co/u/kalyan_M)\ **Replies:** 0\ **Last updated:** [November 9, 2021, 4:02pm UTC](https://discuss.elastic.co/t/logstash-showing-document-count-as-1-only-last-line-of-the-syslog-is-displaying-in-kibana/288787 "2021-11-09T16:02:12Z")
Hi, I am new to ELK configurations. I configured the ubuntu VM logs to ELK VM. I am getting logs from the ubuntu VM to log stash. but it was showing document count as only 1. please help with the issue. [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=180) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=182)