# Logstash

**URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=184

[Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md)

**Page:** 185

---

## [Logstash-input-azureblob | Unable to configure plugin](https://discuss.elastic.co/t/logstash-input-azureblob-unable-to-configure-plugin/288231)

<div class="topic-metadata">

**Author:** [@RS\_D](https://discuss.elastic.co/u/RS_D)\
**Replies:** 0\
**Last updated:** [November 2, 2021, 1:57pm UTC](https://discuss.elastic.co/t/logstash-input-azureblob-unable-to-configure-plugin/288231 "2021-11-02T13:57:49Z")

</div>

ELK\_VERSION=7.15.0 docker compose command to install the plugin : command: bash -c "bin/logstash-plugin install logstash-input-azureblob && logstash -f /usr/share/logstash/pipeline/logstash.conf" Help me to get rid o…

---

## [Can elasticsearch be accessed by http and htpps simultaneously?](https://discuss.elastic.co/t/can-elasticsearch-be-accessed-by-http-and-htpps-simultaneously/287885)

<div class="topic-metadata">

**Author:** [@rodri.gz](https://discuss.elastic.co/u/rodri.gz)\
**Replies:** 2\
**Last updated:** [November 2, 2021, 8:00am UTC](https://discuss.elastic.co/t/can-elasticsearch-be-accessed-by-http-and-htpps-simultaneously/287885 "2021-11-02T08:00:14Z")

</div>

Hello! I am working with a supposedly secure cluster that you access through https://localhost: 9200 but when I went to see the metricbeat.yml configuration file I saw a parameter that surprised me: ssl.verification\_m…

---

## [LogStash not working on Windows](https://discuss.elastic.co/t/logstash-not-working-on-windows/288202)

<div class="topic-metadata">

**Author:** [@Rainbow\_Lee](https://discuss.elastic.co/u/Rainbow_Lee)\
**Replies:** 0\
**Last updated:** [November 2, 2021, 7:21am UTC](https://discuss.elastic.co/t/logstash-not-working-on-windows/288202 "2021-11-02T07:21:15Z")

</div>

I just download logstash-7.15.1 and unzip it. I run ./bin/logstash -f logstash-simple.conf but got errors like this \[FATAL\] 2021-11-02 14:21:27.366 \[main\] Logstash - Logstash stopped processing because of an error: (…

---

## [Loading SQL Server data using Logstash](https://discuss.elastic.co/t/loading-sql-server-data-using-logstash/288097)

<div class="topic-metadata">

**Author:** [@sunilmpatil](https://discuss.elastic.co/u/sunilmpatil)\
**Replies:** 2\
**Last updated:** [November 2, 2021, 5:49am UTC](https://discuss.elastic.co/t/loading-sql-server-data-using-logstash/288097 "2021-11-02T05:49:37Z")

</div>

Hi All, I am planning to load data from SQL Server database to Elastic. While trying to load this data using Logstash JDBC plugin, I am facing issue. I am pasting details as below, please suggest what is missing here. (…

---

## [How To read in logstash the response from elasticsearch using elasticsearch output plugin](https://discuss.elastic.co/t/how-to-read-in-logstash-the-response-from-elasticsearch-using-elasticsearch-output-plugin/288135)

<div class="topic-metadata">

**Author:** [@yeppazu](https://discuss.elastic.co/u/yeppazu)\
**Replies:** 5\
**Last updated:** [November 1, 2021, 2:16pm UTC](https://discuss.elastic.co/t/how-to-read-in-logstash-the-response-from-elasticsearch-using-elasticsearch-output-plugin/288135 "2021-11-01T14:16:31Z")

</div>

Hi :slight\_smile: there is a some method to read response from Elasticsearch and trigger an action over it? (eg: running script, call an http url, ...) Normally, elastic return a JSON after HTTP index request. This JSO…

---

## [Hybrid Grok and Dissect Parsing](https://discuss.elastic.co/t/hybrid-grok-and-dissect-parsing/288021)

<div class="topic-metadata">

**Author:** [@Nico\_Pampaloni](https://discuss.elastic.co/u/Nico_Pampaloni)\
**Replies:** 2\
**Last updated:** [November 1, 2021, 11:25am UTC](https://discuss.elastic.co/t/hybrid-grok-and-dissect-parsing/288021 "2021-11-01T11:25:35Z")

</div>

Hi Everyone! I have a question about using Dissect and Grok together when working with a format that has varying structure. A line in my log file might look like this: word,word.word.word.word|timestamp|number,number,na…

---

## [Logstash Kubernetes Operator?](https://discuss.elastic.co/t/logstash-kubernetes-operator/288127)

<div class="topic-metadata">

**Author:** [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\
**Replies:** 0\
**Last updated:** [November 1, 2021, 10:31am UTC](https://discuss.elastic.co/t/logstash-kubernetes-operator/288127 "2021-11-01T10:31:49Z")

</div>

Hi folks, does anyone know if there is or will be a dedicated Logstash Operator in future? From what I can see from the current official Elasticsearch Operator, Logstash is not considered as a component there at all. D…

---

## [Handling Large XML Files](https://discuss.elastic.co/t/handling-large-xml-files/288068)

<div class="topic-metadata">

**Author:** [@Tim\_Mobley](https://discuss.elastic.co/u/Tim_Mobley)\
**Replies:** 1\
**Last updated:** [October 31, 2021, 11:54pm UTC](https://discuss.elastic.co/t/handling-large-xml-files/288068 "2021-10-31T23:54:03Z")

</div>

I have a large XML file that I'm trying to parse. But when I use max\_lines I somehow lose the @timestamp field. Is there a way to use both? input { file { path =\> \[ "C:/temp/TEST/\*.xml" \] start\_position =\> "be…

---

## [Logstash Pipeline Config API](https://discuss.elastic.co/t/logstash-pipeline-config-api/288087)

<div class="topic-metadata">

**Author:** [@cryonix\_pod](https://discuss.elastic.co/u/cryonix_pod)\
**Replies:** 2\
**Last updated:** [October 31, 2021, 6:00pm UTC](https://discuss.elastic.co/t/logstash-pipeline-config-api/288087 "2021-10-31T18:00:50Z")

</div>

Due to the different flow of traffic I need to change the value of pipeline.workers attribute dynamically. But currently I am using sed and other bash tool to edit the pipeline.yaml file and update the value. Is there a…

---

## [How can I parse nested JSON strings to JSON objects?](https://discuss.elastic.co/t/how-can-i-parse-nested-json-strings-to-json-objects/287944)

<div class="topic-metadata">

**Author:** [@dudwell](https://discuss.elastic.co/u/dudwell)\
**Replies:** 6\
**Last updated:** [October 31, 2021, 2:50pm UTC](https://discuss.elastic.co/t/how-can-i-parse-nested-json-strings-to-json-objects/287944 "2021-10-31T14:50:37Z")

</div>

Could some possibly provide some guidance on how to parse out an array of JSON strings into individual JSON objects with Logstash json parser??? My logstash filter look as follows... filter { grok { add\_tag =\> \[…

---

## [Looking for a well supported jdbc output plugin](https://discuss.elastic.co/t/looking-for-a-well-supported-jdbc-output-plugin/288071)

<div class="topic-metadata">

**Author:** [@David\_Levine](https://discuss.elastic.co/u/David_Levine)\
**Replies:** 3\
**Last updated:** [October 31, 2021, 12:29am UTC](https://discuss.elastic.co/t/looking-for-a-well-supported-jdbc-output-plugin/288071 "2021-10-31T00:29:34Z")

</div>

I'm relatively new to logstash and am hoping some of you might be able to point me in the right direction. I'm looking for a reasonably well supported jdbc output plugin that works against the latest version of logstas…

---

## [Add date and seconds to odd(?) ISO 8601-ish timestamp?](https://discuss.elastic.co/t/add-date-and-seconds-to-odd-iso-8601-ish-timestamp/288031)

<div class="topic-metadata">

**Author:** [@tjswe](https://discuss.elastic.co/u/tjswe)\
**Replies:** 2\
**Last updated:** [October 30, 2021, 6:47pm UTC](https://discuss.elastic.co/t/add-date-and-seconds-to-odd-iso-8601-ish-timestamp/288031 "2021-10-30T18:47:20Z")

</div>

Im logging AIS-data (ship positions) into Elasticsearch. The "timestamp" field from the receiver is 99% of the time a regular ISO8601 timestamp like "2021-10-29T17:55:30Z" which logstash takes in without issues. But som…

---

## [Logstash - Grok dynamic parsing including nested fields](https://discuss.elastic.co/t/logstash-grok-dynamic-parsing-including-nested-fields/287919)

<div class="topic-metadata">

**Author:** [@anandd4](https://discuss.elastic.co/u/anandd4)\
**Replies:** 2\
**Last updated:** [October 30, 2021, 11:21am UTC](https://discuss.elastic.co/t/logstash-grok-dynamic-parsing-including-nested-fields/287919 "2021-10-30T11:21:11Z")

</div>

Hello, We have the following pattern which includes dynamic fields nested inside a tag(tag3 in the example below) that needs to be parsed & grok seems to be the go-to-filter to achieve this. However, given fields inside…

---

## [Parse XML by Document not Element](https://discuss.elastic.co/t/parse-xml-by-document-not-element/288028)

<div class="topic-metadata">

**Author:** [@Tim\_Mobley](https://discuss.elastic.co/u/Tim_Mobley)\
**Replies:** 5\
**Last updated:** [October 29, 2021, 10:32pm UTC](https://discuss.elastic.co/t/parse-xml-by-document-not-element/288028 "2021-10-29T22:32:21Z")

</div>

As a test, I am trying to ingest the following simple XML file (the actual production file is huge). \<note\> \<to\>Tove\</to\> \<from\>Jani\</from\> \<heading\>Reminder\</heading\> \<body\>Don't forget me this weekend!\</body\> \</note\> …

---

## [Logstash not able to start with multiple pipelines](https://discuss.elastic.co/t/logstash-not-able-to-start-with-multiple-pipelines/288029)

<div class="topic-metadata">

**Author:** [@dyadav](https://discuss.elastic.co/u/dyadav)\
**Replies:** 0\
**Last updated:** [October 29, 2021, 6:04pm UTC](https://discuss.elastic.co/t/logstash-not-able-to-start-with-multiple-pipelines/288029 "2021-10-29T18:04:28Z")

</div>

I have setup logstash on EKS in AWS. Now, I am trying to setup multiple pipelines in it but it is not able to pick up my pipeline or just picks the main pipeline. Below are my configurations: ConfigMap apiVersion: v1 …

---

## [Kafka-logstash-elastic disaster resistant](https://discuss.elastic.co/t/kafka-logstash-elastic-disaster-resistant/287994)

<div class="topic-metadata">

**Author:** [@Mamol27](https://discuss.elastic.co/u/Mamol27)\
**Replies:** 2\
**Last updated:** [October 29, 2021, 2:45pm UTC](https://discuss.elastic.co/t/kafka-logstash-elastic-disaster-resistant/287994 "2021-10-29T14:45:47Z")

</div>

Hi! I am testing disaster resistant of pipeline kafka-\>logstash-\>elastic Logstash config input { kafka { bootstrap\_servers =\> "172.29.39.115:9093, 172.29.39.116:9093" topics =\> "test-test" #consumer\_thre…

---

## [Failed to parse a date field in Logstash](https://discuss.elastic.co/t/failed-to-parse-a-date-field-in-logstash/288008)

<div class="topic-metadata">

**Author:** [@dedi27](https://discuss.elastic.co/u/dedi27)\
**Replies:** 2\
**Last updated:** [October 29, 2021, 2:36pm UTC](https://discuss.elastic.co/t/failed-to-parse-a-date-field-in-logstash/288008 "2021-10-29T14:36:00Z")

</div>

Fail to parse a date field in Logstash Hello, I'm trying to parse a date field from a log file coming from an Exim Mail Server where I have a date string like this "2021-10-28 15:44:40". In my filter in Logstash, firs…

---

## [Logstash performance and jdbc\_streaming](https://discuss.elastic.co/t/logstash-performance-and-jdbc-streaming/288009)

<div class="topic-metadata">

**Author:** [@Roberto\_B](https://discuss.elastic.co/u/Roberto_B)\
**Replies:** 0\
**Last updated:** [October 29, 2021, 2:00pm UTC](https://discuss.elastic.co/t/logstash-performance-and-jdbc-streaming/288009 "2021-10-29T14:00:16Z")

</div>

Hi all, i'm using in this days Logstash (I love logstash if I could I would use it to make coffee too :upside\_down\_face: ), like an ETL get the data from mysql transform it enriching the data with multiple jdbc\_stream…

---

## [Aggregate filter plugin - timeout function bug](https://discuss.elastic.co/t/aggregate-filter-plugin-timeout-function-bug/286383)

<div class="topic-metadata">

**Author:** [@MarcelOs](https://discuss.elastic.co/u/MarcelOs)\
**Replies:** 6\
**Last updated:** [October 29, 2021, 7:22am UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-timeout-function-bug/286383 "2021-10-29T07:22:44Z")

</div>

Hello everyone, I have some problems with the Timeout command from the aggregate filter plugin. I want to create a script that calculates the meantime between two log entries. There are examples of such a function (http…

---

## [Java OutOfMemory when synchronizing initially huge resultsets from Oracle via logstash jdbc to Elastic](https://discuss.elastic.co/t/java-outofmemory-when-synchronizing-initially-huge-resultsets-from-oracle-via-logstash-jdbc-to-elastic/287984)

<div class="topic-metadata">

**Author:** [@di.loh](https://discuss.elastic.co/u/di.loh)\
**Replies:** 0\
**Last updated:** [October 29, 2021, 9:40am UTC](https://discuss.elastic.co/t/java-outofmemory-when-synchronizing-initially-huge-resultsets-from-oracle-via-logstash-jdbc-to-elastic/287984 "2021-10-29T09:40:54Z")

</div>

We are getting Java OutOfMemory during initial loading a huge dataset of ~70 Mio records with Logstash and jdbc into Elastic. It seems, the problem starts when using use\_prepared\_statements =\> true. Java Heapsize is 4GB…

---

## [JDBC Logstash with SystemD](https://discuss.elastic.co/t/jdbc-logstash-with-systemd/287979)

<div class="topic-metadata">

**Author:** [@sastorsl](https://discuss.elastic.co/u/sastorsl)\
**Replies:** 2\
**Last updated:** [October 29, 2021, 8:52am UTC](https://discuss.elastic.co/t/jdbc-logstash-with-systemd/287979 "2021-10-29T08:52:35Z")

</div>

Setup logstash jdbc-input, and wanted to use the default Oracle TNS Names lookup instead of storing the JDBC connect string directly in the logstash configuration. Basically the JDBC connect string can be very long, and…

---

## [ADVICE For loadbalance](https://discuss.elastic.co/t/advice-for-loadbalance/287957)

<div class="topic-metadata">

**Author:** [@Tombal](https://discuss.elastic.co/u/Tombal)\
**Replies:** 1\
**Last updated:** [October 28, 2021, 10:24pm UTC](https://discuss.elastic.co/t/advice-for-loadbalance/287957 "2021-10-28T22:24:35Z")

</div>

Hello there , I will get syslog messages from firewall devices and send them to kafka as a producer. Because there may be several consumers which will consume events so they may be on different offsets. Now my questio…

---

## [Can anyone help on nested json parsing with Logstash?](https://discuss.elastic.co/t/can-anyone-help-on-nested-json-parsing-with-logstash/287661)

<div class="topic-metadata">

**Author:** [@dudwell](https://discuss.elastic.co/u/dudwell)\
**Replies:** 10\
**Last updated:** [October 28, 2021, 5:08pm UTC](https://discuss.elastic.co/t/can-anyone-help-on-nested-json-parsing-with-logstash/287661 "2021-10-28T17:08:31Z")

</div>

I am currently looking to parse some json records on logstash to then push to opensearch/kibana for analysis. Specifically I hope to pull the "rtt" and associated "instance" value metric from each message body so I can r…

---

## [How to generate a logstash event within a custom java filter plugin after timeout?](https://discuss.elastic.co/t/how-to-generate-a-logstash-event-within-a-custom-java-filter-plugin-after-timeout/287901)

<div class="topic-metadata">

**Author:** [@Olli](https://discuss.elastic.co/u/Olli)\
**Replies:** 0\
**Last updated:** [October 28, 2021, 10:26am UTC](https://discuss.elastic.co/t/how-to-generate-a-logstash-event-within-a-custom-java-filter-plugin-after-timeout/287901 "2021-10-28T10:26:32Z")

</div>

I want to correlate messages sending to logstash within a filter and send them to the next filter of logstash inside the pipeline. However I have successfully setup a custom logstash filter plugin according to elastic d…

---

## [JSON filter Invalid FieldReference for \[ \]](https://discuss.elastic.co/t/json-filter-invalid-fieldreference-for/287897)

<div class="topic-metadata">

**Author:** [@Wonder\_Garance](https://discuss.elastic.co/u/Wonder_Garance)\
**Replies:** 1\
**Last updated:** [October 28, 2021, 9:55am UTC](https://discuss.elastic.co/t/json-filter-invalid-fieldreference-for/287897 "2021-10-28T09:55:16Z")

</div>

Hello, I'm using JSON filter to parse JSON files, the field title contains characters \[ and \] I got this error: exception=\>#\<RuntimeError: Invalid FieldReference: \[case\] index abc... As the whole value of this field s…

---

## [Follow\_redirects not implemented in http output plugin for Logstash](https://discuss.elastic.co/t/follow-redirects-not-implemented-in-http-output-plugin-for-logstash/48824)

<div class="topic-metadata">

**Author:** [@Sergej-Popov](https://discuss.elastic.co/u/Sergej-Popov)\
**Replies:** 5\
**Last updated:** [October 28, 2021, 7:04am UTC](https://discuss.elastic.co/t/follow-redirects-not-implemented-in-http-output-plugin-for-logstash/48824 "2021-10-28T07:04:17Z")

</div>

I am having difficult time making logstash http output plugin to forward events to http endpoint which returns 307. (no control over that). Redirect needs to be followed to get to the right URL. Docs specify follow\_red…

---

## [Http Output 307 not being redirected](https://discuss.elastic.co/t/http-output-307-not-being-redirected/98493)

<div class="topic-metadata">

**Author:** [@insanityisnice](https://discuss.elastic.co/u/insanityisnice)\
**Replies:** 1\
**Last updated:** [October 28, 2021, 7:03am UTC](https://discuss.elastic.co/t/http-output-307-not-being-redirected/98493 "2021-10-28T07:03:16Z")

</div>

I’m using an http output plugin to push events to EventStore. I am unable to get a successful POST and get an failure do to a 307 http status code. I’ve found several postings indicating that the http output plugin s…

---

## [Need to improve my ruby code to convert the data in right format](https://discuss.elastic.co/t/need-to-improve-my-ruby-code-to-convert-the-data-in-right-format/287801)

<div class="topic-metadata">

**Author:** [@pk.241011](https://discuss.elastic.co/u/pk.241011)\
**Replies:** 2\
**Last updated:** [October 28, 2021, 4:08am UTC](https://discuss.elastic.co/t/need-to-improve-my-ruby-code-to-convert-the-data-in-right-format/287801 "2021-10-28T04:08:19Z")

</div>

Hi, I am using XPATH in XML filter to extract some values. I am getting three arrays. Something on these lines: ArrayOfId = \["SamplingRate","Humidity","DateOfCalibration","ProductFamily"\] ArrayOfType = \["Number","Numb…

---

## [Some application logs are not getting indexed](https://discuss.elastic.co/t/some-application-logs-are-not-getting-indexed/287740)

<div class="topic-metadata">

**Author:** [@prat](https://discuss.elastic.co/u/prat)\
**Replies:** 5\
**Last updated:** [October 27, 2021, 9:20pm UTC](https://discuss.elastic.co/t/some-application-logs-are-not-getting-indexed/287740 "2021-10-27T21:20:44Z")

</div>

Hi All, Some of the application logs are not getting indexed. below are the error log about then in logstash logs. below logs has error like, ....:response=\>{"index"=\>{"\_index"=\>"filebeat-7.14.0-2021.10", "\_type"=\>"\_d…

---

## [Logstash is delayed by 20mins reading from an S3 bucket](https://discuss.elastic.co/t/logstash-is-delayed-by-20mins-reading-from-an-s3-bucket/287833)

<div class="topic-metadata">

**Author:** [@Michael\_Day1](https://discuss.elastic.co/u/Michael_Day1)\
**Replies:** 0\
**Last updated:** [October 27, 2021, 4:54pm UTC](https://discuss.elastic.co/t/logstash-is-delayed-by-20mins-reading-from-an-s3-bucket/287833 "2021-10-27T16:54:01Z")

</div>

I've set up a couple of logstashen running in ECS on AWS. They pull the ALB logs from the S3 buckets they dump then in to and one pokes them in to ES and the other pokes them in to Loki. Both end up being around 20-30mi…

[Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=183)

[Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=185)