# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=161 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 162 --- ## [Logstash - jdbc retry multiple ips](https://discuss.elastic.co/t/logstash-jdbc-retry-multiple-ips/297107)
**Author:** [@Nasser](https://discuss.elastic.co/u/Nasser)\ **Replies:** 1\ **Last updated:** [February 14, 2022, 6:03pm UTC](https://discuss.elastic.co/t/logstash-jdbc-retry-multiple-ips/297107 "2022-02-14T18:03:12Z")
Hi i'm connecting to my database via scan name and my scan name resolve to 3 IPs but the issue is sometimes 1 or 2 IPs get unavailable my question how i can set the connection to retry all IPs as below jdbc\_connectio… --- ## [Adding multiple values to an array](https://discuss.elastic.co/t/adding-multiple-values-to-an-array/296494)
**Author:** [@JeremyP](https://discuss.elastic.co/u/JeremyP)\ **Replies:** 19\ **Last updated:** [February 14, 2022, 5:31pm UTC](https://discuss.elastic.co/t/adding-multiple-values-to-an-array/296494 "2022-02-14T17:31:27Z")
Hello, I have an array which contains a bunch of time stamps which get added every time the record is modified from the source PGSQL database. I'm looking to modify this array to contain both a string and a time stamp. … --- ## [Logstash or Spark?](https://discuss.elastic.co/t/logstash-or-spark/296500)
**Author:** [@Laetitia\_RICHARD](https://discuss.elastic.co/u/Laetitia_RICHARD)\ **Replies:** 4\ **Last updated:** [February 14, 2022, 4:14pm UTC](https://discuss.elastic.co/t/logstash-or-spark/296500 "2022-02-14T16:14:32Z")
Hello, I have a project where I have to send millions of entries from different MYSQL databases to Elasticsearch (in Elastic Cloud instance). I'll have to make transformations to these data before sending it in Elasti… --- ## [Getting continuous logstash tcp input error](https://discuss.elastic.co/t/getting-continuous-logstash-tcp-input-error/297150)
**Author:** [@ksarpong](https://discuss.elastic.co/u/ksarpong)\ **Replies:** 0\ **Last updated:** [February 14, 2022, 2:14pm UTC](https://discuss.elastic.co/t/getting-continuous-logstash-tcp-input-error/297150 "2022-02-14T14:14:39Z")
"level":"ERROR","loggerName":"logstash.inputs.tcp","timeMillis":1644844639537,"thread":"nioEventLoopGroup-54-26","logEvent":{"message":"null: closing due:"}} {"level":"WARN","loggerName":"deprecation.logstash.codecs.plai… --- ## [How to handle rotating directory in input file for logstash](https://discuss.elastic.co/t/how-to-handle-rotating-directory-in-input-file-for-logstash/297104)
**Author:** [@selflabs](https://discuss.elastic.co/u/selflabs)\ **Replies:** 3\ **Last updated:** [February 14, 2022, 9:59am UTC](https://discuss.elastic.co/t/how-to-handle-rotating-directory-in-input-file-for-logstash/297104 "2022-02-14T09:59:09Z")
Hi Team, I wants logstash to read logs from AWS S3 bucket but my log path directory is keep rotating on month basis and am not understand how to handle it. please suggest. input { s3 { "access\_key\_id" =\>… --- ## [Logstash with JDBC performance](https://discuss.elastic.co/t/logstash-with-jdbc-performance/296984)
**Author:** [@HerveSavard](https://discuss.elastic.co/u/HerveSavard)\ **Replies:** 7\ **Last updated:** [February 14, 2022, 9:10am UTC](https://discuss.elastic.co/t/logstash-with-jdbc-performance/296984 "2022-02-14T09:10:21Z")
Hello, I use Logstash with JDBC connector. I have 6 pipelines defined in my pipeline.yml. pipeline1 have 14 SQL pipeline2 have 10 SQL pipeline3 have 2 SQL pipeline4 have 8 SQL pipeline5 have 1 SQL pipeline6 have… --- ## [Split array with different nested elemetnts](https://discuss.elastic.co/t/split-array-with-different-nested-elemetnts/297063)
**Author:** [@Gil\_Brudner](https://discuss.elastic.co/u/Gil_Brudner)\ **Replies:** 6\ **Last updated:** [February 14, 2022, 3:23am UTC](https://discuss.elastic.co/t/split-array-with-different-nested-elemetnts/297063 "2022-02-14T03:23:12Z")
Hi, I have a data stream coming in as array in the following format: \[ { "dataType": 9, "payload": "\[{\\"isHeadingHome\\":0,\\"isTesing\\":0}\]", "data": "9" }, { "dataType": 8, "payload": "\[{\\"FRO… --- ## [Why i should pre-define the fields?](https://discuss.elastic.co/t/why-i-should-pre-define-the-fields/297076)
**Author:** [@Nasser](https://discuss.elastic.co/u/Nasser)\ **Replies:** 1\ **Last updated:** [February 13, 2022, 12:49pm UTC](https://discuss.elastic.co/t/why-i-should-pre-define-the-fields/297076 "2022-02-13T12:49:08Z")
Hi what is the benefit to pre-define the fields while the elastic will identify the fields automatically ? please need some explanation :slight\_smile: --- ## [HTTP output for stadtpuls.com IoT platform](https://discuss.elastic.co/t/http-output-for-stadtpuls-com-iot-platform/296777)
**Author:** [@CargoBikoMeter](https://discuss.elastic.co/u/CargoBikoMeter)\ **Replies:** 4\ **Last updated:** [February 12, 2022, 5:51pm UTC](https://discuss.elastic.co/t/http-output-for-stadtpuls-com-iot-platform/296777 "2022-02-12T17:51:37Z")
Currently I have a working logstash configuration which stores JSON based input data from my IoT data from TTN via output into my Elasticsearch node. Now I would extract some data (e.g. temperature, pressure) and send on… --- ## [Logstash hosts](https://discuss.elastic.co/t/logstash-hosts/296952)
**Author:** [@RusseL](https://discuss.elastic.co/u/RusseL)\ **Replies:** 1\ **Last updated:** [February 12, 2022, 2:02pm UTC](https://discuss.elastic.co/t/logstash-hosts/296952 "2022-02-12T14:02:43Z")
While transferring data with Logstash, there is an increase in write threads on the node that is not in the output host list that I specified. I do not want to transfer data over a single node. This causes me to experie… --- ## [Elastic-Stack7.17 on ubuntu 20.4LTS, still not indexing](https://discuss.elastic.co/t/elastic-stack7-17-on-ubuntu-20-4lts-still-not-indexing/296378)
**Author:** [@Mohammad\_Ramadan\_Abd](https://discuss.elastic.co/u/Mohammad_Ramadan_Abd)\ **Replies:** 10\ **Last updated:** [February 12, 2022, 8:43am UTC](https://discuss.elastic.co/t/elastic-stack7-17-on-ubuntu-20-4lts-still-not-indexing/296378 "2022-02-12T08:43:28Z")
Hello Bro, I have followed up this link How To Install Elasticsearch, Logstash, and Kibana (Elastic Stack) on Ubuntu 20.04 | DigitalOcean to install my elc-stack, and everything went as document explains. But, It stil… --- ## [Logstash document id string does not get evaluated](https://discuss.elastic.co/t/logstash-document-id-string-does-not-get-evaluated/297019)
**Author:** [@Leo\_Baby\_Jacob](https://discuss.elastic.co/u/Leo_Baby_Jacob)\ **Replies:** 1\ **Last updated:** [February 11, 2022, 7:05pm UTC](https://discuss.elastic.co/t/logstash-document-id-string-does-not-get-evaluated/297019 "2022-02-11T19:05:27Z")
To prevent data duplication while injesting data from logstash I am adding a document\_id string for logstash conf with a peopleRowId column. However it does not get evaluated. So in my case I am trying to set document id… --- ## [Load balancer for logstash question](https://discuss.elastic.co/t/load-balancer-for-logstash-question/296993)
**Author:** [@RajuParipelly](https://discuss.elastic.co/u/RajuParipelly)\ **Replies:** 2\ **Last updated:** [February 11, 2022, 4:33pm UTC](https://discuss.elastic.co/t/load-balancer-for-logstash-question/296993 "2022-02-11T16:33:27Z")
We have 3 logstash instances running and we are trying to create a load balancer for logstash using Big IP f5. This load balancer should distribute the logs that it receives from network devices to the 3 logstash instanc… --- ## [Compare message if it's similar to others](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826)
**Author:** [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Replies:** 9\ **Last updated:** [February 11, 2022, 4:08pm UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826 "2022-02-11T16:08:42Z")
Hi, filebeat.yml configuration: multiline.pattern: \\d\\d/\\d\\d/\\d\\d\\d\\d\\s\\d\\d:\\d\\d:\\d\\d\\s\\d\\d\\d-\\sMAIN\\sEXCEPTION\\b multiline.negate: true multiline.match: after multiline.max\_lines: 7 Logstash configuration: input { … --- ## [Multiple paths logstash output](https://discuss.elastic.co/t/multiple-paths-logstash-output/296831)
**Author:** [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Replies:** 0\ **Last updated:** [February 10, 2022, 10:51am UTC](https://discuss.elastic.co/t/multiple-paths-logstash-output/296831 "2022-02-10T10:51:22Z")
Hi, filebeat config: filebeat.inputs: - type: log enabled: true paths: - D:\\elastic\_stack\\testLog\\\* - D:\\elastic\_stack\\LogWal\\\* So in logstash i want for the first path did the split of the message and add fie… --- ## [Getting class not found exception when invoking logstash http input with queue type as persisted](https://discuss.elastic.co/t/getting-class-not-found-exception-when-invoking-logstash-http-input-with-queue-type-as-persisted/296970)
**Author:** [@Sivakumar\_Lakshminar](https://discuss.elastic.co/u/Sivakumar_Lakshminar)\ **Replies:** 0\ **Last updated:** [February 11, 2022, 11:40am UTC](https://discuss.elastic.co/t/getting-class-not-found-exception-when-invoking-logstash-http-input-with-queue-type-as-persisted/296970 "2022-02-11T11:40:02Z")
Hello All, I am trying to setup ELK for internal event monitoring for our platform. Need help with the below, Using logstash-7.10.2 with bundled jdk. with setting as per below snippets. It is working fine when trying … --- ## [Get indexed Elasticsearch document body from Logstash](https://discuss.elastic.co/t/get-indexed-elasticsearch-document-body-from-logstash/296865)
**Author:** [@NominaSumpta](https://discuss.elastic.co/u/NominaSumpta)\ **Replies:** 2\ **Last updated:** [February 11, 2022, 8:01am UTC](https://discuss.elastic.co/t/get-indexed-elasticsearch-document-body-from-logstash/296865 "2022-02-11T08:01:42Z")
Hi, Is there a way to see the exact body of a document that Logstash indexed? I'm using the Elasticsearch output plugin. Context: in our production environment, some data is output into Elasticsearch by Logstash. In El… --- ## [Logstash in K8s: Deployment vs StatefulSet](https://discuss.elastic.co/t/logstash-in-k8s-deployment-vs-statefulset/296595)
**Author:** [@Nadiia\_Maltseva](https://discuss.elastic.co/u/Nadiia_Maltseva)\ **Replies:** 1\ **Last updated:** [February 11, 2022, 6:43am UTC](https://discuss.elastic.co/t/logstash-in-k8s-deployment-vs-statefulset/296595 "2022-02-11T06:43:50Z")
Hi, community! I am running Logstash on an EC2 instance in AWS. And I want to migrate to Kubernetes (since all our applications are running there). Could you, please, advise me on what factors should be taken into acco… --- ## [Logstash cmd displaying wrong output](https://discuss.elastic.co/t/logstash-cmd-displaying-wrong-output/296938)
**Author:** [@anushka1203](https://discuss.elastic.co/u/anushka1203)\ **Replies:** 0\ **Last updated:** [February 11, 2022, 6:40am UTC](https://discuss.elastic.co/t/logstash-cmd-displaying-wrong-output/296938 "2022-02-11T06:40:45Z")
Hi, I am new to ELK stack and I am trying to test out parsing sample syslog messages by following the steps mentioned on the official website -- https://www.elastic.co/guide/en/logstash/current/config-examples.html#\_pro… --- ## [Date filter year not parsing correctly](https://discuss.elastic.co/t/date-filter-year-not-parsing-correctly/296883)
**Author:** [@pkrishnan](https://discuss.elastic.co/u/pkrishnan)\ **Replies:** 3\ **Last updated:** [February 11, 2022, 6:26am UTC](https://discuss.elastic.co/t/date-filter-year-not-parsing-correctly/296883 "2022-02-11T06:26:32Z")
This is my sample log line 2021-06-28 10:25:29.537695 traceID=d283c222257e0e92ba97269a5f780d81 spanID=f46645e821cb4dab A4 LG 4 Logger.cpp:250 - Configuring log file: /home/rxm/log/latest/2021-06-28-102529-log-%3N.log Ar… --- ## [Create multiple logstash configurations for different Filebeat instances](https://discuss.elastic.co/t/create-multiple-logstash-configurations-for-different-filebeat-instances/296781)
**Author:** [@userR](https://discuss.elastic.co/u/userR)\ **Replies:** 9\ **Last updated:** [February 10, 2022, 7:51pm UTC](https://discuss.elastic.co/t/create-multiple-logstash-configurations-for-different-filebeat-instances/296781 "2022-02-10T19:51:49Z")
As of now, I am using Docker compose to set up an image of Logstash. Here is my use case: I have Filebeat installed on 5 machines, each with different log paths and log formats. Instead of having one logstash.conf file … --- ## [DB sessions | logstash](https://discuss.elastic.co/t/db-sessions-logstash/296845)
**Author:** [@Nasser](https://discuss.elastic.co/u/Nasser)\ **Replies:** 1\ **Last updated:** [February 10, 2022, 2:44pm UTC](https://discuss.elastic.co/t/db-sessions-logstash/296845 "2022-02-10T14:44:34Z")
Hi i have 20 sql query, 2 query in each jdbc.conf file my question is how many sessions will jdbc open with DB to execute the query ? will open one session and execute 10 query or one session for each query ? my con… --- ## [Logstash output email](https://discuss.elastic.co/t/logstash-output-email/296856)
**Author:** [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Replies:** 0\ **Last updated:** [February 10, 2022, 2:32pm UTC](https://discuss.elastic.co/t/logstash-output-email/296856 "2022-02-10T14:32:54Z")
Hi, I have log file contains more than one exception so i want for this file when i found a word "EXCEPTION" just sending me one email. logstash.conf: output{ stdout { codec =\> rubydebug } if ("EXCEPTION… --- ## [How you connected Filebeat with Logstash in your Kubernetes Cluster using beats input?](https://discuss.elastic.co/t/how-you-connected-filebeat-with-logstash-in-your-kubernetes-cluster-using-beats-input/296847)
**Author:** [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Replies:** 0\ **Last updated:** [February 10, 2022, 1:50pm UTC](https://discuss.elastic.co/t/how-you-connected-filebeat-with-logstash-in-your-kubernetes-cluster-using-beats-input/296847 "2022-02-10T13:50:04Z")
Hi folks, when I try to send logs from filebeat via output.logstash: hosts: \["logstash-service:5044"\] to my logtash pod with: input { beats { port =\> 5044 } } without any security settings for … --- ## [Logstash sends the log to elasticsearch and Syslog-ng or rsyslog(multiple outputs) which keeps the original log](https://discuss.elastic.co/t/logstash-sends-the-log-to-elasticsearch-and-syslog-ng-or-rsyslog-multiple-outputs-which-keeps-the-original-log/295322)
**Author:** [@tobyvu](https://discuss.elastic.co/u/tobyvu)\ **Replies:** 7\ **Last updated:** [February 10, 2022, 4:17am UTC](https://discuss.elastic.co/t/logstash-sends-the-log-to-elasticsearch-and-syslog-ng-or-rsyslog-multiple-outputs-which-keeps-the-original-log/295322 "2022-02-10T04:17:17Z")
I am wanting to configure the log from: Filebeat -\> Logstash -\> Elasticsearch and syslog-ng(or rsyslog). I configured with Syslog-ng to get the log following the instructions at: Sending logs from Logstash to syslog-ng … --- ## [Problem with JDBC-Streaming filter plugin](https://discuss.elastic.co/t/problem-with-jdbc-streaming-filter-plugin/296791)
**Author:** [@steevhise](https://discuss.elastic.co/u/steevhise)\ **Replies:** 0\ **Last updated:** [February 9, 2022, 11:43pm UTC](https://discuss.elastic.co/t/problem-with-jdbc-streaming-filter-plugin/296791 "2022-02-09T23:43:04Z")
We're running into a problem where we're using jdbc-streaming to query a mysql database using a prepared statement and in some cases we get the error "java.sql.SQLException: No operations allowed after statement closed."… --- ## [Update Logstash OSS rpm to fix netty vulnerabilites](https://discuss.elastic.co/t/update-logstash-oss-rpm-to-fix-netty-vulnerabilites/296717)
**Author:** [@pikopl](https://discuss.elastic.co/u/pikopl)\ **Replies:** 1\ **Last updated:** [February 9, 2022, 10:25pm UTC](https://discuss.elastic.co/t/update-logstash-oss-rpm-to-fix-netty-vulnerabilites/296717 "2022-02-09T22:25:18Z")
Please update logstash OSS rpm to fix following netty-all jars vulnerabilities: \[NVD - CVE-2021-37137\] \[NVD - CVE-2021-37136\] Please update netty-all to at least 4.1.68.Final (currently 4.1.65.Final) --- ## [Filebeat to logstash, no data when logstash run as service, got data when I run by command](https://discuss.elastic.co/t/filebeat-to-logstash-no-data-when-logstash-run-as-service-got-data-when-i-run-by-command/296752)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 0\ **Last updated:** [February 9, 2022, 4:07pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-no-data-when-logstash-run-as-service-got-data-when-i-run-by-command/296752 "2022-02-09T16:07:45Z")
Hi, I got several pipelines getting data from a DB, and other new pipelines that get data from filebeat. When I run logstash as a service, I get data from the DB source, but no data from filebeat. When I run logstash b… --- ## [ERROR logstash.outputs.email](https://discuss.elastic.co/t/error-logstash-outputs-email/296729)
**Author:** [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\ **Replies:** 2\ **Last updated:** [February 9, 2022, 3:51pm UTC](https://discuss.elastic.co/t/error-logstash-outputs-email/296729 "2022-02-09T15:51:03Z")
Hi, My code: email { to =\> "za@gmail.com" via =\> 'smtp' address =\> 'smtp.gmail.com' from =\> 'john@gmail.com' authentication =\> "plain" username =\> "john@gmail.com" password =\> "john123" subject =\> '… --- ## [Tuning number of workers](https://discuss.elastic.co/t/tuning-number-of-workers/296748)
**Author:** [@Sonia1](https://discuss.elastic.co/u/Sonia1)\ **Replies:** 0\ **Last updated:** [February 9, 2022, 3:39pm UTC](https://discuss.elastic.co/t/tuning-number-of-workers/296748 "2022-02-09T15:39:24Z")
Hello, Currently our logstash consists of 7 pipelines that launch queries against a DB accessed by JDBC. With logstash we enrich and normalize this data and index it directly to Elasticsearch. In total 20 jdbc queries a… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=160) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=162)