# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=314 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 315 --- ## [How to made logstash work with io.confluent.kafka.serializers.KafkaAvroSerializer](https://discuss.elastic.co/t/how-to-made-logstash-work-with-io-confluent-kafka-serializers-kafkaavroserializer/245503)
**Author:** [@wysouf](https://discuss.elastic.co/u/wysouf)\ **Replies:** 0\ **Last updated:** [August 18, 2020, 10:57pm UTC](https://discuss.elastic.co/t/how-to-made-logstash-work-with-io-confluent-kafka-serializers-kafkaavroserializer/245503 "2020-08-18T22:57:36Z")
output{ kafka{ topic\_id =\> "myTopic" bootstrap\_servers =\> "127.0.0.1:9092" value\_serializer =\> "io.confluent.kafka.serializers.KafkaAvroSerializer" } } \[\[main\]-pipeline-manager\] kafka - Unable to cr… --- ## [Logstash cloudwatch output plugin configuration](https://discuss.elastic.co/t/logstash-cloudwatch-output-plugin-configuration/245495)
**Author:** [@timragon](https://discuss.elastic.co/u/timragon)\ **Replies:** 0\ **Last updated:** [August 18, 2020, 9:16pm UTC](https://discuss.elastic.co/t/logstash-cloudwatch-output-plugin-configuration/245495 "2020-08-18T21:16:23Z")
We are trying to configure CloudWatch Output Plugin. We are able to add "access\_key\_id" =\> "" "secret\_access\_key" =\> "" "region" =\> "" in logstash configuration file. Could you please guide how to set CloudWatch log… --- ## [Logstash shut down](https://discuss.elastic.co/t/logstash-shut-down/245261)
**Author:** [@sukumar.koganti](https://discuss.elastic.co/u/sukumar.koganti)\ **Replies:** 10\ **Last updated:** [August 18, 2020, 8:01pm UTC](https://discuss.elastic.co/t/logstash-shut-down/245261 "2020-08-18T20:01:37Z")
Hi I have successfully installed/configured the elasticsearh and kibana both along with the logstash. i am trying to process the simple .csv file through logstash.conf file but i am facing the below error. C:\\elastic\_… --- ## [SNMP Polling Very slow boot up time](https://discuss.elastic.co/t/snmp-polling-very-slow-boot-up-time/245046)
**Author:** [@csgeek](https://discuss.elastic.co/u/csgeek)\ **Replies:** 2\ **Last updated:** [August 18, 2020, 4:41pm UTC](https://discuss.elastic.co/t/snmp-polling-very-slow-boot-up-time/245046 "2020-08-18T16:41:39Z")
I'm looking at using Logstash to poll several hosts for SNMP data migrating from an old python script in favor of ELK. The pipeline is really incredibly simple. read data, convert to message format (ruby), send to goo… --- ## [Logstash errors retrying failed action with response code 500](https://discuss.elastic.co/t/logstash-errors-retrying-failed-action-with-response-code-500/245466)
**Author:** [@srk1](https://discuss.elastic.co/u/srk1)\ **Replies:** 2\ **Last updated:** [August 18, 2020, 4:29pm UTC](https://discuss.elastic.co/t/logstash-errors-retrying-failed-action-with-response-code-500/245466 "2020-08-18T16:29:40Z")
Hello ES experts, My logstash logs are completely being filled with below errors which is blocking me to see the information regarding other pipelines. I tried looking for this particular problem but it doesn't look lik… --- ## [Logstash OSS features](https://discuss.elastic.co/t/logstash-oss-features/245347)
**Author:** [@sera](https://discuss.elastic.co/u/sera)\ **Replies:** 6\ **Last updated:** [August 18, 2020, 2:59pm UTC](https://discuss.elastic.co/t/logstash-oss-features/245347 "2020-08-18T14:59:35Z")
What Logstash features are missing from Logstash OSS (latest versions)? Why is there no documentation describing the OSS vs Basic features? Thanks, Sera --- ## [How to parse and index xml in ES7](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269)
**Author:** [@azer\_gorai](https://discuss.elastic.co/u/azer_gorai)\ **Replies:** 8\ **Last updated:** [August 18, 2020, 2:38pm UTC](https://discuss.elastic.co/t/how-to-parse-and-index-xml-in-es7/245269 "2020-08-18T14:38:50Z")
I need to index and parse xml in elasticsearsh. I need one field to store the xml as raw data and another field that will contain a fvalue retrieved from xpath in the same xml --- ## [Plugin Installation Aborted](https://discuss.elastic.co/t/plugin-installation-aborted/245437)
**Author:** [@sonnenteich](https://discuss.elastic.co/u/sonnenteich)\ **Replies:** 0\ **Last updated:** [August 18, 2020, 1:50pm UTC](https://discuss.elastic.co/t/plugin-installation-aborted/245437 "2020-08-18T13:50:58Z")
I've written myself a simple filter plugin in java. I developed and tested it on an Ubuntu 20.04 with java 11.0.8, everything works just fine. Then I copied it to a Debian Baster (version 10.4), also with java version 1… --- ## [Issue capture request response time in nginx logs](https://discuss.elastic.co/t/issue-capture-request-response-time-in-nginx-logs/245397)
**Author:** [@pravej](https://discuss.elastic.co/u/pravej)\ **Replies:** 1\ **Last updated:** [August 18, 2020, 12:31pm UTC](https://discuss.elastic.co/t/issue-capture-request-response-time-in-nginx-logs/245397 "2020-08-18T12:31:08Z")
Dear Team, I am not able to capture response time in nginx logs . please help to find out gork setting in logstash. i am sharing logs sample I need to capture first cloumn response time "0.241" "0.236" 200 "GET / HTT… --- ## [Logstash has suddenly stopped reading input files](https://discuss.elastic.co/t/logstash-has-suddenly-stopped-reading-input-files/242540)
**Author:** [@mskadu](https://discuss.elastic.co/u/mskadu)\ **Replies:** 7\ **Last updated:** [August 18, 2020, 10:44am UTC](https://discuss.elastic.co/t/logstash-has-suddenly-stopped-reading-input-files/242540 "2020-08-18T10:44:13Z")
Hello everyone, I have an interesting one. I have a working v7.7 logstash config (on RHEL 7.x) which has suddenly stopped ingesting files. I am using the file input plugin in read mode to ingest single-use CSV files (ea… --- ## [Logstash stopped when received x line and not push to elasticsearch](https://discuss.elastic.co/t/logstash-stopped-when-received-x-line-and-not-push-to-elasticsearch/245387)
**Author:** [@yannice92](https://discuss.elastic.co/u/yannice92)\ **Replies:** 0\ **Last updated:** [August 18, 2020, 10:16am UTC](https://discuss.elastic.co/t/logstash-stopped-when-received-x-line-and-not-push-to-elasticsearch/245387 "2020-08-18T10:16:13Z")
I have 2 server. server log and server logstash I using only logstash 6.8 and elasticsearch (without filebeat) from server log I send the log to server logstash using cron. and in server logstash I have multiple pipel… --- ## [Data field in kibana discovery getting overlapped](https://discuss.elastic.co/t/data-field-in-kibana-discovery-getting-overlapped/244434)
**Author:** [@Divya\_Mathur\_ELK](https://discuss.elastic.co/u/Divya_Mathur_ELK)\ **Replies:** 4\ **Last updated:** [August 18, 2020, 9:14am UTC](https://discuss.elastic.co/t/data-field-in-kibana-discovery-getting-overlapped/244434 "2020-08-18T09:14:56Z")
Hi Team, I have created logstash pipeline using 2 csv files and getting output also in kibana. But whenever I am viewing data in kibana , output fields are getting overlapped. Please review below configuration files and… --- ## [How to run Logstash Multiple Pipelines?](https://discuss.elastic.co/t/how-to-run-logstash-multiple-pipelines/245346)
**Author:** [@Amantha](https://discuss.elastic.co/u/Amantha)\ **Replies:** 4\ **Last updated:** [August 18, 2020, 6:31am UTC](https://discuss.elastic.co/t/how-to-run-logstash-multiple-pipelines/245346 "2020-08-18T06:31:19Z")
I configured pipelines.yml file as shown as the image and i created two logstash conf files. I want to know how to run multiple pipelines. --- ## [Getting some documents ahead of my current time that alter the sql\_last\_value](https://discuss.elastic.co/t/getting-some-documents-ahead-of-my-current-time-that-alter-the-sql-last-value/241945)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 4\ **Last updated:** [July 21, 2020, 4:53am UTC](https://discuss.elastic.co/t/getting-some-documents-ahead-of-my-current-time-that-alter-the-sql-last-value/241945 "2020-07-21T04:53:32Z")
I get data from a DB every 30 minutes, in the image the current time is 18:50 pm, I should have no data from 18:30 to current time because this data is collected at 19:00 pm It seems that some documents are misconfigu… --- ## [Aggregate filter not creating path](https://discuss.elastic.co/t/aggregate-filter-not-creating-path/245299)
**Author:** [@kvtang](https://discuss.elastic.co/u/kvtang)\ **Replies:** 2\ **Last updated:** [August 18, 2020, 2:48am UTC](https://discuss.elastic.co/t/aggregate-filter-not-creating-path/245299 "2020-08-18T02:48:52Z")
Hi guys, I have used "aggregate\_maps\_path" to a folder where my logstash certainly have accessed (cause no error and that's where the logging created) but there is no map being created. Any idea why? Much thanks! --- ## [Monitoring the SFTP server availability](https://discuss.elastic.co/t/monitoring-the-sftp-server-availability/245287)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 1\ **Last updated:** [August 18, 2020, 12:29am UTC](https://discuss.elastic.co/t/monitoring-the-sftp-server-availability/245287 "2020-08-18T00:29:07Z")
Is it possible to monitor the health/availability of the SFTP server using elastic Stack? --- ## [Consumer threads for topics\_pattern](https://discuss.elastic.co/t/consumer-threads-for-topics-pattern/245303)
**Author:** [@tarunwadhwa13](https://discuss.elastic.co/u/tarunwadhwa13)\ **Replies:** 3\ **Last updated:** [August 17, 2020, 7:43pm UTC](https://discuss.elastic.co/t/consumer-threads-for-topics-pattern/245303 "2020-08-17T19:43:52Z")
The doc mentioned that consumer thread should be equal to number of partition or divided by number of logstash instances if more than 1 is running. Just wanted to understand how do i calculate the value of consumer thre… --- ## [Solved: Dell EMC REST API calls with Logstash plugin http\_poller](https://discuss.elastic.co/t/solved-dell-emc-rest-api-calls-with-logstash-plugin-http-poller/245283)
**Author:** [@Ryan\_Downey](https://discuss.elastic.co/u/Ryan_Downey)\ **Replies:** 0\ **Last updated:** [August 17, 2020, 4:38pm UTC](https://discuss.elastic.co/t/solved-dell-emc-rest-api-calls-with-logstash-plugin-http-poller/245283 "2020-08-17T16:38:56Z")
If anyone is looking to pull performance metrics from the Dell EMC Unity Unisphere servers that they're running I hope that this saves you some time. Useful References: Dell Unisphere® Management REST API Progra… --- ## [Stuck running docker logstash with pipelines.yml (binding)](https://discuss.elastic.co/t/stuck-running-docker-logstash-with-pipelines-yml-binding/245225)
**Author:** [@nino](https://discuss.elastic.co/u/nino)\ **Replies:** 2\ **Last updated:** [August 17, 2020, 5:03pm UTC](https://discuss.elastic.co/t/stuck-running-docker-logstash-with-pipelines-yml-binding/245225 "2020-08-17T17:03:10Z")
Dear all, i can't get logstash working multipipeline mode inside a docker container. pipelines.yml (inside container) /usr/share/logstash/config bash-4.2$ stat pipelines.yml File: ‘pipelines.yml’ Access: (0644/-rw-… --- ## [How hides the password in json file for jmx input plugin logstash](https://discuss.elastic.co/t/how-hides-the-password-in-json-file-for-jmx-input-plugin-logstash/245166)
**Author:** [@ed2050](https://discuss.elastic.co/u/ed2050)\ **Replies:** 7\ **Last updated:** [August 17, 2020, 4:50pm UTC](https://discuss.elastic.co/t/how-hides-the-password-in-json-file-for-jmx-input-plugin-logstash/245166 "2020-08-17T16:50:56Z")
Hello I am using jmx input plugin logstash to read JVM metrics, everything works fine for me, the JVM I access requires authentication by user and password and in the development environment I have this data burned in t… --- ## [Monitoring the health of the Cassandra NO SQL database](https://discuss.elastic.co/t/monitoring-the-health-of-the-cassandra-no-sql-database/245284)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 0\ **Last updated:** [August 17, 2020, 4:42pm UTC](https://discuss.elastic.co/t/monitoring-the-health-of-the-cassandra-no-sql-database/245284 "2020-08-17T16:42:28Z")
Is it possible to monitor the health of the Cassandra server using Elastic Stack? --- ## [Tips and Best Practices for working / debugging Logstash pipelines on Kubernetes](https://discuss.elastic.co/t/tips-and-best-practices-for-working-debugging-logstash-pipelines-on-kubernetes/245280)
**Author:** [@Xavier\_Krantz](https://discuss.elastic.co/u/Xavier_Krantz)\ **Replies:** 0\ **Last updated:** [August 17, 2020, 4:31pm UTC](https://discuss.elastic.co/t/tips-and-best-practices-for-working-debugging-logstash-pipelines-on-kubernetes/245280 "2020-08-17T16:31:28Z")
Hi everyone :wave: I have setup a BLEK (Beats, Logstash, ElasticSearch, Kibana) stack on Kubernetes :stuck\_out\_tongue: I would like to parse and enrich some logs output from my services collected through Filebeat. Hav… --- ## [Help debugging ingestion flow Filebeat + Logstas](https://discuss.elastic.co/t/help-debugging-ingestion-flow-filebeat-logstas/245277)
**Author:** [@Xavier\_Krantz](https://discuss.elastic.co/u/Xavier_Krantz)\ **Replies:** 0\ **Last updated:** [August 17, 2020, 4:23pm UTC](https://discuss.elastic.co/t/help-debugging-ingestion-flow-filebeat-logstas/245277 "2020-08-17T16:23:27Z")
Hi everybody, I have setup an BLEK (Beats, Logstash, ElasticSearch, Kibana) stack :stuck\_out\_tongue: ES and Kibana are running in ES Cloud a "cluster" of Logstash instances are running on a Kubernetes cluster A servic… --- ## [Logstash Config error](https://discuss.elastic.co/t/logstash-config-error/245274)
**Author:** [@Nur\_Syahirah\_Ashri](https://discuss.elastic.co/u/Nur_Syahirah_Ashri)\ **Replies:** 0\ **Last updated:** [August 17, 2020, 4:03pm UTC](https://discuss.elastic.co/t/logstash-config-error/245274 "2020-08-17T16:03:25Z")
hi, i am new in using ELK. I have these problem that i couldn't find the solution yet. it is the cmd for logstash configuration. i have use many kind of commands but still have the same error. if anyone can help me, pl… --- ## [Logstash com.mysql.jdbc.Driver error](https://discuss.elastic.co/t/logstash-com-mysql-jdbc-driver-error/245256)
**Author:** [@Mohammad\_Mousavi](https://discuss.elastic.co/u/Mohammad_Mousavi)\ **Replies:** 2\ **Last updated:** [August 17, 2020, 3:44pm UTC](https://discuss.elastic.co/t/logstash-com-mysql-jdbc-driver-error/245256 "2020-08-17T15:44:19Z")
Hi. I have logstash 7.4.2 , mysql 5.7 , This is my config: filter { jdbc\_static { loaders =\> \[ { id =\> "products-table" query =\> "select id,name from products;" local\_table =\> "prod… --- ## [S3 input plugin OOM (heap space)](https://discuss.elastic.co/t/s3-input-plugin-oom-heap-space/245267)
**Author:** [@lcaflc](https://discuss.elastic.co/u/lcaflc)\ **Replies:** 0\ **Last updated:** [August 17, 2020, 3:38pm UTC](https://discuss.elastic.co/t/s3-input-plugin-oom-heap-space/245267 "2020-08-17T15:38:20Z")
Hi, I have a big issue with the S3 input plugin. I'm using it to restore files created using the s3 output plugin using gzip compression. When triyng to restore files that are more the 100M it basically eat all JVM hea… --- ## [Logstash add field based on external source](https://discuss.elastic.co/t/logstash-add-field-based-on-external-source/245236)
**Author:** [@Mohammad\_Mousavi](https://discuss.elastic.co/u/Mohammad_Mousavi)\ **Replies:** 2\ **Last updated:** [August 17, 2020, 1:57pm UTC](https://discuss.elastic.co/t/logstash-add-field-based-on-external-source/245236 "2020-08-17T13:57:04Z")
Hi, I have a stream server. I store the logs on elasticsearch. The stream server's log contain requested urls from clients. The logs are similar to nginx logs but with more fields. one of the query params in the url is "… --- ## [Logstash Ruby filter error](https://discuss.elastic.co/t/logstash-ruby-filter-error/245216)
**Author:** [@Apollyon](https://discuss.elastic.co/u/Apollyon)\ **Replies:** 4\ **Last updated:** [August 17, 2020, 3:29pm UTC](https://discuss.elastic.co/t/logstash-ruby-filter-error/245216 "2020-08-17T15:29:17Z")
Hello all, Running Logstash v7.8.1. Ruby codec v3.1.5. Been playing around using the ruby filter plugin. Great filter plugin btw!! I have a couple of logstash confs that have small ruby script filters. While all the f… --- ## [Logstash S3 Input Plugin Error](https://discuss.elastic.co/t/logstash-s3-input-plugin-error/245201)
**Author:** [@elasticus3r](https://discuss.elastic.co/u/elasticus3r)\ **Replies:** 5\ **Last updated:** [August 17, 2020, 2:55pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-error/245201 "2020-08-17T14:55:13Z")
Hello All, Hoping someone could help please. I have got a simple ELk stack running and using s3 input plugin to ingest from an endpoint with the following config: input { s3 { bucket =\> "mybucket" endpoint =\>… --- ## [Filebeat, logstash handle logrotation quesiton/help](https://discuss.elastic.co/t/filebeat-logstash-handle-logrotation-quesiton-help/245254)
**Author:** [@lingfei\_ouyang](https://discuss.elastic.co/u/lingfei_ouyang)\ **Replies:** 0\ **Last updated:** [August 17, 2020, 2:52pm UTC](https://discuss.elastic.co/t/filebeat-logstash-handle-logrotation-quesiton-help/245254 "2020-08-17T14:52:35Z")
Hi, Currently hitting an issues when I setup the filebeat -\> logstash. Here is situation that I'm facing: On filebeat (application) server, if the application log been rotated, (i.e app.log -\> rotate to app.log.08172… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=313) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=315)