# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=119 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 120 --- ## [Remove symbol from logstash](https://discuss.elastic.co/t/remove-symbol-from-logstash/312998)
**Author:** [@travlest](https://discuss.elastic.co/u/travlest)\ **Replies:** 1\ **Last updated:** [August 26, 2022, 3:59pm UTC](https://discuss.elastic.co/t/remove-symbol-from-logstash/312998 "2022-08-26T15:59:04Z")
hi, i have a string like this - \['127.0.53.53'\] and \['104.21.57.219', '172.67.192.115'\] i would like to remove the symbol \[' xx '\] I have tried something like this, but it seems fail. can anybody help me? mutate { gs… --- ## [Logstash -\> Elastic Timeouts](https://discuss.elastic.co/t/logstash-elastic-timeouts/312713)
**Author:** [@6igwig](https://discuss.elastic.co/u/6igwig)\ **Replies:** 5\ **Last updated:** [August 26, 2022, 7:17am UTC](https://discuss.elastic.co/t/logstash-elastic-timeouts/312713 "2022-08-26T07:17:13Z")
I have 2 logstash nodes with about 30 different pipelines sending data to a 6 node elastic cluster. For about half of my pipelines I am seeing a ton of errors like this in the /var/log/logstash/pipeline\_mypipeline.log fi… --- ## [Knowing when an event message get sent to pubsub](https://discuss.elastic.co/t/knowing-when-an-event-message-get-sent-to-pubsub/312987)
**Author:** [@Altiano\_Gerung](https://discuss.elastic.co/u/Altiano_Gerung)\ **Replies:** 0\ **Last updated:** [August 26, 2022, 4:14am UTC](https://discuss.elastic.co/t/knowing-when-an-event-message-get-sent-to-pubsub/312987 "2022-08-26T04:14:01Z")
I'm using this plugin to send kafka events to pubsub, I'm running to some issue where I need to know when the message arrived at logstash and when it was attempted to be delivered to pubsub. When I tried to enable log… --- ## [How is log4j-1.2.17 jar being installed on my system](https://discuss.elastic.co/t/how-is-log4j-1-2-17-jar-being-installed-on-my-system/312971)
**Author:** [@brilong](https://discuss.elastic.co/u/brilong)\ **Replies:** 1\ **Last updated:** [August 26, 2022, 1:10am UTC](https://discuss.elastic.co/t/how-is-log4j-1-2-17-jar-being-installed-on-my-system/312971 "2022-08-26T01:10:16Z")
I am not very familiar with logstash but I am trying to determine why the file /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-log4j-3.1.3-java/vendor/jar-dependencies/runtime-jars/log4j-1.2.17.jar exis… --- ## [How do I match a newline in logstash grok](https://discuss.elastic.co/t/how-do-i-match-a-newline-in-logstash-grok/311596)
**Author:** [@yts85205107](https://discuss.elastic.co/u/yts85205107)\ **Replies:** 5\ **Last updated:** [August 25, 2022, 10:50pm UTC](https://discuss.elastic.co/t/how-do-i-match-a-newline-in-logstash-grok/311596 "2022-08-25T22:50:48Z")
this is my log sample as below: \[2022-08-07T15:57:54+08:00\] 9.9.9.9 "Request-Method-URL: GET XXX Sex - Free Porn Videos on XXX.com" "Status-Code: 200" "Request-Length: 1" "Request-Time: 1.23" "Upstream-Server: 1.1.1.1:1… --- ## [Failed to install template](https://discuss.elastic.co/t/failed-to-install-template/312949)
**Author:** [@Queren\_Santos](https://discuss.elastic.co/u/Queren_Santos)\ **Replies:** 1\ **Last updated:** [August 25, 2022, 5:58pm UTC](https://discuss.elastic.co/t/failed-to-install-template/312949 "2022-08-25T17:58:07Z")
Can anyone help? My template in Elasticsearch is ready, but Logstash does not send the index to which it is already created: ERROR\]\[logstash.outputs.elasticsearch\]\[main\] Failed to install template {:message=\>"Got respon… --- ## [Logstash only logs the word 'log' on completion](https://discuss.elastic.co/t/logstash-only-logs-the-word-log-on-completion/312916)
**Author:** [@usman1](https://discuss.elastic.co/u/usman1)\ **Replies:** 3\ **Last updated:** [August 25, 2022, 5:03pm UTC](https://discuss.elastic.co/t/logstash-only-logs-the-word-log-on-completion/312916 "2022-08-25T17:03:14Z")
I am using the following input configurations for deleting the files once read and log the information. input { file { path =\> "C:/.log" mode =\> "read" file\_completed\_action =\> "log\_and\_delete" file\_co… --- ## [Invalid Index Name Exception](https://discuss.elastic.co/t/invalid-index-name-exception/312908)
**Author:** [@hnf](https://discuss.elastic.co/u/hnf)\ **Replies:** 2\ **Last updated:** [August 25, 2022, 4:42pm UTC](https://discuss.elastic.co/t/invalid-index-name-exception/312908 "2022-08-25T16:42:29Z")
Hello everyone, I have an issue when I try to make an index a field created using add\_field by referencing them when outputing. I create three .conf files that begin by a number , like 1file.conf, 2file.conf and 99outp… --- ## [Logstash IndexOutOfBoundsException: writerIndex + minWritableBytes exceeds maxCapacity](https://discuss.elastic.co/t/logstash-indexoutofboundsexception-writerindex-minwritablebytes-exceeds-maxcapacity/312943)
**Author:** [@cotjoey](https://discuss.elastic.co/u/cotjoey)\ **Replies:** 0\ **Last updated:** [August 25, 2022, 3:37pm UTC](https://discuss.elastic.co/t/logstash-indexoutofboundsexception-writerindex-minwritablebytes-exceeds-maxcapacity/312943 "2022-08-25T15:37:17Z")
Hello, Lately my logstash instance has been throwing the following exceptions intermittently. I cannot find a solution anywhere. All the posts in this forum with the string "writerIndex" with similar issues are left una… --- ## [Logstash http input plugin and azure load balancer](https://discuss.elastic.co/t/logstash-http-input-plugin-and-azure-load-balancer/312940)
**Author:** [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\ **Replies:** 0\ **Last updated:** [August 25, 2022, 3:25pm UTC](https://discuss.elastic.co/t/logstash-http-input-plugin-and-azure-load-balancer/312940 "2022-08-25T15:25:25Z")
I am not able to get the http input plugin to connect to our azure load balancer. The input input { http { host =\> "igemsng-eastus2-nprd-elkcluster.test.att.com" port =\> 8808 password =\> XXXX… --- ## [MongoDB to Elasticsearch synchronisation using Logstash JDBC Input plugin](https://discuss.elastic.co/t/mongodb-to-elasticsearch-synchronisation-using-logstash-jdbc-input-plugin/312934)
**Author:** [@hrasheed90](https://discuss.elastic.co/u/hrasheed90)\ **Replies:** 0\ **Last updated:** [August 25, 2022, 2:27pm UTC](https://discuss.elastic.co/t/mongodb-to-elasticsearch-synchronisation-using-logstash-jdbc-input-plugin/312934 "2022-08-25T14:27:53Z")
I have defined several pipelines (each pipeline represents one to one relation with mongodb collection to elasticsearch index i.e. each mongodb collection will have its own index in elasticsearch). all my pipelines look… --- ## [Deleting processed objects from S3 when using Beats input (and Filebeat with aws-s3 input)](https://discuss.elastic.co/t/deleting-processed-objects-from-s3-when-using-beats-input-and-filebeat-with-aws-s3-input/312922)
**Author:** [@G.E](https://discuss.elastic.co/u/G.E)\ **Replies:** 0\ **Last updated:** [August 25, 2022, 1:15pm UTC](https://discuss.elastic.co/t/deleting-processed-objects-from-s3-when-using-beats-input-and-filebeat-with-aws-s3-input/312922 "2022-08-25T13:15:37Z")
Hi, Can anyone suggest a (hopefully uncomplicated) way to delete processed log files from S3 when using Filebeat and Logstash with SQS and S3? I had previously used the Logstash S3 plugin alone which does support this … --- ## [Sending Log from Fluentd to Filebeat or Logstash](https://discuss.elastic.co/t/sending-log-from-fluentd-to-filebeat-or-logstash/312829)
**Author:** [@yago82](https://discuss.elastic.co/u/yago82)\ **Replies:** 2\ **Last updated:** [August 25, 2022, 8:49am UTC](https://discuss.elastic.co/t/sending-log-from-fluentd-to-filebeat-or-logstash/312829 "2022-08-25T08:49:53Z")
Having to take logs from Rhel 5.x machines and not being able to install neither Logstash nor Filebeat. I opted for the installation of Fluentd v0.12. So the question is: is it possible to send logs from Fluentd (Server … --- ## [Logstash filter elasticsearch aggregation](https://discuss.elastic.co/t/logstash-filter-elasticsearch-aggregation/312883)
**Author:** [@Moshe\_Sharon1](https://discuss.elastic.co/u/Moshe_Sharon1)\ **Replies:** 0\ **Last updated:** [August 25, 2022, 8:11am UTC](https://discuss.elastic.co/t/logstash-filter-elasticsearch-aggregation/312883 "2022-08-25T08:11:44Z")
Using logstash I am trying to filter elasticsearch index to get one of few documents with same value thanks --- ## [Needs to drop audit beat data](https://discuss.elastic.co/t/needs-to-drop-audit-beat-data/312676)
**Author:** [@rajvel](https://discuss.elastic.co/u/rajvel)\ **Replies:** 1\ **Last updated:** [August 25, 2022, 4:08am UTC](https://discuss.elastic.co/t/needs-to-drop-audit-beat-data/312676 "2022-08-25T04:08:49Z")
Hello Team, Have configured the auditbeat with file integraity module to capture the folder/file date and time change. In the same server have configured the filebeat to capture the application logs. both beats output… --- ## [\[warn \]\[logstash.config.source.multilocal\] ignoring the 'pipelines.yml' file because modules or command line options are specified](https://discuss.elastic.co/t/warn-logstash-config-source-multilocal-ignoring-the-pipelines-yml-file-because-modules-or-command-line-options-are-specified/312836)
**Author:** [@Vitoria\_De\_Lara](https://discuss.elastic.co/u/Vitoria_De_Lara)\ **Replies:** 0\ **Last updated:** [August 24, 2022, 6:38pm UTC](https://discuss.elastic.co/t/warn-logstash-config-source-multilocal-ignoring-the-pipelines-yml-file-because-modules-or-command-line-options-are-specified/312836 "2022-08-24T18:38:36Z")
input { file { path =\> "/tmp/spreadsheet\_data.csv" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { csv { separator =\> "," skip\_header =\> "true" columns =\> \['Name','Class','Dorm','Room','… --- ## [LogStash::ConfigurationError", :message=\>"Expected one of \[ \\\\t\\\\r\\\\n\], \\"#\\", \\"if\\", \[A-Za-z0-9\_-\], '\\"', \\"'\\", \\"}\\"](https://discuss.elastic.co/t/logstash-configurationerror-message-expected-one-of-t-r-n-if-a-za-z0-9/312402)
**Author:** [@Queren\_Santos](https://discuss.elastic.co/u/Queren_Santos)\ **Replies:** 12\ **Last updated:** [August 24, 2022, 3:33pm UTC](https://discuss.elastic.co/t/logstash-configurationerror-message-expected-one-of-t-r-n-if-a-za-z0-9/312402 "2022-08-24T15:33:18Z")
I'm having trouble understanding my code configuration error. Shows the following message when trying to run the logs and their settings: \[ERROR\]\[logstash.agent\] Failed to execute action {:action=\>LogStash::PipelineActi… --- ## [MongoDB (5+, json based) : some objects have variable types](https://discuss.elastic.co/t/mongodb-5-json-based-some-objects-have-variable-types/312445)
**Author:** [@LaBonave](https://discuss.elastic.co/u/LaBonave)\ **Replies:** 3\ **Last updated:** [August 24, 2022, 2:30pm UTC](https://discuss.elastic.co/t/mongodb-5-json-based-some-objects-have-variable-types/312445 "2022-08-24T14:30:51Z")
Hi (new here) I've successfully managed to send all of our mongodb 5 instances (JSON-based) logs, via filebeat, on a Logstash instance. MongoDB 5+ are JSON-based logs (https://www.mongodb.com/docs/manual/reference/log… --- ## [Logstash - Grok Syntax Issues](https://discuss.elastic.co/t/logstash-grok-syntax-issues/312807)
**Author:** [@RaiZiStyle](https://discuss.elastic.co/u/RaiZiStyle)\ **Replies:** 6\ **Last updated:** [August 24, 2022, 2:17pm UTC](https://discuss.elastic.co/t/logstash-grok-syntax-issues/312807 "2022-08-24T14:17:25Z")
I'm using filebeat to send log to logstash but I'm having issues with grok syntax on Logstash. I used the grok debugger on Kibanna and manager to come to a solution. The problem is that I can't find the same syntax for … --- ## [Print arabic characters in logstash](https://discuss.elastic.co/t/print-arabic-characters-in-logstash/312793)
**Author:** [@Didi\_Sisi](https://discuss.elastic.co/u/Didi_Sisi)\ **Replies:** 0\ **Last updated:** [August 24, 2022, 10:31am UTC](https://discuss.elastic.co/t/print-arabic-characters-in-logstash/312793 "2022-08-24T10:31:52Z")
Hello, I created a python script which gathers tweets using “Tweepy”. Then, I return each tweet in a certain dict format. My tweets may contain arabic and latin caracters. This is my python script: import tweepy impor… --- ## [Logstash does not parse if 'if' condition change](https://discuss.elastic.co/t/logstash-does-not-parse-if-if-condition-change/312700)
**Author:** [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Replies:** 5\ **Last updated:** [August 24, 2022, 8:59am UTC](https://discuss.elastic.co/t/logstash-does-not-parse-if-if-condition-change/312700 "2022-08-24T08:59:55Z")
Using single node ELK cluster version 7.16.3. Index won't appear in kibana if i used this configuration in logstash pipeline file output { if \[tags\] == "average\_weight" { elasticsearch { hosts =\> \["http://localhos… --- ## [HAProxy conf for Logstash](https://discuss.elastic.co/t/haproxy-conf-for-logstash/312773)
**Author:** [@magnorod](https://discuss.elastic.co/u/magnorod)\ **Replies:** 0\ **Last updated:** [August 24, 2022, 7:41am UTC](https://discuss.elastic.co/t/haproxy-conf-for-logstash/312773 "2022-08-24T07:41:35Z")
Hello, I'm contacting you because I can't get the load balancing of Logstash via HAproxy to work properly. I always have 2 Logstash out of the 4 that share the flow. I would like the flow to be fairly distributed betwe… --- ## [Merge two Json arrays in logstash+ruby](https://discuss.elastic.co/t/merge-two-json-arrays-in-logstash-ruby/312759)
**Author:** [@deep08](https://discuss.elastic.co/u/deep08)\ **Replies:** 0\ **Last updated:** [August 24, 2022, 5:54am UTC](https://discuss.elastic.co/t/merge-two-json-arrays-in-logstash-ruby/312759 "2022-08-24T05:54:56Z")
Hello , I want to merge two json array based on one id matching fields and create an new array in logstash , I have two Json array data1,data2 below - "data1 "{ "tenant":"0", "city":"FLORENCE", "id":"AXY798", "stat… --- ## [Problem logstash runner](https://discuss.elastic.co/t/problem-logstash-runner/312730)
**Author:** [@Vitoria\_De\_Lara](https://discuss.elastic.co/u/Vitoria_De_Lara)\ **Replies:** 0\ **Last updated:** [August 23, 2022, 6:00pm UTC](https://discuss.elastic.co/t/problem-logstash-runner/312730 "2022-08-23T18:00:58Z")
I'm trying to run logstash with this command, I would like to know if the syntax is right, and if not what would be the rule to run logstash. command /usr/share/logstash/bin/logstash --config.test\_and\_exit -f /etc/logs… --- ## [Missing rsyslog/logstash data: rescan?](https://discuss.elastic.co/t/missing-rsyslog-logstash-data-rescan/312706)
**Author:** [@cvcv](https://discuss.elastic.co/u/cvcv)\ **Replies:** 0\ **Last updated:** [August 23, 2022, 12:46pm UTC](https://discuss.elastic.co/t/missing-rsyslog-logstash-data-rescan/312706 "2022-08-23T12:46:55Z")
Hi, I'm an ELK newbie. I configured rsyslogd and sending logs via the json template into logstash. Things worked fine for awhile but my index got "full". I lost a few days of logs. The rsyslog data/files are still t… --- ## [Error: Index pattern does not contain any geospatial fields](https://discuss.elastic.co/t/error-index-pattern-does-not-contain-any-geospatial-fields/312686)
**Author:** [@Ricard-CT](https://discuss.elastic.co/u/Ricard-CT)\ **Replies:** 0\ **Last updated:** [August 23, 2022, 8:34am UTC](https://discuss.elastic.co/t/error-index-pattern-does-not-contain-any-geospatial-fields/312686 "2022-08-23T08:34:32Z")
Hello everyone, I'm trying to use the geo\_ip filter on my logstash, but it is not working. When I try to create the map, this error appears: "Index pattern does not contain any geospatial fields" The problem is tha… --- ## [Slow Inexing and ES Cluster Getting Throttled after some time](https://discuss.elastic.co/t/slow-inexing-and-es-cluster-getting-throttled-after-some-time/312678)
**Author:** [@arjun\_thakur](https://discuss.elastic.co/u/arjun_thakur)\ **Replies:** 0\ **Last updated:** [August 23, 2022, 7:23am UTC](https://discuss.elastic.co/t/slow-inexing-and-es-cluster-getting-throttled-after-some-time/312678 "2022-08-23T07:23:27Z")
I am using ELK stack 7.10.0. The cluster has 7 nodes and each node is assigned approx 30 GB JVM heap. Over the past few weeks, the indexing speed is getting extremely slow. Before that System was able to ingest 700 GB o… --- ## [Logstash currently bundles openjdk 11.0.15 but 11.0.16.1 is released](https://discuss.elastic.co/t/logstash-currently-bundles-openjdk-11-0-15-but-11-0-16-1-is-released/312626)
**Author:** [@lunarfs](https://discuss.elastic.co/u/lunarfs)\ **Replies:** 1\ **Last updated:** [August 23, 2022, 1:45am UTC](https://discuss.elastic.co/t/logstash-currently-bundles-openjdk-11-0-15-but-11-0-16-1-is-released/312626 "2022-08-23T01:45:04Z")
Hi, I was under the impression that Logstash would bundle the latest Eclipse Temurin jdk when a new version was released. this however does not sem to be the case. I find it a bit hard to figure what determines what dec… --- ## [Logstash did not listen to port 5044 after startup，but listened to port 9600](https://discuss.elastic.co/t/logstash-did-not-listen-to-port-5044-after-startup-but-listened-to-port-9600/312595)
**Author:** [@YodHao](https://discuss.elastic.co/u/YodHao)\ **Replies:** 4\ **Last updated:** [August 23, 2022, 1:41am UTC](https://discuss.elastic.co/t/logstash-did-not-listen-to-port-5044-after-startup-but-listened-to-port-9600/312595 "2022-08-23T01:41:26Z")
Hello, I would like to ask what is wrong with my logstash configuration. I configured the port to listen to 5044 in the pipeline.yml file, but only 9600 is monitored after logstash is started; the port 5044 is not monito… --- ## [Logstash Lumberjack Output](https://discuss.elastic.co/t/logstash-lumberjack-output/312646)
**Author:** [@6igwig](https://discuss.elastic.co/u/6igwig)\ **Replies:** 1\ **Last updated:** [August 22, 2022, 7:05pm UTC](https://discuss.elastic.co/t/logstash-lumberjack-output/312646 "2022-08-22T19:05:51Z")
In the documentation for lumberjack output, under the ssl\_certificate setting it just says, "This is a required setting" and "ssl certificate to use." My questions are: what is this certificate used for, and why is it r… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=118) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=120)