# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=63 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 64 --- ## [Getting 'Badly formatted index, after interpolation still contains placeholder' error when trying to ingest AWS WAF logs](https://discuss.elastic.co/t/getting-badly-formatted-index-after-interpolation-still-contains-placeholder-error-when-trying-to-ingest-aws-waf-logs/340862)
**Author:** [@feo13](https://discuss.elastic.co/u/feo13)\ **Replies:** 3\ **Last updated:** [August 16, 2023, 4:38am UTC](https://discuss.elastic.co/t/getting-badly-formatted-index-after-interpolation-still-contains-placeholder-error-when-trying-to-ingest-aws-waf-logs/340862 "2023-08-16T04:38:19Z")
Hi there, I'm trying to ingest AWS WAF logs with logstash-8.9.0 and send them to my local ELK stack but am getting a 'Badly formatted index, after interpolation still contains placeholder' error. Here's my logstash co… --- ## [How to parse array of objects into separate field](https://discuss.elastic.co/t/how-to-parse-array-of-objects-into-separate-field/340692)
**Author:** [@Subhashini](https://discuss.elastic.co/u/Subhashini)\ **Replies:** 3\ **Last updated:** [August 15, 2023, 4:01pm UTC](https://discuss.elastic.co/t/how-to-parse-array-of-objects-into-separate-field/340692 "2023-08-15T16:01:00Z")
I have some log look like ########2023-08-12######### {‘crewrosters’: \[ { ‘crew\_roster’ : ‘det1’, 'empno': 1} , {‘crew\_roster’ : ‘det2’, 'empno': 2} , {‘crew\_roster’ : ‘det3’, 'empno': 3} \] } I need to parse the data … --- ## [Erro ao executar o logstash](https://discuss.elastic.co/t/erro-ao-executar-o-logstash/340664)
**Author:** [@Gustavo](https://discuss.elastic.co/u/Gustavo)\ **Replies:** 14\ **Last updated:** [August 14, 2023, 6:24pm UTC](https://discuss.elastic.co/t/erro-ao-executar-o-logstash/340664 "2023-08-14T18:24:15Z")
This error persists and I don't know how to solve it anymore. Can someone help me. Follow my .conf files input { file { path =\> "C:/Elastic/logstash-8.9.0/config/logs.log" start\_position =\> "beginning" } } filter … --- ## [Error pulling image configuration: download failed after attempts=6: net/http: TLS handshake timeout](https://discuss.elastic.co/t/error-pulling-image-configuration-download-failed-after-attempts-6-net-http-tls-handshake-timeout/339341)
**Author:** [@ppafford](https://discuss.elastic.co/u/ppafford)\ **Replies:** 5\ **Last updated:** [August 15, 2023, 2:01am UTC](https://discuss.elastic.co/t/error-pulling-image-configuration-download-failed-after-attempts-6-net-http-tls-handshake-timeout/339341 "2023-08-15T02:01:25Z")
Running docker pull logstash:8.8.1 works but running docker pull docker.elastic.co/logstash/logstash:8.8.1 does not Is there a redirect on Elastic side that I'm missing, I've whitelisted a few but feel like I'm still mi… --- ## [Logstash parsing](https://discuss.elastic.co/t/logstash-parsing/339929)
**Author:** [@dilipchiru](https://discuss.elastic.co/u/dilipchiru)\ **Replies:** 4\ **Last updated:** [August 14, 2023, 5:44am UTC](https://discuss.elastic.co/t/logstash-parsing/339929 "2023-08-14T05:44:21Z")
Hi Team, I have 2 Fields which is From and TO which contains set of values which is comma separated. For example: "from" : "Loin, Elephant, cat, movie, John" "to" : "Loin, Elephant, cat, movie, John, USA " Now we wo… --- ## [Error log "Couldn't index event to elastic"](https://discuss.elastic.co/t/error-log-couldnt-index-event-to-elastic/340704)
**Author:** [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Replies:** 1\ **Last updated:** [August 13, 2023, 6:22pm UTC](https://discuss.elastic.co/t/error-log-couldnt-index-event-to-elastic/340704 "2023-08-13T18:22:07Z")
Hi there, I want to confirm, if I got "Could not index event to elasticsearch" error, will it be retried if the reason that log has been resolved? I got this error and the reason shows me it caused by "Limit total field… --- ## [Bin/logstash-plugin not found](https://discuss.elastic.co/t/bin-logstash-plugin-not-found/340574)
**Author:** [@roel82](https://discuss.elastic.co/u/roel82)\ **Replies:** 1\ **Last updated:** [August 11, 2023, 1:03pm UTC](https://discuss.elastic.co/t/bin-logstash-plugin-not-found/340574 "2023-08-11T13:03:21Z")
I have deployed LogStash on kubernetes using this image (logstash:8.8.1) and now I would like to install some plugins. I understand that I would need to use the 'logstash-plugin' tool, whis should be located in the bin … --- ## [If condition for null in json field](https://discuss.elastic.co/t/if-condition-for-null-in-json-field/340475)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 1\ **Last updated:** [August 11, 2023, 12:59pm UTC](https://discuss.elastic.co/t/if-condition-for-null-in-json-field/340475 "2023-08-11T12:59:42Z")
I'm using the JDBC filter to pull info from a SQL database. If the field is blank, it generates the below: "example": \[ { "contoso": "" } \] I've tried the below to remove the empty field, but i… --- ## [I have a older version of Logstash 7.16.2 , is there a output plugin for email. i dont see it for 7.16.2 version](https://discuss.elastic.co/t/i-have-a-older-version-of-logstash-7-16-2-is-there-a-output-plugin-for-email-i-dont-see-it-for-7-16-2-version/339705)
**Author:** [@AKAM14](https://discuss.elastic.co/u/AKAM14)\ **Replies:** 3\ **Last updated:** [August 11, 2023, 11:06am UTC](https://discuss.elastic.co/t/i-have-a-older-version-of-logstash-7-16-2-is-there-a-output-plugin-for-email-i-dont-see-it-for-7-16-2-version/339705 "2023-08-11T11:06:40Z")
Hi Team, I have an older version of logstash 7.16.2 and i need install an Email output plugin for it . Is there a plugin available for this version . I see the 7.17.x versions have the output plugins. while i cannot f… --- ## [Error executing logstash pipeline with jdbc select SQLDataException: ORA-01846: not a valid day of the week](https://discuss.elastic.co/t/error-executing-logstash-pipeline-with-jdbc-select-sqldataexception-ora-01846-not-a-valid-day-of-the-week/340368)
**Author:** [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)\ **Replies:** 4\ **Last updated:** [August 11, 2023, 6:23am UTC](https://discuss.elastic.co/t/error-executing-logstash-pipeline-with-jdbc-select-sqldataexception-ora-01846-not-a-valid-day-of-the-week/340368 "2023-08-11T06:23:14Z")
We have into logstash pipeline the config to search into database and get the data, after the first search we want only select the new data, to do this we use the config of jdbc plugin, my pipeline config. input { jdb… --- ## [Ruby code include?](https://discuss.elastic.co/t/ruby-code-include/340336)
**Author:** [@michaelv](https://discuss.elastic.co/u/michaelv)\ **Replies:** 6\ **Last updated:** [August 11, 2023, 5:48am UTC](https://discuss.elastic.co/t/ruby-code-include/340336 "2023-08-11T05:48:03Z")
Hi All, I have this code that used to be working in ELK 7.12 now that I've upgrade to 8.7.1 it gives a weird error in logstash code =\> " ip\_src = Array.new ip\_… --- ## [Logstash JSON Filter Error](https://discuss.elastic.co/t/logstash-json-filter-error/340496)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 2\ **Last updated:** [August 10, 2023, 2:25pm UTC](https://discuss.elastic.co/t/logstash-json-filter-error/340496 "2023-08-10T14:25:22Z")
I've pulled data from a SQL database that gets put into a field like below. "assignment": \[ { "assignedto": "1234", "assignedtoname": "John Doe", "assignedgroupid": 1 } \] I'm… --- ## [I use elk in docker , i open xpack secuirty ,but when i restart by docker ,there some error log](https://discuss.elastic.co/t/i-use-elk-in-docker-i-open-xpack-secuirty-but-when-i-restart-by-docker-there-some-error-log/340544)
**Author:** [@yichitgo](https://discuss.elastic.co/u/yichitgo)\ **Replies:** 0\ **Last updated:** [August 10, 2023, 10:11am UTC](https://discuss.elastic.co/t/i-use-elk-in-docker-i-open-xpack-secuirty-but-when-i-restart-by-docker-there-some-error-log/340544 "2023-08-10T10:11:45Z")
aiting for Elasticsearch cluster to respond (1/30) logstash started. Starting Kibana5 \[ OK \] touch: cannot touch '/var/log/elasticsearch/{"error":{"root\_cause":\[{"… --- ## [The issue of data corruption in Logstash's Netflow plugin under high data concurrency](https://discuss.elastic.co/t/the-issue-of-data-corruption-in-logstashs-netflow-plugin-under-high-data-concurrency/340541)
**Author:** [@chenlx594](https://discuss.elastic.co/u/chenlx594)\ **Replies:** 0\ **Last updated:** [August 10, 2023, 10:02am UTC](https://discuss.elastic.co/t/the-issue-of-data-corruption-in-logstashs-netflow-plugin-under-high-data-concurrency/340541 "2023-08-10T10:02:38Z")
The Logstash Netflow plugin encounters a problem of misinterpreted fields like first\_switched , last\_switched , and bytes under a netflow data copy rate of 0.4 Gbps. How can this issue be resolved? --- ## [I get this error in Logstash coming even when the pipeline is working just fine. What could it be?](https://discuss.elastic.co/t/i-get-this-error-in-logstash-coming-even-when-the-pipeline-is-working-just-fine-what-could-it-be/340531)
**Author:** [@SamuelSMendes](https://discuss.elastic.co/u/SamuelSMendes)\ **Replies:** 0\ **Last updated:** [August 10, 2023, 9:16am UTC](https://discuss.elastic.co/t/i-get-this-error-in-logstash-coming-even-when-the-pipeline-is-working-just-fine-what-could-it-be/340531 "2023-08-10T09:16:47Z")
\[2023-08-10T06:10:02,974\]\[ERROR\]\[logstash.licensechecker.licensereader\] Unable to retrieve license information from license server {:message=\>"No Available connections"} \[2023-08-10T06:10:06,662\]\[INFO \]\[logstash.license… --- ## [Failed to pull data from Salesforce into logstash](https://discuss.elastic.co/t/failed-to-pull-data-from-salesforce-into-logstash/340503)
**Author:** [@Lazaro\_O\_Farrill](https://discuss.elastic.co/u/Lazaro_O_Farrill)\ **Replies:** 0\ **Last updated:** [August 10, 2023, 3:44am UTC](https://discuss.elastic.co/t/failed-to-pull-data-from-salesforce-into-logstash/340503 "2023-08-10T03:44:35Z")
I am trying to pull my data from my Salesforce sandbox into logstash, and I am getting the following error. Does anyone have any idea what it might mean? I have tested the credentials directly through the API endpoints a… --- ## [How can i update the data in index when i have multiple docementId](https://discuss.elastic.co/t/how-can-i-update-the-data-in-index-when-i-have-multiple-docementid/340167)
**Author:** [@Mohit\_Rajput](https://discuss.elastic.co/u/Mohit_Rajput)\ **Replies:** 1\ **Last updated:** [August 9, 2023, 1:06pm UTC](https://discuss.elastic.co/t/how-can-i-update-the-data-in-index-when-i-have-multiple-docementid/340167 "2023-08-09T13:06:34Z")
How can i update the data in index when i have multiple docementtId? --- ## [Upgrade Elastic Stack 7.15.1 to 7.17.10](https://discuss.elastic.co/t/upgrade-elastic-stack-7-15-1-to-7-17-10/339706)
**Author:** [@SAMY-ELK](https://discuss.elastic.co/u/SAMY-ELK)\ **Replies:** 24\ **Last updated:** [August 9, 2023, 12:50pm UTC](https://discuss.elastic.co/t/upgrade-elastic-stack-7-15-1-to-7-17-10/339706 "2023-08-09T12:50:14Z")
Hello Team, After Upgrade ELK from 7.15.1 to 7.17.10 : logstash-kibana-filebeat-Elastic search , i can't receive log IIS in KIBANA. when i check log logstash i get this error : " LogStash::PipelineAction::Create/pipel… --- ## [JVM for logstash](https://discuss.elastic.co/t/jvm-for-logstash/340424)
**Author:** [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Replies:** 5\ **Last updated:** [August 9, 2023, 12:48pm UTC](https://discuss.elastic.co/t/jvm-for-logstash/340424 "2023-08-09T12:48:48Z")
Hi there, just want to confirm, is there any limit for JVM for logstash? if the JVM limit for elastic is 30 - 32 GB, does it also apply for logstash? Thanks --- ## [Logstash date filter](https://discuss.elastic.co/t/logstash-date-filter/340427)
**Author:** [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Replies:** 10\ **Last updated:** [August 9, 2023, 8:29am UTC](https://discuss.elastic.co/t/logstash-date-filter/340427 "2023-08-09T08:29:42Z")
Hi i have a short\_date field in the following format 09/Aug/2023:12:44:15 +0530 This field is created as text. To convert it to date i am doing the following date { match =\> \[ "short\_date", "dd/MMM/yyyy… --- ## [Unable to configure oracle stored procedure in logstash jdbc pipeline](https://discuss.elastic.co/t/unable-to-configure-oracle-stored-procedure-in-logstash-jdbc-pipeline/340369)
**Author:** [@Sreenivas1](https://discuss.elastic.co/u/Sreenivas1)\ **Replies:** 3\ **Last updated:** [August 9, 2023, 7:30am UTC](https://discuss.elastic.co/t/unable-to-configure-oracle-stored-procedure-in-logstash-jdbc-pipeline/340369 "2023-08-09T07:30:43Z")
Hi all, I'm trying to call stored procedure created in oracle database using logstash jdbc pipeline but even I tried with many ways to pass stored procedure in statement it's getting failed with sql error exceptions . … --- ## [Logstash @timestamp in the input file](https://discuss.elastic.co/t/logstash-timestamp-in-the-input-file/340426)
**Author:** [@Tal\_Blat](https://discuss.elastic.co/u/Tal_Blat)\ **Replies:** 0\ **Last updated:** [August 9, 2023, 7:18am UTC](https://discuss.elastic.co/t/logstash-timestamp-in-the-input-file/340426 "2023-08-09T07:18:37Z")
Hello What will happen if my input file contain a field called @timestamp ? will it replace the logstash @timestamp automatically ? thanks --- ## [Rejected execution of primary operation](https://discuss.elastic.co/t/rejected-execution-of-primary-operation/340391)
**Author:** [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Replies:** 4\ **Last updated:** [August 9, 2023, 2:27am UTC](https://discuss.elastic.co/t/rejected-execution-of-primary-operation/340391 "2023-08-09T02:27:23Z")
Hi there, sometimes my logstash had printed the log that said "rejected execution of primary operation" with the error type "es\_rejected\_execution\_exception" can anyone explain to me what's going on actually? Thanks --- ## [Logstash: MalformedCSVError](https://discuss.elastic.co/t/logstash-malformedcsverror/340330)
**Author:** [@benhartwich](https://discuss.elastic.co/u/benhartwich)\ **Replies:** 9\ **Last updated:** [August 8, 2023, 8:20pm UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330 "2023-08-08T20:20:56Z")
Hi, can anybody help me to find the right mutate =\> gsub definition to avoid these warnings / errors: \[WARN \] 2023-08-08 07:05:15.003 \[\[main\]\>worker20\] csv - Error parsing csv {:field=\>"message", :source=\>"16600,26200,… --- ## [NameError with Cloudwatch plugin in logstash](https://discuss.elastic.co/t/nameerror-with-cloudwatch-plugin-in-logstash/340388)
**Author:** [@rmunjuluri](https://discuss.elastic.co/u/rmunjuluri)\ **Replies:** 0\ **Last updated:** [August 8, 2023, 4:13pm UTC](https://discuss.elastic.co/t/nameerror-with-cloudwatch-plugin-in-logstash/340388 "2023-08-08T16:13:33Z")
Hi, I am trying to pull Cloudwatch logs (specifically EC2 status) into Logstash. I can retrieve the status using AWS\_CLI, but the CloudWatch plugin throws the following error: Pipeline\_id:main Plugin: \ **Author:** [@sta02](https://discuss.elastic.co/u/sta02)\ **Replies:** 1\ **Last updated:** [August 8, 2023, 12:33pm UTC](https://discuss.elastic.co/t/logstash-troubleshooting/340115 "2023-08-08T12:33:35Z") Hello, We have logstash performing dual feed. The first output writes to Elastic and the second output writes to Azure Sentinel. There is a clear delta in number of logs sent to Azure Sentinel and Elastic. Elastic rece… --- ## [Possible Feature Request: Redis Authentication with Username/Password](https://discuss.elastic.co/t/possible-feature-request-redis-authentication-with-username-password/340349)
**Author:** [@alces](https://discuss.elastic.co/u/alces)\ **Replies:** 0\ **Last updated:** [August 8, 2023, 9:25am UTC](https://discuss.elastic.co/t/possible-feature-request-redis-authentication-with-username-password/340349 "2023-08-08T09:25:01Z")
Hi everyone. We are planing to use redis between beats and logstash as a buffer for high utilization timespots. In this setup currently there is only a "password" option for the redis output/input plugin, so every part… --- ## [Combined grok pattern for customized logs](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535)
**Author:** [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Replies:** 20\ **Last updated:** [August 8, 2023, 3:09am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535 "2023-08-08T03:09:23Z")
i am looking some help and guidenace for parsing the customized logs in one file. i have httpd access logs which have two format and i need to prepare the logstash config/filtering the data. so i tried two different pat… --- ## [Fetch substring from a string in logstash filter](https://discuss.elastic.co/t/fetch-substring-from-a-string-in-logstash-filter/340223)
**Author:** [@Neelam\_Zanvar](https://discuss.elastic.co/u/Neelam_Zanvar)\ **Replies:** 1\ **Last updated:** [August 7, 2023, 5:36pm UTC](https://discuss.elastic.co/t/fetch-substring-from-a-string-in-logstash-filter/340223 "2023-08-07T17:36:04Z")
Hi, I have a field called url in elasticsearch document. The sample value for the field is /3dpassport/login I want to extract only the first string before / that is 3dpassport and store it in a field. Tried this copy… --- ## [Logstash Twitter error - no address for stream.twitter.com](https://discuss.elastic.co/t/logstash-twitter-error-no-address-for-stream-twitter-com/340238)
**Author:** [@Yochai\_Ben-Chaim](https://discuss.elastic.co/u/Yochai_Ben-Chaim)\ **Replies:** 1\ **Last updated:** [August 7, 2023, 5:27pm UTC](https://discuss.elastic.co/t/logstash-twitter-error-no-address-for-stream-twitter-com/340238 "2023-08-07T17:27:56Z")
I am trying to use the twitter plugin with the latest ELK stack (8.9.0). When I activate logstash -f myconf\_file.conf I am getting error messages messages : "no address for stream.twitter.com" My conf file is very bas… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=62) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=64)