# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=86 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 87 --- ## [Missing headers even with include\_headers (Csv codec plugin) set to true](https://discuss.elastic.co/t/missing-headers-even-with-include-headers-csv-codec-plugin-set-to-true/327555)
**Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 5\ **Last updated:** [March 14, 2023, 4:34pm UTC](https://discuss.elastic.co/t/missing-headers-even-with-include-headers-csv-codec-plugin-set-to-true/327555 "2023-03-14T16:34:56Z")
Hello World! per Csv codec plugin | Logstash Reference \[7.17\] | Elastic I'm set include\_headers flag to value true, yet even though headers gets included into output on first run, at later time on re-run schedule of ve… --- ## [Logstash S3 input slow ingestion](https://discuss.elastic.co/t/logstash-s3-input-slow-ingestion/327653)
**Author:** [@true64gurus](https://discuss.elastic.co/u/true64gurus)\ **Replies:** 5\ **Last updated:** [March 14, 2023, 3:55pm UTC](https://discuss.elastic.co/t/logstash-s3-input-slow-ingestion/327653 "2023-03-14T15:55:12Z")
I have setup where logstash reads Kubernetes logs from 20 different buckets and send them to ELK. The logs seems to be coming 3-5 minutes late to ELK. The logstash running docker on VM with 31GB Xms/Xmx. I am using one … --- ## [How to set auto-reloading conf files in logstash.yml or pipeline.yml](https://discuss.elastic.co/t/how-to-set-auto-reloading-conf-files-in-logstash-yml-or-pipeline-yml/327425)
**Author:** [@terrymu](https://discuss.elastic.co/u/terrymu)\ **Replies:** 2\ **Last updated:** [March 14, 2023, 9:39am UTC](https://discuss.elastic.co/t/how-to-set-auto-reloading-conf-files-in-logstash-yml-or-pipeline-yml/327425 "2023-03-14T09:39:51Z")
Hi All, I know here is a feature that can auto-reloading conf files without logstash restart action. So my question is easy, how to turn on auto-reloading function in logstash.yml or pipeline.yml ? I need an workin… --- ## [Logstash error failed to install template](https://discuss.elastic.co/t/logstash-error-failed-to-install-template/327521)
**Author:** [@supraja\_inamadugu](https://discuss.elastic.co/u/supraja_inamadugu)\ **Replies:** 2\ **Last updated:** [March 13, 2023, 6:06pm UTC](https://discuss.elastic.co/t/logstash-error-failed-to-install-template/327521 "2023-03-13T18:06:17Z")
\[ERROR\] 2023-03-12 22:42:30.743 \[Ruby-0-Thread-10: /opt/homebrew/Cellar/logstash/8.6.1/libexec/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.12.1-java/lib/logstash/plugin\_mixins/elasticsearch/common.rb:… --- ## [Grok parser and nested brackets](https://discuss.elastic.co/t/grok-parser-and-nested-brackets/327454)
**Author:** [@ddoroshenko](https://discuss.elastic.co/u/ddoroshenko)\ **Replies:** 5\ **Last updated:** [March 13, 2023, 4:03pm UTC](https://discuss.elastic.co/t/grok-parser-and-nested-brackets/327454 "2023-03-13T16:03:57Z")
Hi, I have log event like this 2023-03-03T11:11:11.000Z INFO (foo (bar) bla bla \[bla\]) 2023-03-03T11:11:11.000Z \[foo (bar) bla bla \[bla\]\] I want to parse it with grok filter like timestamp: 2023-03-03T11:11:11.000Z l… --- ## [Posting logs of underlying plugin libraries to Logstash log stream](https://discuss.elastic.co/t/posting-logs-of-underlying-plugin-libraries-to-logstash-log-stream/327105)
**Author:** [@alromos](https://discuss.elastic.co/u/alromos)\ **Replies:** 1\ **Last updated:** [March 13, 2023, 11:15am UTC](https://discuss.elastic.co/t/posting-logs-of-underlying-plugin-libraries-to-logstash-log-stream/327105 "2023-03-13T11:15:16Z")
Is it possible to post logs of underlying libraries to Logstash log stream? For instance, I use input JDBC plugin with MSSQL JDBC driver and I would like to see logs of the driver library for debug purposes. Is it poss… --- ## [Convert a string field to number, but only brand new indices recognized](https://discuss.elastic.co/t/convert-a-string-field-to-number-but-only-brand-new-indices-recognized/327512)
**Author:** [@KeithTt](https://discuss.elastic.co/u/KeithTt)\ **Replies:** 0\ **Last updated:** [March 13, 2023, 3:39am UTC](https://discuss.elastic.co/t/convert-a-string-field-to-number-but-only-brand-new-indices-recognized/327512 "2023-03-13T03:39:55Z")
Logstash version: 6.3.0 Here is my config: mutate { convert =\> { "bytes\_sent" =\> "integer" } } I find that a indice which first created can recognized the config, but the others ones can not, even they created erv… --- ## [Grok (or any alternative) to search for keywords in logs](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351)
**Author:** [@Mark\_S](https://discuss.elastic.co/u/Mark_S)\ **Replies:** 14\ **Last updated:** [March 12, 2023, 12:34pm UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351 "2023-03-12T12:34:26Z")
Hello, Is it possible to create keywords in logstash, by searching for them in the message? The logs are formatted in the following way, however they are not always in the same place - they could be embedded in other m… --- ## [Logstash Keeps Restarting](https://discuss.elastic.co/t/logstash-keeps-restarting/327494)
**Author:** [@kirkofthefleet](https://discuss.elastic.co/u/kirkofthefleet)\ **Replies:** 1\ **Last updated:** [March 12, 2023, 3:06am UTC](https://discuss.elastic.co/t/logstash-keeps-restarting/327494 "2023-03-12T03:06:51Z")
Hello! Please forgive any "syntax errors" as I am a complete newb to all of the elastic stack. I am working on getting elasticstack working for my small IT business. I have a few servers that I am interested in monitor… --- ## [How to detect and avoid UDP input loss](https://discuss.elastic.co/t/how-to-detect-and-avoid-udp-input-loss/327483)
**Author:** [@YvesZhi](https://discuss.elastic.co/u/YvesZhi)\ **Replies:** 1\ **Last updated:** [March 11, 2023, 3:22pm UTC](https://discuss.elastic.co/t/how-to-detect-and-avoid-udp-input-loss/327483 "2023-03-11T15:22:26Z")
I'm using Envoy, which is kind of similar to Nginx, as the gateway of my micro-services backend. Since it's micro-service, there are five Envoys. All of envoys are deployed by Docker and their logs are sent to my Logsta… --- ## [How to cut off the part of syslog](https://discuss.elastic.co/t/how-to-cut-off-the-part-of-syslog/327242)
**Author:** [@YvesZhi](https://discuss.elastic.co/u/YvesZhi)\ **Replies:** 2\ **Last updated:** [March 11, 2023, 6:25am UTC](https://discuss.elastic.co/t/how-to-cut-off-the-part-of-syslog/327242 "2023-03-11T06:25:51Z")
I've some micro services, which are deployed with Docker. They send their logs to my Logstash with the log driver syslog. Here is the config of my Logstash: input { syslog { port =\> 9771 type =\> "syslog" } }… --- ## [Insert multiple fields in nested array](https://discuss.elastic.co/t/insert-multiple-fields-in-nested-array/327415)
**Author:** [@Claudio\_Ract\_Costa](https://discuss.elastic.co/u/Claudio_Ract_Costa)\ **Replies:** 2\ **Last updated:** [March 11, 2023, 1:21am UTC](https://discuss.elastic.co/t/insert-multiple-fields-in-nested-array/327415 "2023-03-11T01:21:59Z")
Hi Guys I have the following input as example: generator { count =\> 1 lines =\> \[ '{ "RATING\_GROUP": "7,843,13", "CONSUMO": "328994,29715,13948" }' \] codec =\> json } Which filter can I use in Logstash to obtain a outp… --- ## [Logstash second conf file is not taking](https://discuss.elastic.co/t/logstash-second-conf-file-is-not-taking/327354)
**Author:** [@ekambaram\_varathan](https://discuss.elastic.co/u/ekambaram_varathan)\ **Replies:** 1\ **Last updated:** [March 10, 2023, 6:47pm UTC](https://discuss.elastic.co/t/logstash-second-conf-file-is-not-taking/327354 "2023-03-10T18:47:42Z")
I am using docker-compose version: v2.9.0. and version: '3.7' in docker-compose yml file and using the below command in logstash section., when i see the logstash logs only its taking the first conf file, second conf fil… --- ## [Can i send message from logstash to pagerduty by http output plugin](https://discuss.elastic.co/t/can-i-send-message-from-logstash-to-pagerduty-by-http-output-plugin/326129)
**Author:** [@YasuhiroOkumura](https://discuss.elastic.co/u/YasuhiroOkumura)\ **Replies:** 1\ **Last updated:** [March 10, 2023, 3:22pm UTC](https://discuss.elastic.co/t/can-i-send-message-from-logstash-to-pagerduty-by-http-output-plugin/326129 "2023-03-10T15:22:25Z")
Are there any having sample code using http output plugin of logstash to send message to pagerduty. Can I see the code. --- ## [Not able to parse geojson data in logstash](https://discuss.elastic.co/t/not-able-to-parse-geojson-data-in-logstash/325247)
**Author:** [@aaryan](https://discuss.elastic.co/u/aaryan)\ **Replies:** 1\ **Last updated:** [March 9, 2023, 3:35pm UTC](https://discuss.elastic.co/t/not-able-to-parse-geojson-data-in-logstash/325247 "2023-03-09T15:35:45Z")
This is the config I am using. input { file { path =\> "D:/Softwares/ELK/data/geojson/features.geojson" start\_position =\> "beginning" sincedb\_path =\> "D:/Softwares/ELK/data/cache/geojsontry.txt" codec =\> mult… --- ## [Logstash log file location](https://discuss.elastic.co/t/logstash-log-file-location/327307)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 2\ **Last updated:** [March 9, 2023, 2:57pm UTC](https://discuss.elastic.co/t/logstash-log-file-location/327307 "2023-03-09T14:57:19Z")
I am running logstash as daemon via systemd I get my log in to my special log dir /log/logstash/logstash-plain.log but I also get that in /var/log/message. I want to stop them and I read that it is control by log4j2 fi… --- ## [Stackoverflow error on logstash when using es\_bulk codec](https://discuss.elastic.co/t/stackoverflow-error-on-logstash-when-using-es-bulk-codec/327337)
**Author:** [@eeijlar](https://discuss.elastic.co/u/eeijlar)\ **Replies:** 1\ **Last updated:** [March 9, 2023, 2:17pm UTC](https://discuss.elastic.co/t/stackoverflow-error-on-logstash-when-using-es-bulk-codec/327337 "2023-03-09T14:17:45Z")
Using the following pipeline with logstash: - pipeline.id: export-process pipeline.workers: 4 config.string: | input { elasticsearch { hosts =\> "http://elastic:80/elasticsearch/… --- ## [Issue with removing tag](https://discuss.elastic.co/t/issue-with-removing-tag/327193)
**Author:** [@Harika](https://discuss.elastic.co/u/Harika)\ **Replies:** 1\ **Last updated:** [March 9, 2023, 1:27pm UTC](https://discuss.elastic.co/t/issue-with-removing-tag/327193 "2023-03-09T13:27:03Z")
we are having the \\\ tag in our XML File. Due to this tag it could not index and throwing the below Error: "error"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"can't merge a non … --- ## [Logstash stopped processing logs after enabling minimal security](https://discuss.elastic.co/t/logstash-stopped-processing-logs-after-enabling-minimal-security/327232)
**Author:** [@A.Hani](https://discuss.elastic.co/u/A.Hani)\ **Replies:** 4\ **Last updated:** [March 9, 2023, 11:02am UTC](https://discuss.elastic.co/t/logstash-stopped-processing-logs-after-enabling-minimal-security/327232 "2023-03-09T11:02:25Z")
I was wondering what should be configured on logstash side after enabling basic on Elasticsearch node? I set x.pack.security.enabled to true on elasticsearch.yml, generated passwords for the cluster users, added the ki… --- ## [Logstash compliance with RFC5425 and RFC5426](https://discuss.elastic.co/t/logstash-compliance-with-rfc5425-and-rfc5426/327243)
**Author:** [@Nikhitha\_Karennagari](https://discuss.elastic.co/u/Nikhitha_Karennagari)\ **Replies:** 2\ **Last updated:** [March 9, 2023, 3:13am UTC](https://discuss.elastic.co/t/logstash-compliance-with-rfc5425-and-rfc5426/327243 "2023-03-09T03:13:49Z")
From the official logstash docs , the syslog output plugin of logstash supports any of RFC5424, RFC3164 formats only. Syslog output plugin | Logstash Reference \[8.6\] | Elastic Does logstash syslog output plugin comply w… --- ## [Logstash syslog message, doesn't choose right if statement](https://discuss.elastic.co/t/logstash-syslog-message-doesnt-choose-right-if-statement/327181)
**Author:** [@splitmessage88](https://discuss.elastic.co/u/splitmessage88)\ **Replies:** 2\ **Last updated:** [March 8, 2023, 2:36pm UTC](https://discuss.elastic.co/t/logstash-syslog-message-doesnt-choose-right-if-statement/327181 "2023-03-08T14:36:26Z")
Hi, I'm trying to create a logstash pipeline for cisco FMC audit log. I have create 3 if statements and would like for logstash to parse the syslog message according to the if statement. Here is two example syslog mes… --- ## [Logstash ignore\_older opposite](https://discuss.elastic.co/t/logstash-ignore-older-opposite/327279)
**Author:** [@GinkoLucas](https://discuss.elastic.co/u/GinkoLucas)\ **Replies:** 1\ **Last updated:** [March 8, 2023, 2:13pm UTC](https://discuss.elastic.co/t/logstash-ignore-older-opposite/327279 "2023-03-08T14:13:47Z")
Hello, I would like Logstash to read only files older than one day. How can I do that? It would be sort of the opposite of "ignore\_older". Thx --- ## [Not able to remove tag from xml](https://discuss.elastic.co/t/not-able-to-remove-tag-from-xml/326771)
**Author:** [@Navya\_04](https://discuss.elastic.co/u/Navya_04)\ **Replies:** 14\ **Last updated:** [March 8, 2023, 8:32am UTC](https://discuss.elastic.co/t/not-able-to-remove-tag-from-xml/326771 "2023-03-08T08:32:22Z")
I am trying to load xml through logstash. I have an unwnated tag which needs to be removed from xml while parsing. Used remove\_tag but not able to remove the tag while indexing to Elasticsearch xml File \ **Author:** [@jfs1](https://discuss.elastic.co/u/jfs1)\ **Replies:** 0\ **Last updated:** [March 6, 2023, 3:45pm UTC](https://discuss.elastic.co/t/logstash-writer-permissions/327099 "2023-03-06T15:45:03Z") On a new on-premises 8.6 logstash+elasticsearch deployment, I have the following error when configuring my "logstash\_writer" role as explained in Secure your connection to Elasticsearch | Logstash Reference \[8.6\] | Elast… --- ## [Extracting year in short format from the log file name](https://discuss.elastic.co/t/extracting-year-in-short-format-from-the-log-file-name/327172)
**Author:** [@lupsya](https://discuss.elastic.co/u/lupsya)\ **Replies:** 1\ **Last updated:** [March 7, 2023, 5:38pm UTC](https://discuss.elastic.co/t/extracting-year-in-short-format-from-the-log-file-name/327172 "2023-03-07T17:38:01Z")
Hello, I am extracting Year, Month, and Day from the following testing log name and converting it to timestamp later. log20230225.log I am using the following grok filter: log%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:d… --- ## [Difference between Timestamp and @timestamp in kibana logs](https://discuss.elastic.co/t/difference-between-timestamp-and-timestamp-in-kibana-logs/327203)
**Author:** [@Amani188](https://discuss.elastic.co/u/Amani188)\ **Replies:** 3\ **Last updated:** [March 7, 2023, 4:33pm UTC](https://discuss.elastic.co/t/difference-between-timestamp-and-timestamp-in-kibana-logs/327203 "2023-03-07T16:33:17Z")
Hi everyone, I noticed that there is a difference of time between Timestamp and @timestamp generated with logstash . Is there a way to synchronise the value of @timestamp to be equal to Timestamp on kibana logs? Thank … --- ## [LogStash - Issue with sql\_last\_value and last\_run\_metadata\_path](https://discuss.elastic.co/t/logstash-issue-with-sql-last-value-and-last-run-metadata-path/327087)
**Author:** [@CedMathis](https://discuss.elastic.co/u/CedMathis)\ **Replies:** 1\ **Last updated:** [March 7, 2023, 11:08am UTC](https://discuss.elastic.co/t/logstash-issue-with-sql-last-value-and-last-run-metadata-path/327087 "2023-03-07T11:08:07Z")
Hello, I'm new to ELK and I'm currently struggling with some setup - maybe I missed a point. I have set up my Logstash to parse my DB (MySql), and I've got 2 cases: "Unforeseen maintenance" -\> In this case, I would… --- ## [Parsing an html inside a Json](https://discuss.elastic.co/t/parsing-an-html-inside-a-json/327104)
**Author:** [@Mhag](https://discuss.elastic.co/u/Mhag)\ **Replies:** 2\ **Last updated:** [March 6, 2023, 11:06pm UTC](https://discuss.elastic.co/t/parsing-an-html-inside-a-json/327104 "2023-03-06T23:06:35Z")
Hi, \*\* a longer explanation of the problem is in the second response to @Badger \*\* I need to parse a log with a JSON that contain a field which contains an HTML document, ex : 2023-03-04 20:20:06,817 \[http-nio-8080-ex… --- ## [Debugging lost data in logstash coming from filebeat](https://discuss.elastic.co/t/debugging-lost-data-in-logstash-coming-from-filebeat/327110)
**Author:** [@mayer](https://discuss.elastic.co/u/mayer)\ **Replies:** 0\ **Last updated:** [March 6, 2023, 5:07pm UTC](https://discuss.elastic.co/t/debugging-lost-data-in-logstash-coming-from-filebeat/327110 "2023-03-06T17:07:51Z")
Dear All, I am running a central ELK stack 8.6.2 with logstash to collect data from some server around. More than 2 years ago I compiled filebeat by myself as it was not available on ARM architecture. With a minimal con… --- ## [Logstash still holding onto deleted logstash application logs](https://discuss.elastic.co/t/logstash-still-holding-onto-deleted-logstash-application-logs/325479)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 3\ **Last updated:** [March 6, 2023, 3:09pm UTC](https://discuss.elastic.co/t/logstash-still-holding-onto-deleted-logstash-application-logs/325479 "2023-03-06T15:09:12Z")
Hello, It seems logstash refuses to let go of deleted logs (logstash's own logs) and this takes up all the space on disks , until a service restart takes place . Is there a way , we could fix this ? Is something need to… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=85) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=87)