# Logstash reading events from March 14th on March 16th **URL:** \ **Category:** Logstash\ **Created:** [March 16, 2025, 5:41pm UTC](https://discuss.elastic.co/t/logstash-reading-events-from-march-14th-on-march-16th/375982 "2025-03-16T17:41:24Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![devops\_training](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devops_training/32/131845_2.png) [@devops\_training](https://discuss.elastic.co/u/devops_training)\ **Post date:** [March 16, 2025, 5:41pm UTC](https://discuss.elastic.co/t/logstash-reading-events-from-march-14th-on-march-16th/375982/1 "2025-03-16T17:41:24Z")
```auto [2025-03-16T00:07:37,943][INFO][logstash.outputs.opensearch][main][f636cb73983bcc650332a7ed143a2c1655e1be6017f7947da7fc02b194adad2d] Retrying failed action {:status=>429, :action=>["index", {:_id=>nil, :_index=>"http_server-2025.03.14", :routing=>nil}, {"log"=>{"file"=>{"path"=>"C:/Program Files (x86)/Server/logs/localhost_access_log.log"}}, "event"=>{"hash"=>"c60225b5468e886bff692d09664d5e686b269476", "original"=>"[14/Mar/2025:11:26:10 +0000] 0:0:0:0:0:0:0:1 GET /remote/core.list-plugins HTTP/1.1 7354 200 [http-nio-8080-exec-1] [F2FC4D635FAE927072C1A24B03BD5F55.route1] admin 109ms\r"}, "url"=>{"uripath"=>"/remote/core.list-plugins"}, "@version"=>"1", "source"=>{"address"=>"0:0:0:0:0:0:0:1"}, "nodeRole"=>"http_server", "process"=>{"name"=>"http-nio-8080-exec-1"}, "http"=>{"user"=>"admin", "version"=>"1.1", "referrer"=>"F2FC4D635FAE927072C1A24B03BD5F55.route1", "response"=>{"body"=>{"bytes"=>7354}, "status_code"=>200, "time"=>109}, "request"=>{"method"=>"GET"}}, "message"=>"[14/Mar/2025:11:26:10 +0000] 0:0:0:0:0:0:0:1 GET /remote/core.list-plugins HTTP/1.1 7354 200 [http-nio-8080-exec-1] [F2FC4D635FAE927072C1A24B03BD5F55.route1] admin 109ms\r", "podName"=>"ntt", "type"=>"localhost_access_log", "host"=>{"name"=>"http_sever"}, "partition"=>"ntt0", "@timestamp"=>2025-03-14T11:26:10.000Z, "pr_name"=>"server-po"}], :error=>{"type"=>"cluster_block_exception", "reason"=>"index [http_server-2025.03.14] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}} ``` ```auto input { file { path => "C:/Program Files (x86)/Server/logs/localhost_access_log.log*" type => "localhost_access_log" start_position => "beginning" ignore_older => 86400 # ignore files older than 24 hours close_older => 86400 # free the resources } file { path => "C:/Program Files (x86)/Server/logs/orlog.log*" type => "orlog" codec => multiline { pattern => "^%{TIMESTAMP_ISO8601}" negate => true what => "previous" charset => "ISO-8859-1" } start_position => "beginning" ignore_older => 86400 # ignore files older than 24 hours close_older => 86400 # free the resources } } filter { # Process logs of type localhost_access_log if [type] == "localhost_access_log" { grok { match => { "message" => [ # Pattern 2: Matches structured HTTP logs "\[%{HAPROXYDATE:[@metadata][timestamp]}\] %{IPORHOST:[source][address]} %{WORD:[http][request][method]} %{URIPATH:[url][uripath]} HTTP/%{NUMBER:[http][version]} (?:-|%{INT:[http][response][body][bytes]:int}) %{INT:[http][response][status_code]:int} \[%{DATA:[process][name]}\] \[(?:-|%{DATA:[http][referrer]})\] (?:-|%{WORD:[http][user]}) %{NUMBER:[http][response][time]:int}ms", # Pattern 1: Matches logs with timestamp + message "^\[%{HTTPDATE:[@metadata][timestamp]}\] %{GREEDYDATA:log_message}" ] } } # Convert extracted `[@metadata][timestamp]` to `@timestamp` date { match => ["[@metadata][timestamp]", "dd/MMM/yyyy:HH:mm:ss Z" ] timezone => "UTC" target => "@timestamp" tag_on_failure => ["_dateparsefailure"] } # Debugging: Log any date parsing failures if "_dateparsefailure" in [tags] { mutate { add_field => { "failed_timestamp" => "%{[@metadata][timestamp]}" } } } # Handle Grok failures gracefully if "_grokparsefailure" in [tags] { mutate { add_field => { "log_message_raw" => "%{message}" } remove_tag => ["_grokparsefailure"] } } } # Process logs of type orlog if [type] == "orlog" { grok { match => { "message" => ["%{TIMESTAMP_ISO8601:[@metadata][timestamp]} %{LOGLEVEL:[log][level]}%{SPACE}%{GREEDYDATA:msg}" ] } } date { match => ["[@metadata][timestamp]", "ISO8601"] timezone => "UTC" target => "@timestamp" } } } ``` ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [March 16, 2025, 5:50pm UTC](https://discuss.elastic.co/t/logstash-reading-events-from-march-14th-on-march-16th/375982/2 "2025-03-16T17:50:31Z")
`logstash.outputs.opensearch` from GitHub page: The logstash-output-opensearch plugin helps to ship events from Logstash to OpenSearch cluster. This is not an OpenSearch forum, and that plugin is not supported by Elastic. btw, the config you shared does not appear to have an output section? ---
**Author:** ![devops\_training](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devops_training/32/131845_2.png) [@devops\_training](https://discuss.elastic.co/u/devops_training)\ **Post date:** [March 16, 2025, 5:52pm UTC](https://discuss.elastic.co/t/logstash-reading-events-from-march-14th-on-march-16th/375982/3 "2025-03-16T17:52:47Z")
@RainTown You are right we are shipping to opensearch. This is a POC. I want to understand the error. We are using logstash downloaded from elastic. Please shed some knowledge. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [March 16, 2025, 6:22pm UTC](https://discuss.elastic.co/t/logstash-reading-events-from-march-14th-on-march-16th/375982/4 "2025-03-16T18:22:05Z")
> [@devops\_training](#): > > Please shed some knowledge. What? Good luck, maybe someone else helps you. I tend to avoid those that dont ask a clear question, just post a naked config, and miss out key details. I will point you to `disk usage exceeded flood-stage watermark, index has read-only-allow-delete block` If you are sending to _opensearch_, thats an error from _opensearch_. i.e. your disk is too close to full and was not accepting more documents. ---
**Author:** ![devops\_training](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devops_training/32/131845_2.png) [@devops\_training](https://discuss.elastic.co/u/devops_training)\ **Post date:** [March 16, 2025, 6:28pm UTC](https://discuss.elastic.co/t/logstash-reading-events-from-march-14th-on-march-16th/375982/5 "2025-03-16T18:28:58Z")
@RainTown I have provided logstash configuration and the error message. Except the output config. Shared the output as well. we were performing POC with logstash to push logs to opensearch. ```auto output { opensearch { ecs_compatibility => disabled hosts => ["https:// *************************** :443"] ssl => true index => " **************** -%{+YYYY.MM.dd}" } } } ``` As I downloaded logstash from this forum I reached out you guys. No other thoughts. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 16, 2025, 6:30pm UTC](https://discuss.elastic.co/t/logstash-reading-events-from-march-14th-on-march-16th/375982/6 "2025-03-16T18:30:54Z")
> [@devops\_training](#): > > "reason"=\>"index [http\_server-2025.03.14] blocked by: [TOO\_MANY\_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}}` You can read about what you would do if elasticsearch returned that error [here](https://discuss.elastic.co/t/error-disk-usage-exceeded-flood-stage-watermark-index-has-read-only-allow-delete-block/323357/2). If you want advice specific to opensearch then post in their support forum. ---
**Author:** ![devops\_training](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devops_training/32/131845_2.png) [@devops\_training](https://discuss.elastic.co/u/devops_training)\ **Post date:** [March 16, 2025, 6:37pm UTC](https://discuss.elastic.co/t/logstash-reading-events-from-march-14th-on-march-16th/375982/7 "2025-03-16T18:37:43Z")
@Badger and @RainTown Thanks for the help. We have ISM policy write to index is permitted for only 2 days. hence the `[TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}}` File rotation happens everyday C:/Program Files (x86)/Server/logs/localhost\_access\_log.log C:/Program Files (x86)/Server/logs/localhost\_access\_log.2025-03-16 C:/Program Files (x86)/Server/logs/localhost\_access\_log.2025-03-15 C:/Program Files (x86)/Server/logs/localhost\_access\_log.2025-03-14 The content it is trying to read is from file C:/Program Files (x86)/Server/logs/localhost\_access\_log.2025-03-14 `[14/Mar/2025:11:26:10 +0000] 0:0:0:0:0:0:0:1 GET /remote/core.list-plugins HTTP/1.1 7354 200 [http-nio-8080-exec-1] [F2FC4D635FAE927072C1A24B03BD5F55.route1] admin 109ms ` The question and I m not able to understand is why did logstash pick this event on march 16th ```auto { "persistent": { "cluster": { "ignore_dot_indexes": "true", "routing": { "allocation": { "disk": { "watermark": { "low": "24.561262130737305gb", "flood_stage": "9.824504852294922gb", "high": "21.613910675048828gb" } } } }, "metadata": { "perf_analyzer": { "state": "0" } } }, "search": { "max_buckets": "10000", "query": {} }, "plugins": { "index_state_management": { "template_migration": { "control": "-1" } }, "query": { "executionengine": {} } }, "search_backpressure": { "node_duress": { "heap_threshold": "0.85" } }, "cluster_manager": { "throttling": { "thresholds": { "remove-index-template-v2": { "value": "50" }, "delete-pipeline": { "value": "200" }, "restore-snapshot": { "value": "50" }, "delete-repository": { "value": "50" }, "create-data-stream": { "value": "50" }, "put-script": { "value": "200" }, "create-snapshot": { "value": "50" }, "update-task-state": { "value": "200" }, "index-aliases": { "value": "200" }, "delete-index": { "value": "50" }, "update-snapshot-state": { "value": "5000" }, "rollover-index": { "value": "200" }, "create-index": { "value": "50" }, "update-settings": { "value": "50" }, "delete-snapshot": { "value": "50" }, "remove-data-stream": { "value": "50" }, "delete-script": { "value": "200" }, "remove-persistent-task": { "value": "200" }, "auto-create": { "value": "200" }, "cluster-reroute-api": { "value": "50" }, "delete-dangling-index": { "value": "50" }, "cluster-update-settings": { "value": "50" }, "create-index-template": { "value": "50" }, "create-index-template-v2": { "value": "50" }, "put-repository": { "value": "50" }, "put-mapping": { "value": "10000" }, "create-persistent-task": { "value": "200" }, "finish-persistent-task": { "value": "200" }, "create-component-template": { "value": "50" }, "remove-component-template": { "value": "50" }, "put-pipeline": { "value": "200" }, "remove-index-template": { "value": "50" } } } } }, "transient": { "cluster": { "routing": { "allocation": { "disk": { "watermark": { "low": "24.561262130737305gb", "flood_stage": "9.824504852294922gb", "high": "21.613910675048828gb" } } } } }, "search_backpressure": { "node_duress": { "heap_threshold": "0.85" } } } } ``` I have checked logstash-plain.log no errors on march 14 and march 15th. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [March 16, 2025, 6:50pm UTC](https://discuss.elastic.co/t/logstash-reading-events-from-march-14th-on-march-16th/375982/8 "2025-03-16T18:50:52Z")
> [@devops\_training](#): > > I have checked logstash-plain.log no errors on march 14 and march 15th. Have you checked _opensearch_ logs? btw the (single) log entry you posted said " **Retrying** failed action". ---
**Author:** ![devops\_training](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devops_training/32/131845_2.png) [@devops\_training](https://discuss.elastic.co/u/devops_training)\ **Post date:** [March 16, 2025, 7:01pm UTC](https://discuss.elastic.co/t/logstash-reading-events-from-march-14th-on-march-16th/375982/9 "2025-03-16T19:01:39Z")
@RainTown Yeah I checked there is no issue with opensearch. Request you to shed some light how to handle file input during file rotation. I removed output opensearch section. start\_position =\> end or sincepath specifically needs to be specified. Thought in my mind is why did logstash read march 14th event on 16th and why only this one event. Added ```auto output{stdout { codec => rubydebug }} ``` I could not reproduce. I ran it in debug mode. ---
**Author:** ![devops\_training](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devops_training/32/131845_2.png) [@devops\_training](https://discuss.elastic.co/u/devops_training)\ **Post date:** [March 24, 2025, 10:56am UTC](https://discuss.elastic.co/t/logstash-reading-events-from-march-14th-on-march-16th/375982/10 "2025-03-24T10:56:47Z")
Thanks team. I read about the deduplication issue. [I Want to remove the duplicate events inside Logstash filter how could I do that? I mention the events below please have a look and suggest](https://discuss.elastic.co/t/i-want-to-remove-the-duplicate-events-inside-logstash-filter-how-could-i-do-that-i-mention-the-events-below-please-have-a-look-and-suggest/349175#:~:text=If%20you%20really,Decidely%20non%2Dtrivial). Duplicate events are overwritten. If you have ILM policy where you can write to index for 2 days and if logstash will try to write that index ( duplciate events are updated not dropped) we need to find a way to drop the event if it is overwriting or index based on logstash timestamp no issue will be seen, if it is from timestamp generated from the event metadata then issue starts. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [March 24, 2025, 12:48pm UTC](https://discuss.elastic.co/t/logstash-reading-events-from-march-14th-on-march-16th/375982/11 "2025-03-24T12:48:05Z")
> [@devops\_training](#): > > we need to find a way to drop the event if it is overwriting or index based on logstash timestamp no issue will be seen, if it is from timestamp generated from the event metadata then issue starts. I have not quite followed. Is above sentence missing some words or punctuation? Maybe you can re-word to make it clearer. For sure logstash cannot know (without significant effort) if an event it would send to opensearch will be accepted or not. It cannot know about your ISM policy. Same as with elasticsearch/ILM. What you could do, I guess, is have a filter in logstash that just drops "old" events, for some definition of "old". But that sort of thing carries significant risks.