Logstash reading events from March 14th on March 16th

Elastic Stack Logstash
1 
[2025-03-16T00:07:37,943][INFO ][logstash.outputs.opensearch][main][f636cb73983bcc650332a7ed143a2c1655e1be6017f7947da7fc02b194adad2d] Retrying failed action {:status=>429, :action=>["index", {:_id=>nil, :_index=>"http_server-2025.03.14", :routing=>nil}, {"log"=>{"file"=>{"path"=>"C:/Program Files (x86)/Server/logs/localhost_access_log.log"}}, "event"=>{"hash"=>"c60225b5468e886bff692d09664d5e686b269476", "original"=>"[14/Mar/2025:11:26:10 +0000] 0:0:0:0:0:0:0:1 GET /remote/core.list-plugins HTTP/1.1 7354 200 [http-nio-8080-exec-1] [F2FC4D635FAE927072C1A24B03BD5F55.route1] admin 109ms\r"}, "url"=>{"uripath"=>"/remote/core.list-plugins"}, "@version"=>"1", "source"=>{"address"=>"0:0:0:0:0:0:0:1"}, "nodeRole"=>"http_server", "process"=>{"name"=>"http-nio-8080-exec-1"}, "http"=>{"user"=>"admin", "version"=>"1.1", "referrer"=>"F2FC4D635FAE927072C1A24B03BD5F55.route1", "response"=>{"body"=>{"bytes"=>7354}, "status_code"=>200, "time"=>109}, "request"=>{"method"=>"GET"}}, "message"=>"[14/Mar/2025:11:26:10 +0000] 0:0:0:0:0:0:0:1 GET /remote/core.list-plugins HTTP/1.1 7354 200 [http-nio-8080-exec-1] [F2FC4D635FAE927072C1A24B03BD5F55.route1] admin 109ms\r", "podName"=>"ntt", "type"=>"localhost_access_log", "host"=>{"name"=>"http_sever"}, "partition"=>"ntt0", "@timestamp"=>2025-03-14T11:26:10.000Z, "pr_name"=>"server-po"}], :error=>{"type"=>"cluster_block_exception", "reason"=>"index [http_server-2025.03.14] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}}

input {
  file {
    path => "C:/Program Files (x86)/Server/logs/localhost_access_log.log*"
    type => "localhost_access_log"
    start_position => "beginning"
    ignore_older => 86400   # ignore files older than 24 hours
    close_older => 86400  # free the resources 
  }
  file {
    path => "C:/Program Files (x86)/Server/logs/orlog.log*"
    type => "orlog"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
      charset => "ISO-8859-1"
    }
    start_position => "beginning"
    ignore_older => 86400   # ignore files older than 24 hours
    close_older => 86400  # free the resources  
  }
}

filter {
  # Process logs of type localhost_access_log
  if [type] == "localhost_access_log" {
    grok {
      match => {
        "message" => [
         # Pattern 2: Matches structured HTTP logs
          "\[%{HAPROXYDATE:[@metadata][timestamp]}\] %{IPORHOST:[source][address]} %{WORD:[http][request][method]} %{URIPATH:[url][uripath]} HTTP/%{NUMBER:[http][version]} (?:-|%{INT:[http][response][body][bytes]:int}) %{INT:[http][response][status_code]:int} \[%{DATA:[process][name]}\] \[(?:-|%{DATA:[http][referrer]})\] (?:-|%{WORD:[http][user]}) %{NUMBER:[http][response][time]:int}ms",
		  # Pattern 1: Matches logs with timestamp + message
          "^\[%{HTTPDATE:[@metadata][timestamp]}\] %{GREEDYDATA:log_message}"
        ]
      }
    }

    # Convert extracted `[@metadata][timestamp]` to `@timestamp`
    date {
      match => [ "[@metadata][timestamp]", "dd/MMM/yyyy:HH:mm:ss Z" ]
      timezone => "UTC"
      target => "@timestamp"
      tag_on_failure => ["_dateparsefailure"]
    }

    # Debugging: Log any date parsing failures
    if "_dateparsefailure" in [tags] {
      mutate {
        add_field => { "failed_timestamp" => "%{[@metadata][timestamp]}" }
      }
    }

    # Handle Grok failures gracefully
    if "_grokparsefailure" in [tags] {
      mutate {
        add_field => { "log_message_raw" => "%{message}" }
        remove_tag => ["_grokparsefailure"]
      }
    }
  }

  # Process logs of type orlog
  if [type] == "orlog" {
    grok {
      match => {
        "message" => [ "%{TIMESTAMP_ISO8601:[@metadata][timestamp]} %{LOGLEVEL:[log][level]}%{SPACE}%{GREEDYDATA:msg}" ]
      }
    }
    date {
      match => ["[@metadata][timestamp]", "ISO8601"]
      timezone => "UTC" 
      target => "@timestamp"
    }
  }
}
2

logstash.outputs.opensearch

from GitHub page:

The logstash-output-opensearch plugin helps to ship events from Logstash to OpenSearch cluster.

This is not an OpenSearch forum, and that plugin is not supported by Elastic.

btw, the config you shared does not appear to have an output section?

3

@RainTown You are right we are shipping to opensearch. This is a POC. I want to understand the error. We are using logstash downloaded from elastic. Please shed some knowledge.

4

What? Good luck, maybe someone else helps you. I tend to avoid those that dont ask a clear question, just post a naked config, and miss out key details.

I will point you to

disk usage exceeded flood-stage watermark, index has read-only-allow-delete block

If you are sending to opensearch, thats an error from opensearch. i.e. your disk is too close to full and was not accepting more documents.

5

@RainTown I have provided logstash configuration and the error message. Except the output config. Shared the output as well.
we were performing POC with logstash to push logs to opensearch.

output {
    opensearch {
      ecs_compatibility => disabled
      hosts => ["https://***************************:443"]
      ssl => true
      index => "****************-%{+YYYY.MM.dd}"
    }
  }
}

As I downloaded logstash from this forum I reached out you guys. No other thoughts.

6

You can read about what you would do if elasticsearch returned that error here. If you want advice specific to opensearch then post in their support forum.

7

@Badger and @RainTown Thanks for the help.

We have ISM policy write to index is permitted for only 2 days. hence the [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}}

File rotation happens everyday

C:/Program Files (x86)/Server/logs/localhost_access_log.log
C:/Program Files (x86)/Server/logs/localhost_access_log.2025-03-16
C:/Program Files (x86)/Server/logs/localhost_access_log.2025-03-15
C:/Program Files (x86)/Server/logs/localhost_access_log.2025-03-14

The content it is trying to read is from file C:/Program Files (x86)/Server/logs/localhost_access_log.2025-03-14
[14/Mar/2025:11:26:10 +0000] 0:0:0:0:0:0:0:1 GET /remote/core.list-plugins HTTP/1.1 7354 200 [http-nio-8080-exec-1] [F2FC4D635FAE927072C1A24B03BD5F55.route1] admin 109ms
The question and I m not able to understand is why did logstash pick this event on march 16th

{
  "persistent": {
    "cluster": {
      "ignore_dot_indexes": "true",
      "routing": {
        "allocation": {
          "disk": {
            "watermark": {
              "low": "24.561262130737305gb",
              "flood_stage": "9.824504852294922gb",
              "high": "21.613910675048828gb"
            }
          }
        }
      },
      "metadata": {
        "perf_analyzer": {
          "state": "0"
        }
      }
    },
    "search": {
      "max_buckets": "10000",
      "query": {}
    },
    "plugins": {
      "index_state_management": {
        "template_migration": {
          "control": "-1"
        }
      },
      "query": {
        "executionengine": {}
      }
    },
    "search_backpressure": {
      "node_duress": {
        "heap_threshold": "0.85"
      }
    },
    "cluster_manager": {
      "throttling": {
        "thresholds": {
          "remove-index-template-v2": {
            "value": "50"
          },
          "delete-pipeline": {
            "value": "200"
          },
          "restore-snapshot": {
            "value": "50"
          },
          "delete-repository": {
            "value": "50"
          },
          "create-data-stream": {
            "value": "50"
          },
          "put-script": {
            "value": "200"
          },
          "create-snapshot": {
            "value": "50"
          },
          "update-task-state": {
            "value": "200"
          },
          "index-aliases": {
            "value": "200"
          },
          "delete-index": {
            "value": "50"
          },
          "update-snapshot-state": {
            "value": "5000"
          },
          "rollover-index": {
            "value": "200"
          },
          "create-index": {
            "value": "50"
          },
          "update-settings": {
            "value": "50"
          },
          "delete-snapshot": {
            "value": "50"
          },
          "remove-data-stream": {
            "value": "50"
          },
          "delete-script": {
            "value": "200"
          },
          "remove-persistent-task": {
            "value": "200"
          },
          "auto-create": {
            "value": "200"
          },
          "cluster-reroute-api": {
            "value": "50"
          },
          "delete-dangling-index": {
            "value": "50"
          },
          "cluster-update-settings": {
            "value": "50"
          },
          "create-index-template": {
            "value": "50"
          },
          "create-index-template-v2": {
            "value": "50"
          },
          "put-repository": {
            "value": "50"
          },
          "put-mapping": {
            "value": "10000"
          },
          "create-persistent-task": {
            "value": "200"
          },
          "finish-persistent-task": {
            "value": "200"
          },
          "create-component-template": {
            "value": "50"
          },
          "remove-component-template": {
            "value": "50"
          },
          "put-pipeline": {
            "value": "200"
          },
          "remove-index-template": {
            "value": "50"
          }
        }
      }
    }
  },
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "disk": {
            "watermark": {
              "low": "24.561262130737305gb",
              "flood_stage": "9.824504852294922gb",
              "high": "21.613910675048828gb"
            }
          }
        }
      }
    },
    "search_backpressure": {
      "node_duress": {
        "heap_threshold": "0.85"
      }
    }
  }
}

I have checked logstash-plain.log no errors on march 14 and march 15th.

8

Have you checked opensearch logs? btw the (single) log entry you posted said "Retrying failed action".

9

@RainTown Yeah I checked there is no issue with opensearch.
Request you to shed some light how to handle file input during file rotation.
I removed output opensearch section.
start_position => end or sincepath specifically needs to be specified.

Thought in my mind is why did logstash read march 14th event on 16th and why only this one event.

Added

output{stdout { codec => rubydebug }}

I could not reproduce. I ran it in debug mode.

10

Thanks team.

I read about the deduplication issue.

I Want to remove the duplicate events inside Logstash filter how could I do that? I mention the events below please have a look and suggest.

Duplicate events are overwritten.
If you have ILM policy where you can write to index for 2 days and if logstash will try to write that index ( duplciate events are updated not dropped)

we need to find a way to drop the event if it is overwriting or index based on logstash timestamp no issue will be seen, if it is from timestamp generated from the event metadata then issue starts.

11

I have not quite followed. Is above sentence missing some words or punctuation? Maybe you can re-word to make it clearer.

For sure logstash cannot know (without significant effort) if an event it would send to opensearch will be accepted or not. It cannot know about your ISM policy. Same as with elasticsearch/ILM.

What you could do, I guess, is have a filter in logstash that just drops "old" events, for some definition of "old". But that sort of thing carries significant risks.