Logstash writing to older index

Elastic Stack Logstash
1

Below is the error message from logstash.

The Elasticsearch is designed with hot, warm and cold nodes. Index is created on daily basis.

[2024-10-29T11:14:22,632][INFO ][logstash.outputs.elasticsearch][main][1dde6b9e90686623464913a83aa66d3d68284a23f9523852fe779664b16dd7a1] Retrying failed action {:status=>429, :action=>["index", {:_id=>"717162ethbnbxmansksdjioqdjalmx,amaksdjaknd11", :_index=>"western2-2024.10.18", :routing=>nil}, {"container"=>{"name"=>"service", "id"=>"ape-service-1.2.500-54c4788d55-lnmwb"}, "path"=>"/nfs/logs/ape2/nfs.service+ape-service-1.2.500-54c4788d55-lnmwb", "type"=>"ape2-stats", "labels"=>{"direction"=>"read", "eventglobalid"=>"unknown", "apeeventid"=>19134, "failed"=>"no", "apeseverity"=>"MAJOR", "EventClassID"=>"EventStats", "autoid"=>29928, "autoguid"=>"c174f5c8-6a75-44f4-8e57-0de13a180581", "Severity"=>"Medium", "tenant"=>"12345jajnammm", "source"=>"["dxl://kafka-prod-usw-1a-0.kafka-dev-heeadless.svc.cluster.local:9009,kafka-prod-usw-1a-1.kafka-dev-heeadless.svc.cluster.local:9009,kafka-prod-usw-1a-2.kafka-dev-heeadless.svc.cluster.local:9009/ape.incident.raw/group0/0"]", "Description"=>"Statistical Information Per Event"}, "@version"=>"1", "log"=>{"level"=>"Information"}, "host"=>"logstash-service-49fk8", "fingerprint"=>"717162ethbnbxmansksdjioqdjalmx,amaksdjaknd11", "@timestamp"=>2024-10-18T00:00:50.061Z}], :error=>{"type"=>"cluster_block_exception", "reason"=>"index [western2-2024.10.18] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}}

Below is the index name western2-2024.10.18

why is logstash trying to write to older index western2-2024.10.18. How does it know that it should write events index.

please help.

2

What does your logstash output looks like?

You said that index is created on a daily basis, so your output is probably something like this: western2-%{+YYYY.MM.dd}

So Logstash will use the value of the @timestamp field to populate the YYYY.MM.dd string.

The value of @timestamp field in the event you shared is 2024-10-18T00:00:50.061Z, so tihs is expected.

3

Thanks @leandrojmp. But the day this event was happening is 10/29/2024. [2024-10-29T11:14:22,632][INFO ][logstash.outputs.elasticsearch][main] so why is this happening on 29th?

4

There was not enough disk space on ES side at the moment. As you know ILM automatically manage indices, which means it moves data/shards between hot, warm,cold. LS doesn't have anything with this.
GET _cluster/settings will provide values:

cluster.routing.allocation.disk.watermark.low: (Default 85%)
cluster.routing.allocation.disk.watermark.high: (Default 90%)
cluster.routing.allocation.disk.watermark.flood_stage: (Default 95%)

It's possible that you didn't have enough disk space at the moment, which is the reason why LS recorded the error. Then ILM policy move/delete data from the disk and LS continue to work.

5

This is the date that logstash processed the event.

If you have a date filter that will parse some field from your event and your event has an older date, then this is expected.

Also, your input seems to be a file input reading from some nfs share, so this could mean that Logstash read some old files.

1 Like
6

This style of index naming can cause problems, I've had windows systems get configured without ignore_older and load years of indices.

Use data streams for time series data, @timestamp will still be old, but only the current index will be written. FYI, nobody ever finds this data, they never look for old data. I have some filters for network devices that always log 1980's date at reboot, until they update their time, I just have logic that says "older than 3 days, nope, set @timestamp to now.".