Logstash does not use the correct index with Elasticsearch output

Elastic Stack Logstash
1

Hi,

I have an ELK stack which I updated 2 weeks ago from 7.2.0 to 7.16.1. The configuration did not change and is as follow:

# all input will come from filebeat, no local logs
input {
  beats {
    port => 5044
  }
}
filter {
  if [message] =~ /^\{.*\}$/ {
    json {
      source => "message"
    }
  }
  if [ClientHost] {
    geoip {
      source => "ClientHost"
    }
  }
}
output {
  elasticsearch {
      hosts => [ "elasticsearch:9200" ]
      user => "elastic"
      password => "xxx"
      index => "logstash-%{+yyyy.MM.dd}"
  }    
}

Previously, it worked well. However, now as I check, all the logs are written to the index logstash. I could not find where was it set, and I am pretty sure that I didn't change anything.

Does anyone know what is the problem. Thanks a lot in advance.

2

Can you explain better what is the issue?

Your Elasticsearch output has the index option set to logstash-%{yyyy.MM.dd}, so all your logs will be written in daily index starting with logstash- and the date will be extracted from the @timestamp field, which will be generated by logstash when it received an event.

For examples, for events received today, they will be stored in the index logstash-2021.12.28, this is what your configuration does.

3

Hi,

yes, something like logstash-2021.12.28 is exactly what I expect, and it was working like this when I was using 7.2.0. However, right now, all logs are via Logstash are written to the Elasticsearch index named logstash.

4

A quick update from me: I tested by changing the index name to "abc", then the index is created and logs are going to there. I will post my next findings.

6

Ok I found the issue. Took me the whole afternoon:
Correct:
logstash-%{+YYYY.MM.dd}

What I used:
logstash-%{+yyyy.MM.dd}

Anw, it is funny here: in the 7.2. manual, it tells me to use the "correct" one: Elasticsearch output plugin | Logstash Reference [7.2] | Elastic

Meanwhile, the 7.16 manual tells me to use the "wrong" one: Elasticsearch output plugin | Logstash Reference [7.16] | Elastic

????

7

It does not make much sense and this is probably not the cause of your issue as both YYYY and yyyy can be used and will give you the same result, which is the four digit year number, like 2021.

You can check it in the DateTimeFormat class that is used by logstash when parsing the date.

But since it is working now, don' t think if it is worth troubleshooting the issue.

8

In 7.16 I would expect the index option to be ignored because it will have ILM enabled by default. The default rollover alias is "logstash" which is what you say you were seeing.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.