Elasticsearch output - index name pattern

Elastic Stack Logstash
1

The elasticsearch output takes a setting for the index name, with a default of 'logstash-YYYY.MM.dd'. Where does the 'YYYY.MM.dd' come from? Are there other options, such as time (e.g. HHmmss), week of year, day of year, locale month name, etc?

Thanks,
John Ouellette

2

Basically when LS sends the request to create the index it just asks for now in UTC.

3

Ok, but that isn't quite what I was asking for. Let's say I wanted to create an index that is named by the year, week of year, and day of week (ok, stupid example). If I assume that the default 'YYYY.MM.dd' specification comes from the same Joda DateTimeFormat as the logstash date filter plugin, my ugly index name in the elasticsearch output would be:

index => "logstash-%{+YYYY.ww.ee}"

So, is the format specification the Joda DateTimeFormat, or something completely different?

Are there other options? With the %{} format, can I access other logstash terms, like the type, or, I don't know, the host name, etc.?

Sorry if this isn't clear :slight_smile:

4

Basically when LS sends the request to create the index it just asks for now in UTC.

Surely it uses the @timestamp field?

5

So, is the format specification the Joda DateTimeFormat, or something completely different?

No, it's Joda-Time alright.

Are there other options? With the %{} format, can I access other logstash terms, like the type, or, I don't know, the host name, etc.?

Yes, you can access any field. Here's the relevant part of the documentation: Accessing event data and fields | Logstash Reference [8.11] | Elastic

6

Surely it uses the @timestamp field?

Indeed it does. Filed a PR to clarify this in the documentation:

7

Thanks Magnus!

8

Is it possible to tell it to use a different date field, other than @timestamp?

9

Is it possible to tell it to use a different date field, other than @timestamp?

No.

10

Thanks.

I came up with a work-around to use this, and drop or rename @timestamp, if anyone else needs it.

filter {
    mutate {
        add_field => {
            "[@metadata][indexDate]" => "%{+YYYY.MM.dd}"
        }
        rename => {
            "@timestamp" => "receivedTimestamp"
        }
    }
}
output {
    elasticsearch {
        hosts => [
            'elastic01.example.com:9200',
            'elastic02.example.com:9200',
            'elastic03.example.com:9200'
        ]
        index => "logstash-%{[@metadata][indexDate]}"
    }
}