Problem with @timestamp

Elastic Stack Elasticsearch
1

I use logstash for ingest in elasstic a CSV file, i have a error from the time stamps is not the daily date
my config logstash is

input {


file {
    path => "/data/volumes/monitoring/logstash/logCtrlM/stock.csv"
    sincedb_path => "/dev/null"
    start_position => "beginning"
    }
}

filter {

csv {
    separator => ","
    columns => [  "Univers",  "Entrepot", "Sous application", "Chaine", "Job", "Statut", "couleur", "date plan", "numero passage", "satatut global", "couleurG" ]
    skip_header => "true"
}


date {
    match => [ "date plan" , "YYYY-MM-dd" ]
    timezone => "Europe/Paris"
}

mutate {
    convert => { "numero passage" => "integer" }
}

mutate {
    convert => { "couleur" => "integer" }
}
 
mutate {
    convert => { "couleurG" => "integer" }
} 

}

output {
    elasticsearch {
    hosts => "http://elasticsearch:9200"
    index => "<logstash-{now/d}>"

       }

    stdout {
        codec => rubydebug
    }
}

image
image1244×396 13 KB

2

It looks good to me. Midnight in Paris is 22h00 in GMT/UTC.

3

the @timestamp " => 2020-06-16T22:00:00.000Z is not good is not the dailly date ?

4

What do you mean by daily date? "La date du jour"?
The date when the event happened or when logstash is running?

5

Yes it is "La date du jour"

6

for exemple my index is logstsh-2020.06.18 but the timestamp is @timestamp " => 2020-06-16T22:00:00.000Z

7

This is expected because you wrote this:

date {
    match => [ "date plan" , "YYYY-MM-dd" ]
    timezone => "Europe/Paris"
}

This reads the content of the text field date plan and parse it as a date, according the format and time zone you set and write the computed date to @timestamp. Which overwrites the date of the event.

Then you wrote that:

elasticsearch {
    index => "<logstash-{now/d}>"
}

This means that the index name is generated not by using the value of the @timestamp field but the current date. The default value for index is logstash-%{+yyyy.MM.dd} which uses the date of the event. See https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.