I use Logstash for ingestion in elastic but my index not creating index daily based on date
my config logstsh
input {
beats {
port => "5044"
}
}
filter {
csv {
separator => ","
skip_header => "true"
columns => ["chaine", "job", "date_plan", "statut", "date_debut", "date_fin", "serveur", "numero_passage", "application", "sous_application"]
}
date {
match => [ "date_plan" , "YYYY-MM-dd" ]
timezone => "Europe/Paris"
}
date {
match => [ "date_debut" , "YYYY-MM-dd HH:mm:ss" ]
timezone => "Europe/Paris"
}
date {
match => [ "date_fin" , "YYYY-MM-dd HH:mm:ss" ]
timezone => "Europe/Paris"
}
mutate {
convert => { "numero_passage" => "integer" }
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "test-%{+YYYY.MM.dd}"
stdout {
codec => rubydebug
}
}
A_B
June 4, 2020, 2:21pm
2
Hi @Youssef_SBAI ,
do you see any errors on stdout?
The only thing I can think of is that you are missing the @timestamp field...
From the docs
LS uses Joda to format the index pattern from event timestamp.
No i have any errror in stdout
A_B
June 4, 2020, 2:35pm
4
Do you have a field named @timestamp in the stdout output?
What are your indices named? If there is no error there should have been some indices created...
yes i have a field named @timestamp in the stdout output
I have auther index created test- 2020.05.29 test- 2020.05.28 test- 2020.05.27 ..........
A_B
June 5, 2020, 7:18am
7
These index names
Match exactly your Logstash output config with
Those are the daily indices. The date for the index name is taken from the @timestamp field, not current time on the Logstash machine.
Could you share the content of one of those @timestamp fields
The content of @timestamp fields is
A_B
June 5, 2020, 7:34am
9
Thanks
Those logs would end up in an index with the name test-2020.05.29, no matter when they pass through Logstash.
I have an other content with the index
test-2020.05.29
it is a problem of @timestamp fields?
A_B
June 5, 2020, 7:59am
12
If you want to call it a problem , I guess yes.
As I posted before
event timestamp = @timestamp field.
If you would rather have now (current time) instead of the event timestamp, you can replace @timestamp with a filter. Simply removing @timestamp might have the same result, not 100% sure about that.
I use Joda to format the index: index => "<logstash-{now/d}>" Thank You
system
July 6, 2020, 6:17am
14
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
