Kibana and @timestamp - out by 4 Months

[peterlarb (Peter)]

I understand that Kibana uses UTC.

Logstash imports a CSV and shows that @timestamp as 27 May 2020 for the file import.

But my data in the rest of the index is showing Jan 2020 and not May 2020.

Now I just can't work it out after so many hours of reading. I have also changed my default timezone to Sydney etc. Can anyone shed some light, please? Thank you Peter

[cheiligers]

Hi @peterlarb
It looks like the ingest timestamp and the index pattern time fields are different.
Can you check how many 'date' fields you have in your index pattern?

I suspect that the index pattern may either have a different date field set as its timestamp field or, possibly, that the date format for the time field in the index pattern might be incorrect.

I'd also check your logstash config file to see if there are any filters or mutations that are running on ingest.

One more thing I'd check is your elasticsearch index mapping for the date field being used as the default date field for the index pattern. In short: compare all the date fields you have and, if there is only one, make sure that the formats are all the same.

One last thing that might be an issue is any ingest pipelines running in ES after logstash has sent the docs to your es instance.

For es:
Queries to issue in the Kibana console (dev tools) (I'm using a sample data set here):
GET kibana_sample_data_flights/_mapping/field/timestamp --> should show date
GET _ingest/pipeline --> will give you all the ingest pipelines

Then in Kibana, you can check your date field in the index pattern from the 'Management -> Index Patterns' page (select the relevant index pattern).

I hope this helps!

[peterlarb (Peter)]

@cheiligers much appreciate your reply as I am new to ELK. Here is the Logstash parser for my csv.
logstash for csv format
filter {
csv {
separator => ","
columns => ["message", "severity", "host", "datetime"]
}
date {
match => [ "datetime" , "DD/MM/YYYY HH:mm" ]
}
}

In regards to the Index pattern the @timestamp was set to Default.

A sample date is: 27/05/2020 14:36

Cheers

[cheiligers]

Your logstash config looks fine.
It might be the @timestamp for your index pattern that's causing issues.
Check what it's set to from the Management -> Index Patterns page. You should also be able to change the format for that:

[peterlarb (Peter)]

@cheiligers yep, I did that and it was set to default.

[peterlarb (Peter)]

@cheiligers

it seems that the issue is ensuring that you have a lower case for DD.

match => [ "datetime" , "DD/MM/YYYY HH:mm" ]

should be:

dd/MM/YYYY HH:mm

Thank you for trying to help me.