Auto Detect Date/Time

Elastic Stack Kibana
1

Hello!

So I have records coming in CSV format and they are parsed out properly.

One issue is that a few of the fields have a time or date in them. That time/date is accurate when I look at the raw data but after the record is parsed they change to a default number: 1 day behind and 18:00 for the time. My parsing doesn't change any formatting, it only splits the CSV into fields.

I think the reason is that Kibana auto detects time/date and changes the field type. I do not want Kibana to do this. How can I turn that setting off?

Thanks!

2

What are you using to import those CSV documents? And can you give us an example?

3

I recently discovered the answer to this question incase anyone else has this problem with mainframe SMF records:

Within the SMF record there are fields that contain dates. These dates do not have a time.
Elastic doesn't like that. Why? Because the field type is set to date, which is required by elastic to have a time.
So, elastic automatically assigns a default time to that date.
The default time used by elastic is midnight, UTC timezone.
When Kibana gets the timezone from the browser, which is -0600 , it modifies all times within the record based on that timezone.
So now, instead of midnight "today" it sets it to -0600 , which comes out to 18:00 (6pm) on the previous day.The only two ways to help fix my problem are this:

  1. Re-index making ALL date fields a string field instead.

  2. Create a new string field, in the logstash config, that takes the value from the date field. This needs to be done for every date field (there's like 6) which is why re-indexing is better. (edited)

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.