Timezone causing problem when doing a search query to an index!

Elastic Stack Logstash
1

I'm trying to find out the results from a search query (ie: searching results for the given date range) of a particular index. So that I could get the results in a daily basis.

This is the query : http://localhost:9200/dialog_test/_search?q=timestamp:[2016-08-03T00:00:00.128%20TO%202016-08-03T23:59:59.128]

In the above, timestamp is a field which i added using my logstash.conf in order to get the actual log time. When i tried querying this, surprisingly i got a number of hits (total hits: 24) which should've been 0 since I didn't have any log records from the date of (2016-08-03) . It actually displays the count for the next day (ie: (2016-08-04), which has 24 records in the log file. I'm sure something has gone wrong with the timezone.

My timezone is GMT+5:30,

Here is my filtering part of logstash conf:

filter {		
	grok {
			patterns_dir => ["D:/ELK Stack/logstash/logstash-2.3.4/bin/patterns"]
			match => { "message" => "^%{LOGTIMESTAMP:logtimestamp}%{GREEDYDATA}" }			
	}
        mutate {
			add_field => { "timestamp" => "%{logtimestamp}" }
			remove_field => ["logtimestamp"]
	}
	date {
			match => [ "timestamp" , "ISO8601" , "yyyyMMdd HH:mm:ss.SSS" ]
			target => "timestamp"
			locale => "en"
	}}

I could provide the log if needed. Where am i going wrong? Any help could be appreciated.

2

The timezone will be stored as UTC, so if you want to get events during a calendar day in local time you have to adjust your query accordingly. If you make the query using the query DSL rather than a query string you can use a range query and set the timezone option to make ES do the timezone adjustment for you.

1 Like
3

Thanks for the reply. I was doing the querying (GET request) through postman without using the CURL.

If you make the query using the query DSL rather than a query string you can use a range query and set the timezone option to make ES do the timezone adjustment for you.

I didn't quite understand how could i make the query using query DSL if I'm going to test it through Postman kinda thing and how am I going to use the range query in there for a GET request?

4

Inserting the timezone as UTC within the date filter did the trick.

Updated date filter:

date {
	match => [ "timestamp" , "ISO8601" , "yyyyMMdd HH:mm:ss.SSS" ]
	target => "timestamp"
	locale => "en"
        timezone => "UTC"
}
5

That's the wrong solution to the problem since you'll be storing local time as UTC while ES and Kibana continue to assume that the stored timestamps are UTC when they're in fact local time.

1 Like
6

@magnusbaeck Thanks again. Now i'm confused. :joy:

How could I make both ES & Kibana tally with the same timezone so I don't mess with it?

I didn't quite understand how could i make the query using query DSL if I'm going to test it through Postman kinda thing and how am I going to use the range query in there for a GET request?

How can I proceed with the above ?

7

How could I make both ES & Kibana tally with the same timezone so I don't mess with it?

Insert the timestamps as UTC (the date filter will do the conversion for you based on the server's timezone, but otherwise you can override it with the timezone option) and either use UTC for all other direct interaction with ES. In some cases, like the range query's timezone option you can specify which timezone you want things.

How can I proceed with the above ?

The ES documentation contains plenty of pages with examples about the query DSL.

1 Like