Logstash date offset issue and indicies


(Pirate Trkmn) #1

Hi guys i have two problem cant solve.

1.i have log_time field and want to load it @timestamp with date filter. but there is time difference after i load it to @timestamp example: log_time is 09:34, but @timestamp is 05:34. i know @timestamp work on utc timezone, now how can solve this problem and load correct time to @timestamp

2.I have very big indicies nearly 3000, and it is working very slow how can i customize it.


(Christian Dahlqvist) #2

As you pointed out, the @timestamp field has to be in UTC timezone, so I am not sure what you mean by correcting it. Which timezone is the log_time field in? What would you expect @timestamp to be?

Having very large number of indices and/or shards can be very inefficient and cause performance problems. Please read this blog post for some practical guidelines.


(Pirate Trkmn) #3

Thanks for reply. log_time is in Australia/Sydney and now how can i load it to @timestamp


(Christian Dahlqvist) #4

Are you specifying the correct timezone in the Logstash date filter? What does your config look like?


(Pirate Trkmn) #5

yes
date {
match => ["log_timestamp", "yyyy-MM-dd HH:mm:ss" ]
timezone => "Australia/Sydney"
}


(Christian Dahlqvist) #6

Can you show an example of a full event?


(Pirate Trkmn) #8

yes please:
{
"_index": "iis-2017.08.01",
"_type": "doc",
"_id": "HOJce2YBnfWJqk2v0LFB",
"_version": 1,
"_score": null,
"_source": {
"path": "/etc/logstash/logs/prod_user_login/user_logins_prod.csv",
"Login Date": "2/08/2017 09:11",
"type": "prod_user_login",
"@timestamp": "2017-08-01T23:11:00.000Z",
"tags": [
"_grokparsefailure",
"_geoip_lookup_failure",
"PROD_USER_LOGIN"
],
"UserID": "1857",
"User Type": "HLRA_Client",
"@version": "1",
"host": "elk-on-ubuntu",
"Role Name": "Diversa Superannuation Services"
},
"fields": {
"@timestamp": [
"2017-08-01T23:11:00.000Z"
]
},
"sort": [
1501629060000
]
}


(Pirate Trkmn) #9

and it is filter which i use
filter {
if [type] == "prod_user_login" {
csv {
separator => ","
columns => ["UserID","User Type","Role Name","Login Date"]
add_tag => [ "PROD_USER_LOGIN" ]
remove_field => ["message"]
}
date {
match => ["Login Date", "dd/MM/yyyy HH:mm"]
}
}
}


(Christian Dahlqvist) #10

That looks like the correct UTC timestamp corresponding to the Sydney timezone timestamp. As timestamps in Elasticsearch has to be in UTC, I do not see what the problem is.


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.