Logstash date offset issue and indicies

[Pirate_Trkmn]

Hi guys i have two problem cant solve.

1.i have log_time field and want to load it @timestamp with date filter. but there is time difference after i load it to @timestamp example: log_time is 09:34, but @timestamp is 05:34. i know @timestamp work on utc timezone, now how can solve this problem and load correct time to @timestamp

2.I have very big indicies nearly 3000, and it is working very slow how can i customize it.

[Christian_Dahlqvist]

@Pirate_Trkmn

1.i have log_time field and want to load it @timestamp with date filter. but there is time difference after i load it to @timestamp example: log_time is 09:34, but @timestamp is 05:34. i know @timestamp work on utc timezone, now how can solve this problem and load correct time to @timestamp

As you pointed out, the @timestamp field has to be in UTC timezone, so I am not sure what you mean by correcting it. Which timezone is the log_time field in? What would you expect @timestamp to be?

@Pirate_Trkmn

2.I have very big indicies nearly 3000, and it is working very slow how can i customize it.

Having very large number of indices and/or shards can be very inefficient and cause performance problems. Please read this blog post for some practical guidelines.

[Pirate_Trkmn]

Thanks for reply. log_time is in Australia/Sydney and now how can i load it to @timestamp

[Christian_Dahlqvist]

Are you specifying the correct timezone in the Logstash date filter? What does your config look like?

[Pirate_Trkmn]

yes
date {
match => ["log_timestamp", "yyyy-MM-dd HH:mm:ss" ]
timezone => "Australia/Sydney"
}

[Christian_Dahlqvist]

Can you show an example of a full event?

[Pirate_Trkmn]

yes please:
{
"_index": "iis-2017.08.01",
"_type": "doc",
"_id": "HOJce2YBnfWJqk2v0LFB",
"_version": 1,
"_score": null,
"_source": {
"path": "/etc/logstash/logs/prod_user_login/user_logins_prod.csv",
"Login Date": "2/08/2017 09:11",
"type": "prod_user_login",
"@timestamp": "2017-08-01T23:11:00.000Z",
"tags": [
"_grokparsefailure",
"_geoip_lookup_failure",
"PROD_USER_LOGIN"
],
"UserID": "1857",
"User Type": "HLRA_Client",
"@version": "1",
"host": "elk-on-ubuntu",
"Role Name": "Diversa Superannuation Services"
},
"fields": {
"@timestamp": [
"2017-08-01T23:11:00.000Z"
]
},
"sort": [
1501629060000
]
}

[Pirate_Trkmn]

and it is filter which i use
filter {
if [type] == "prod_user_login" {
csv {
separator => ","
columns => ["UserID","User Type","Role Name","Login Date"]
add_tag => [ "PROD_USER_LOGIN" ]
remove_field => ["message"]
}
date {
match => ["Login Date", "dd/MM/yyyy HH:mm"]
}
}
}

[Christian_Dahlqvist]

That looks like the correct UTC timestamp corresponding to the Sydney timezone timestamp. As timestamps in Elasticsearch has to be in UTC, I do not see what the problem is.