Two timezones in JSON log

Elastic Stack Beats
1

Hi,

New doubt...
We are using Filebeat > ElasticSearch > Kibana 5.6.4 to visualize JSON log files like...

{"@timestamp":"2018-01-21T23:50:18.000Z","EMP":"ARA","INST":"PRI","TIPO":"T2","MSG":"*** FPS COV: [PCW](E 1466196 2001) -> 2018/01/21 23:50:04.310 Est:255 Cal:1 *** ","MOD":"AbdFPS","FNT":"fpsDD.c","LIN":47,"ID1":0,"ID2":0,"ID3":0,"STR1":"","STR2":"","STR3":""}

Filebeat asume that the timestamp is UTC, but in some cases is GMT+1 and in other UTC. How can I change my logs to set the timezone?

I know that it's possible to change all the filebeat timezone but... to change it per line in filebeat?
Thanks

2

Please clarify what timestamp you are talking about and where you are seeing it.

@timestamp as sent by Filebeat is always in UTC (as indicated by Z).

Kibana renders all date type fields using the browser's timezone. The @timestamp field is of date type.

If you are parsing timestamps out of log lines and the timestamp doesn't specify a timezone then you can use the add_locale to instruct Filebeat to add the machine's timezone to outgoing events. Then on the parsing machine (Logstash or Ingest Node) you can pass this timezone to the date filter so that it interprets the timestamp properly.

3

I'm sorry for not being specific :frowning:

We want to change the timezone in the JSON lines. Now is always in UTC (as indicated by Z) but...
How we can send a log line (like the line below) and set the timezone to GMT+1,? We must change the Z?

{"@timestamp":"2018-01-21T23:50:18.000Z","EMP":"ARA","INST":"PRI","TIPO":"T2","MSG":"*** FPS COV: [PCW](E 1466196 2001) -> 2018/01/21 23:50:04.310 Est:255 Cal:1 *** ","MOD":"AbdFPS","FNT":"fpsDD.c","LIN":47,"ID1":0,"ID2":0,"ID3":0,"STR1":"","STR2":"","STR3":""}

4

It is not possible to modify the format or timezone used in the @timestamp field for Beats. It's not a practice we encourage. Why do you want to change the timezone of @timestamp?

This has been discussed in the context of Logstash in another post: How to set @timestamp timezone?

You could probably use a ruby filter to mutate the timezone in Logstash. ES will always store the value as UTC internally. https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html

5

We can change the log PREVIOUSLY of being parsed in filebeat.

Out original logs are not in JSON and we've a script that converts it to JSON, is not a problem for us change all the JSON lines

My question is... when filebeat understand that @timestamp is GMT+1? What changes we must made in the JSON line to set the timezone to GMT+1 (or other)

I read this https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html#custom-date-formats
But changing is not working (adding +0100)
{"@timestamp":"2018-01-21T23:50:18.000Z+0100","EMP":"ARA","INST":"PRI","TIPO":"T2","MSG":"*** FPS COV: [PCW](E 1466196 2001) -> 2018/01/21 23:50:04.310 Est:255 Cal:1 *** ","MOD":"AbdFPS","FNT":"fpsDD.c","LIN":47,"ID1":0,"ID2":0,"ID3":0,"STR1":"","STR2":"","STR3":""}

I know, Im a newbie :slight_smile:

6

I think I'm starting to understand your issue. Could you please share your Filebeat config so I can confirm I understand your setup.

When parsing JSON with Filebeat that contains @timestamp I believe Filebeat will parse the @timestamp value as RFC3339. I don't see this in the docs, but maybe I missed it (we should add it).

So instead of 2018-01-21T23:50:18.000Z+0100, try 2018-01-21T23:50:18+01:00 in your logs.

You can test parsing here: https://play.golang.org/p/XNM6bGehtMl

1 Like
7

Here my filebeat.yml

> ###################### Filebeat Configuration Example #########################
> #=========================== Filebeat prospectors =============================
> 
> filebeat.prospectors:
> 
> - input_type: log
> 
>   # Paths that should be crawled and fetched. Glob based paths.
>   paths:
>     - C:\sac4\elastic5.6.6\logs\*json
>     #- c:\programdata\elasticsearch\logs\*
> 
>    ### JSON configuration
> 
>   json.message_key: message
>   json.keys_under_root: true
>   json.overwrit
> 
> #================================ Outputs =====================================
> #-------------------------- Elasticsearch output ------------------------------
> output.elasticsearch:
>   # Array of hosts to connect to.
>   hosts: ["localhost:9200"]
>   bulk_max_size: 8192
8

Solved,

Fot UTC @timestamp":"2018-01-30T10:00:00Z
Fot UTC+5 @timestamp":"2018-01-30T10:00:00+05:00

Thank you <3

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.