Filebeat with JSON logs


(Douglas Ribas) #1

Hi people.
I have this situation, all import dates stay in message space
I have logs in format JSON line by line
{'date':'2016-08-22','time':'04:42:59', 'user':'joana','busca':'uol.com','servidor':'Node-03'} {'date':'2016-08-22','time':'04:42:59', 'user':'rafisa','busca':'aol.com.br','servidor':'Node-02'} {'date':'2016-08-22','time':'04:42:59', 'user':'roberto','busca':'palmeiras.com.br','servidor':'Node-01'} {'date':'2016-08-22','time':'04:42:59', 'user':'joana','busca':'google.com','servidor':'Node-03'}
And i use kibana to see all date from elasticsearch and my result is:

@timestamp:August 22nd 2016, 04:43:50.984, 22nd/08beat.hostname:douglas-Inspiron-5548beat.name:douglas-Inspiron-5548count:1fields: - input_type:logmessage:{'date':'2016-08-22','time':'04:42:58', 'user':'wagner','busca':'angelica.globo.com','servidor':'Node-01'}offset:22,339,929source:/opt/logs/cadvlanview/vips.logtype:object_id:AVaxNQESVeR5vHPNwmXN_type:object_index:cadvlanview-2016.08.22_score:

my elastic maping is:

{ "cadvlanview-2016.08.22": { "mappings": { "object": { "properties": { "@timestamp": { "type": "date", "format": "strict_date_optional_time||epoch_millis" }, "beat": { "properties": { "hostname": { "type": "string" }, "name": { "type": "string" } } }, "count": { "type": "long" }, "input_type": { "type": "string" }, "message": { "type": "string" }, "offset": { "type": "long" }, "source": { "type": "string" }, "type": { "type": "string" } } } } } }

how i do mapping menssage text to json format?


(Magnus B├Ąck) #2

Filebeat 5 supports parsing JSON logs. Otherwise you need to pipe things through Logstash.


(system) #3

This topic was automatically closed after 21 days. New replies are no longer allowed.