Hello everyone...
I'm using json config on my nginx logs, that's parsed ans shipped by filebeat to elasticshearch and works very fine.
log_format json '{ "time": "$time_local", '
'"remote_ip": "$remote_addr", '
'"remote_user": "$remote_user", '
'"request": "$request", '
'"response": "$status", '
'"bytes": "$body_bytes_sent", '
'"referrer": "$http_referer", '
'"agent": "$http_user_agent" }';
but, on elasticsearch the resultant json isn't exactly I want, for example:
{
"took" : 63,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 8842259,
"max_score" : 1.0,
"hits" : [ {
"_index" : "logs",
"_type" : "log",
"_id" : "AVktD_xOzZme0fqPZkS3",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2016-12-23T19:02:14.450Z",
"beat" : {
"hostname" : "CMD-KALTURA-NGI03",
"name" : "CMD-KALTURA-NGI03",
"version" : "5.1.1"
},
"input_type" : "log",
"message" : "{ \"time\": \"23/Dec/2016:16:02:13 -0300\", \"remote_ip\": \"200.43.80.131\", \"remote_user\": \"-\", \"request\": \"GET /hls/p/116/sp/11600/serveFlavor/entryId/0_qo9vt9ie/v/2/flavorId/0_rs2urn52/name/a.mp4/seg-2-v1-a1.ts HTTP/1.1\", \"response\": \"200\", \"bytes\": \"183864\", \"referrer\": \"https://domain.com/html5/html5lib/v2.49/mwEmbedFrame.php/p/116/uiconf_id/23448615/entry_id/0_qo9vt9ie?wid=_116&iframeembed=true&playerId=kaltura_player_1482494144&entry_id=0_qo9vt9ie\", \"agent\": \"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36\" }",
"offset" : 1768277605,
"source" : "/opt/kaltura/log/json_access.log",
"type" : "log"
}
},
As you see, the entire line from nginx log, are included in one key "message" instead separated and own keys like:
"time": "23/Dec/2016:16:02:13 -0300\",
"agent":""Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
How I can get the config I want...?
I'm looking filebeat.template.json but i'm not sure if is the place I need to edit.
Regards