Nginx json logs + filebeat + elasticsearch but unwanted key: value order

Elastic Stack Beats
1

Hello everyone...

I'm using json config on my nginx logs, that's parsed ans shipped by filebeat to elasticshearch and works very fine.

        log_format json '{ "time": "$time_local", '
                       '"remote_ip": "$remote_addr", '
                       '"remote_user": "$remote_user", '
                       '"request": "$request", '
                       '"response": "$status", '
                       '"bytes": "$body_bytes_sent", '
                       '"referrer": "$http_referer", '
                       '"agent": "$http_user_agent" }';

but, on elasticsearch the resultant json isn't exactly I want, for example:

 {
"took" : 63,
"timed_out" : false,
"_shards" : {
  "total" : 5,
  "successful" : 5,
 "failed" : 0
},
"hits" : {
"total" : 8842259,
"max_score" : 1.0,
"hits" : [ {
  "_index" : "logs",
  "_type" : "log",
  "_id" : "AVktD_xOzZme0fqPZkS3",
  "_score" : 1.0,
  "_source" : {
    "@timestamp" : "2016-12-23T19:02:14.450Z",
    "beat" : {
      "hostname" : "CMD-KALTURA-NGI03",
      "name" : "CMD-KALTURA-NGI03",
      "version" : "5.1.1"
    },
    "input_type" : "log",
    "message" : "{ \"time\": \"23/Dec/2016:16:02:13 -0300\", \"remote_ip\": \"200.43.80.131\", \"remote_user\": \"-\", \"request\": \"GET /hls/p/116/sp/11600/serveFlavor/entryId/0_qo9vt9ie/v/2/flavorId/0_rs2urn52/name/a.mp4/seg-2-v1-a1.ts HTTP/1.1\", \"response\": \"200\", \"bytes\": \"183864\", \"referrer\": \"https://domain.com/html5/html5lib/v2.49/mwEmbedFrame.php/p/116/uiconf_id/23448615/entry_id/0_qo9vt9ie?wid=_116&iframeembed=true&playerId=kaltura_player_1482494144&entry_id=0_qo9vt9ie\", \"agent\": \"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36\" }",
    "offset" : 1768277605,
    "source" : "/opt/kaltura/log/json_access.log",
    "type" : "log"
  }
},

As you see, the entire line from nginx log, are included in one key "message" instead separated and own keys like:

"time": "23/Dec/2016:16:02:13 -0300\",
"agent":""Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"

How I can get the config I want...?

I'm looking filebeat.template.json but i'm not sure if is the place I need to edit.

Regards

2

You need to configure Filebeat to decode the JSON payloads: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#config-json

3

Well...

Finally I get this:

  "_index" : "logs",
  "_type" : "log",
  "_id" : "AVlqg8Cy00hSCEIh5Vig",
  "_score" : 1.0,
  "_source" : {
    "@timestamp" : "2017-01-04T17:25:31.296Z",
    "beat" : {
      "hostname" : "CMD-KALTURA-NGI03",
      "name" : "CMD-KALTURA-NGI03",
      "version" : "5.1.1"
    },
    "input_type" : "log",
    "json" : {
      "agent" : "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36",
      "bytes" : "334823",
      "referrer" : "http://www.clarin.com/sociedad/ven-espacio-incendios-pampa-rio-negro_0_SyFgZ2FHg.html",
      "remote_ip" : "201.216.242.69",
      "request" : "GET /hds/p/102/sp/10200/serveFlavor/entryId/0_lc72fcd5/v/2/flavorId/0_,cl6h1161,f92y1jm1,35tgklp2,x05gtxp0,/forceproxy/true/name/a.mp4.urlset/frag-f1-v1-a1-Seg1-Frag39 HTTP/1.1",
      "response" : "200",
      "time" : "04/Jan/2017:14:25:30 -0300"
    },
    "offset" : 73804472,
    "source" : "/opt/kaltura/log/json_access.log",
    "type" : "log"
  }

That's more usefull for me, and the only thing I need to do was this:

### JSON configuration
#json.message_key: log
 
# By default, the decoded JSON is placed under a "json" key in the output document.
# If you enable this setting, the keys are copied top level in the output document.
json.keys_under_root: false
 
# If keys_under_root and this setting are enabled, then the values from the decoded
# JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.)
# in case of conflicts.
json.overwrite_keys: false

# If this setting is enabled, Filebeat adds a "json_error" key in case of JSON
# unmarshaling errors or when a text key is defined in the configuration but cannot
# be used.
json.add_error_key: false
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.