Filter the content of "message" as independent fields

[David_Oceans]

Hi!

I'm using filebeat in my GKE cluster. All my logs messages are in json format.

My surprise is that the message field is treated as a single field, and what I would like is for it to be treated as separate fields in order to have columns with the value key.

Example (extract) of json log received:

"_score": null,
  "_source": {
    "@timestamp": "2019-09-19T15:48:21.178Z",
    "message": "{\"target\":{\"requestID\":\"42170572-bea6-45fd-aba7-6cad3f75182b\",\"resourceID\":\"xxx-clique-v1-explorer-1\",\"actionPath\":\"/api/blocks?returnTransactions=true&returnReceipts=true&height=3080060\"},\"resourceInfo\":\"plugin=xxx-clique-v1-explorer-1\",\"authentication\":{\"identity\":{\"id\":12,\"identifier\":\"xxx-transactions-monitor-service-account@xxx-transactions-monitor-project.mserviceaccount.com\",\"maId\":\"61c577cc-77ab-4bd5-a369-e8b7bff583fa\",\"type\":\"SERVICE\",\"identityProfile\":{\"displayName\":\"xxxx TX Monitor Service Account\",\"avatarFilename\":\"\"},\"dateCreated\":1568886389000,\"dateUpdated\":1568886389000},\"project\":null,\"marpp\":{\"id\":8,\"name\":\"xxx Transactions Monitor\",\"marppId\":\"xxxx-transactions-monitor\",\"appToken\":\"\",\"organization\":\"abc-org\",\"imageURL\":\"https://storage.googleapis.com/xxx-img/image_marpp_8.png\",\"appURL\":\"https://txmonitor.xxxx.abc.com\"},\"valueTokenVersion\":1,\"iat\":1568908101,\"exp\":1568909001},\"topic\":\"plugin-proxy.request-arrival\"}",
    "log": {
      "offset": 8764173,
      "file": {
        "path": "/var/log/containers/resource-proxy-deployment-58886c655d-g8w4p_default_resource-proxy-6817ab68aaaa537ef0c25a65dbd747c7e8c9d7b9cbbb802f011dccc5751ed692.log"
      }
    },
    "input": {
      "type": "container"
    },

I would like filter by field "topic" that it is in message field.

That's its possible? I have to activate something in filebeat configuration?

Thank you so much

[kumarabhi (Abhishek)]

processors:

• decode_json_fields:
  fields: ["message"]
  target: ""
  overwrite_keys: true

Reference : Condition with decode_json_fields processor

[David_Oceans]

Thank you very much, works pretty well!

[kumarabhi (Abhishek)]

You may also want to convert fields to proper data types before sending to ES (you can check on ES side / Kibana the data type of fields). Ex : dateCreated can be converted to number and you can do range queries on that field.
https://www.elastic.co/guide/en/beats/filebeat/master/convert.html

Cheers