Filter the content of "message" as independent fields


I'm using filebeat in my GKE cluster. All my logs messages are in json format.

My surprise is that the message field is treated as a single field, and what I would like is for it to be treated as separate fields in order to have columns with the value key.

Example (extract) of json log received:

"_score": null,
  "_source": {
    "@timestamp": "2019-09-19T15:48:21.178Z",
    "message": "{\"target\":{\"requestID\":\"42170572-bea6-45fd-aba7-6cad3f75182b\",\"resourceID\":\"xxx-clique-v1-explorer-1\",\"actionPath\":\"/api/blocks?returnTransactions=true&returnReceipts=true&height=3080060\"},\"resourceInfo\":\"plugin=xxx-clique-v1-explorer-1\",\"authentication\":{\"identity\":{\"id\":12,\"identifier\":\"\",\"maId\":\"61c577cc-77ab-4bd5-a369-e8b7bff583fa\",\"type\":\"SERVICE\",\"identityProfile\":{\"displayName\":\"xxxx TX Monitor Service Account\",\"avatarFilename\":\"\"},\"dateCreated\":1568886389000,\"dateUpdated\":1568886389000},\"project\":null,\"marpp\":{\"id\":8,\"name\":\"xxx Transactions Monitor\",\"marppId\":\"xxxx-transactions-monitor\",\"appToken\":\"\",\"organization\":\"abc-org\",\"imageURL\":\"\",\"appURL\":\"\"},\"valueTokenVersion\":1,\"iat\":1568908101,\"exp\":1568909001},\"topic\":\"plugin-proxy.request-arrival\"}",
    "log": {
      "offset": 8764173,
      "file": {
        "path": "/var/log/containers/resource-proxy-deployment-58886c655d-g8w4p_default_resource-proxy-6817ab68aaaa537ef0c25a65dbd747c7e8c9d7b9cbbb802f011dccc5751ed692.log"
    "input": {
      "type": "container"

I would like filter by field "topic" that it is in message field.

That's its possible? I have to activate something in filebeat configuration?

Thank you so much


  • decode_json_fields:
    fields: ["message"]
    target: ""
    overwrite_keys: true

Reference : Condition with decode_json_fields processor

Thank you very much, works pretty well!

You may also want to convert fields to proper data types before sending to ES (you can check on ES side / Kibana the data type of fields). Ex : dateCreated can be converted to number and you can do range queries on that field.

Cheers :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.