Unexpected format of ES document generated by FileBeats

[Pablo_Albertengo]

Hi guys!

I'm starting to test with FileBeats and I can't generate documents in ElasticSearch that suit what I want.

This is my scenario.
I want to send some audit information to ElasticSearch from my Java application.
To do this, I am using the encoder net.logstash.logback.encoder.LogstashEncoder to format the messages to json.

Additionally, I modify the default FileBeats template (fields.yml) for one that has the fields of my interest.

I understand that the template was successfully uploaded to ES

The problem is that instead of generating the document as I expect:

{
  "@timestamp": "2020-09-24T09: 16: 50.957-03: 00",
  "messageType": "rq",
  "operation": "search",
  "operationDt": "2020-09-24 09:16:50",
  "rawMessage": "xmlxml",
  "supplierId": "BAP"
}

FileBeats generates the document like this:

{
  "_index": "filebeat-7.6.2-2020.09.24-000001",
  "_type": "_doc",
  "_id": "OrRKwHQBxk1i8MjjUNrR",
  "_Version 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2020-09-24T13: 25: 27.801Z",
    "host": {
      "containerized": false,
      "name": "nemo-development-1",
      "hostname": "nemo-development-1",
      "architecture": "x86_64",
      "os": {
        "kernel": "4.18.0-25-generic",
        "codename": "cosmic",
        "platform": "ubuntu",
        "version": "18.10 (Cosmic Cuttlefish)",
        "family": "debian",
        "name": "Ubuntu"
      },
      "id": "4590f65319a441daa3312002a2028743"
    },
    "agent": {
      "type": "filebeat",
      "ephemeral_id": "303ee170-25ac-4008-87bb-3e8675d61232",
      "hostname": "nemo-development-1",
      "id": "43e47003-bb81-4651-81d6-2b8cb6c1dce6",
      "version": "7.6.2"
    },
    "container": {
      "id": "2020-09"
    },
    "log": {
      "file": {
        "path": "/home/pricesurferlogs/DEV/suppliers/2020-09/24/files/xmlfiles.log"
      },
      "offset": 0
    },
    "message": "{\" @ timestamp \ ": \" 2020-09-24T09: 16: 50.957-03: 00 \ ", \" @ version \ ": \" 1 \ ", \" logger_name \ ": \ "net.nemogroup.audit.logging.AuditLogger \", \ "thread_name \": \ "main \", \ "level \": \ "TRACE \", \ "level_value \": 5000, \ "LOGS_SUPPLIER \ ": \" / home / pricesurferlogs / DEV / supplier \ ", \" LOGS_HOME \ ": \" / home / pricesurferlogs / DEV \ ", \" messageType \ ": \" rq \ ", \" operation \ ": \ "search \", \ "operationDt \": \ "2020-09-24 09:16:50 \", \ "rawMessage \": \ "xmlxml \", \ "supplierId \": \ "BAP \" } ",
    "input": {
      "type": "log"
    },
    "ecs": {
      "version": "1.4.0"
    }
  }
}

That is, it puts the entire line in the message field.

Do you have any idea what I may be doing wrong?
Thanks in advance,
Pablo

[Pablo_Albertengo]

I have already solved the problem.
Basically I was missing some configuration in the filebeat.yml file related to decode the log in json format.

The added configuration lines are:

json.keys_under_root: true
json.message_key: log

processors:
- decode_json_fields:
    fields: ["log"]
    target: ""
    overwrite_keys: true