# Unexpected format of ES document generated by FileBeats

**URL:** <https://discuss.elastic.co/t/unexpected-format-of-es-document-generated-by-filebeats/249834>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 24, 2020, 2:35pm UTC](https://discuss.elastic.co/t/unexpected-format-of-es-document-generated-by-filebeats/249834 "2020-09-24T14:35:48Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pablo\_Albertengo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pablo_albertengo/32/47685_2.png) [@Pablo\_Albertengo](https://discuss.elastic.co/u/Pablo_Albertengo)\
**Post date:** [September 24, 2020, 2:35pm UTC](https://discuss.elastic.co/t/unexpected-format-of-es-document-generated-by-filebeats/249834/1 "2020-09-24T14:35:48Z")

</div>

Hi guys!

I'm starting to test with FileBeats and I can't generate documents in ElasticSearch that suit what I want.

This is my scenario.  
I want to send some audit information to ElasticSearch from my Java application.  
To do this, I am using the encoder net.logstash.logback.encoder.LogstashEncoder to format the messages to json.

Additionally, I modify the default FileBeats template (fields.yml) for one that has the fields of my interest.

 ![CustomFields](https://us1.discourse-cdn.com/elastic/original/3X/5/7/57967f4eceefe620fb0215d55750152a758896eb.png)

I understand that the template was successfully uploaded to ES

 ![TemplateKibana](https://us1.discourse-cdn.com/elastic/original/3X/2/5/2512c1f067a9848cf0b674726be0dfae0f67b56d.png)

The problem is that instead of generating the document as I expect:

```auto
{
  "@timestamp": "2020-09-24T09: 16: 50.957-03: 00",
  "messageType": "rq",
  "operation": "search",
  "operationDt": "2020-09-24 09:16:50",
  "rawMessage": "xmlxml",
  "supplierId": "BAP"
}

```

FileBeats generates the document like this:

```auto
{
  "_index": "filebeat-7.6.2-2020.09.24-000001",
  "_type": "_doc",
  "_id": "OrRKwHQBxk1i8MjjUNrR",
  "_Version 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2020-09-24T13: 25: 27.801Z",
    "host": {
      "containerized": false,
      "name": "nemo-development-1",
      "hostname": "nemo-development-1",
      "architecture": "x86_64",
      "os": {
        "kernel": "4.18.0-25-generic",
        "codename": "cosmic",
        "platform": "ubuntu",
        "version": "18.10 (Cosmic Cuttlefish)",
        "family": "debian",
        "name": "Ubuntu"
      },
      "id": "4590f65319a441daa3312002a2028743"
    },
    "agent": {
      "type": "filebeat",
      "ephemeral_id": "303ee170-25ac-4008-87bb-3e8675d61232",
      "hostname": "nemo-development-1",
      "id": "43e47003-bb81-4651-81d6-2b8cb6c1dce6",
      "version": "7.6.2"
    },
    "container": {
      "id": "2020-09"
    },
    "log": {
      "file": {
        "path": "/home/pricesurferlogs/DEV/suppliers/2020-09/24/files/xmlfiles.log"
      },
      "offset": 0
    },
    "message": "{\" @ timestamp \ ": \" 2020-09-24T09: 16: 50.957-03: 00 \ ", \" @ version \ ": \" 1 \ ", \" logger_name \ ": \ "net.nemogroup.audit.logging.AuditLogger \", \ "thread_name \": \ "main \", \ "level \": \ "TRACE \", \ "level_value \": 5000, \ "LOGS_SUPPLIER \ ": \" / home / pricesurferlogs / DEV / supplier \ ", \" LOGS_HOME \ ": \" / home / pricesurferlogs / DEV \ ", \" messageType \ ": \" rq \ ", \" operation \ ": \ "search \", \ "operationDt \": \ "2020-09-24 09:16:50 \", \ "rawMessage \": \ "xmlxml \", \ "supplierId \": \ "BAP \" } ",
    "input": {
      "type": "log"
    },
    "ecs": {
      "version": "1.4.0"
    }
  }
}

```

That is, it puts the entire line in the message field.

Do you have any idea what I may be doing wrong?  
Thanks in advance,  
Pablo

---

<div class="post-metadata">

**Author:** ![Pablo\_Albertengo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pablo_albertengo/32/47685_2.png) [@Pablo\_Albertengo](https://discuss.elastic.co/u/Pablo_Albertengo)\
**Post date:** [September 25, 2020, 12:36pm UTC](https://discuss.elastic.co/t/unexpected-format-of-es-document-generated-by-filebeats/249834/2 "2020-09-25T12:36:36Z")

</div>

I have already solved the problem.  
Basically I was missing some configuration in the filebeat.yml file related to decode the log in json format.

The added configuration lines are:

```auto
json.keys_under_root: true
json.message_key: log

processors:
- decode_json_fields:
    fields: ["log"]
    target: ""
    overwrite_keys: true

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 23, 2020, 2:36pm UTC](https://discuss.elastic.co/t/unexpected-format-of-es-document-generated-by-filebeats/249834/3 "2020-10-23T14:36:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.