Beats not parsing `message` correctly with filebeat protocol

Elastic Stack Beats
1

Hi, I've ran into an issue with filebeat parsing a json message field when using the output.logstash.

filebeat version: 8.15.2
logstash version: 8.13.0

The thing is, when I use:

processors:
  - decode_json_fields:
      fields: ["message"]

on a valid json log, the input in Logstash looks like this:
"{test=field}"
I would expect a valid json object:
{"test": "field"}

2 interesting points:

  1. when you set the output of the filebeat to a local file instead of logstash, it outputs a valid JSON
  2. when you use any other field than message, it also produces a valid JSON

I need to do the decode on filebeat side. Because in production, I am doing some other processing on the fields.
The current workaround is to use different field as target, e.g. message-json, but it is quite clumsy and it needs additional adjustments on the logstash side, which I want to avoid.

I have prepared repository with minimal setup here, you can test it there.

Please advice if & how can this be tackled.

Thank you.

2

Hello, and welcome.

Please share some example of this, it is not clear what is the issue here.

3

input file (log.log) for logstream:

{"test": "field"}

filebeat.yml config:

output.logstash:
  hosts: ${LOGSTREAM_HOSTS}

filebeat.inputs:
  - type: log
    paths:
      - /var/log/log.log
    tags: ["test-tag"]
processors:
  - decode_json_fields:
      fields: ["message"]

logstash.conf:

input {
  beats {
    port => 443
    ssl_enabled => false
  }
}

output {
    stdout {
      codec => rubydebug {
        metadata => true
      }
    }
}

expected output on logstash:

"message" => {
   "test" => "field"
}

actual output on logstash:

"message" => "{test=field}"

check repo to reproduce

4

Can you please share the output you are getting? Copy the output in th screen or take a screenshot.

Unfortunately I cannot replicate it at the moment.

Also, you are using the rubydebug codec, so it will update the logs like this:

{
       "message" => {
        "test" => "field"
    },
    "@timestamp" => 2024-10-09T13:47:28.325083866Z,
      "@version" => "1"
}

If you want it in a json format you need to use the json codec, which will output something like this:

{"message":{"teste":"field"},"@timestamp":"2024-10-09T13:48:22.032225940Z","@version":"1"}
5

after changing the codec to json, this is the output:

{"host":{"name":"c95347091bc4"},"@version":"1","@timestamp":"2024-10-09T14:42:16.596Z","event":{"original":"{test=field}"},"message":"{test=field}","log":{"offset":0,"file":{"path":"/var/log/log.log"}},"agent":{"id":"a0c54e93-72bd-478c-b39c-a8016e767ba2","type":"filebeat","version":"8.15.2","name":"c95347091bc4","ephemeral_id":"50f25619-11b7-4431-918c-30c9870ad3bf"},"ecs":{"version":"8.0.0"},"tags":["test-tag","beats_input_codec_plain_applied"],"input":{"type":"log"}}

aren't you able to replicate it via the repo I provided? (check the readme there)

6

I didn't try, unfortunately I do not have time for it at the moment.

But the log you share has this information:

"event":{"original":"{test=field}"},"message":"{test=field}"

This means that your original message is {test=field}, this is not a valid json.

I'm not sure what you are trying to achieve here, if you use the decode_json_fields filebeat will parse your json message, but if your message looks like this one, then it will not work because it is not a valid json.

Please share the content of the file: /var/log/log.log

I do not see any issue in Filebeat nor Logstash, they are doing what they are configured to do, but your original message is not a valid json.

7

That's why I believe there is an issue, the original log.log file looks like this:

{"test":"field"}
8

Hello again, did you have any more time to look into this? :pray:

I've been debugging the beats go code and it is seems to be sending the body correctly

"message": {"test":"field"}

This would indicate to me, that logstash is treating the message field in a special way and somehow forcing the value to be:

"{test=field}"

Is it possible? If I send the data in any other field than message, it seems to be working as expected.

I am probably not able to dig into logstash deeper, since it's in Java and I would need too much time to set the tools up.