How to stop Filebeat from shipping incorrect JSON to Elastic?

Elastic Stack Beats
,
1

Hi,

Our applications emit logs in ECS format, a log entry per line. In Filebeat config we have a parser defined:

parsers:
  - ndjson:
      target: ''
      expand_keys: true
      overwrite_keys: true
      add_error_key: true

Then there is a processor in the ingest pipeline which converts some of the fields into lower case:

processors:
  - lowercase:
      field: 'log.level'
on_failure:
- set:
    field: error.message
    value: '{{ _ingest.on_failure_message }}'

Sometimes we have issues with free disk space which lead to incorrect log entries in the file, where one log entry is concatenated with previous incompletely written log entry, e.g.:

{"@timestamp":"2024-10-17T10:35:59.1814105+02:00","log.level":"Information","message":"Requ{"@timestamp":"2024-10-17T12:00:01.0603052+02:00","log.level":"Information","message":"Ok"}

Notice how the first message property is not written completely and then the second log entry is immediately appended to it.

When Filebeat processes such lines, it emits an error like "Error decoding JSON: invalid character '@' after object key:value pair" (error message varies a bit depending on where exactly previous log line was abrupted).

Finally, broken log lines are delivered into Elastic where error.message is set to "field [level] not present as part of path [log.level]".

We would like to just completely ignore such broken log lines. How to implement it properly? ignore_decoding_error option of ndjson parser seems to just control whether error log is emitted or not.

2

Hi @andreycha

processors:

Are executed in order, so perhaps the last processors should just be a drop_event processor based on the contents or existence of one of the two following...

add_error_key
If this setting is enabled, Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors or when a message_key is defined in the configuration but cannot be used.

Something like

  - drop_event:
      when:
        equals:
          error.type: "json"
1 Like
3

Hi @stephenb ,

Thank you, that should do the job!