Repeat: Filebeat include is not working when logs are in both json and non-json format

Elastic Stack Beats
1

This is more or less a duplicate of the problem described here: Filebeat include is not working when logs are in both json and non-json format. It was closed without resolution :frowning: .

We are trying to parse JSON content in live Jenkins build logs, we only want the JSON bits, not all the extra Jenkins log text.

prospector config:

- type: log
    paths: "/var/jenkins_home/jobs/*/jobs/*/jobs/test/builds/*/log"
    json.keys_under_root: true
    fields_under_root: true
    json.message_key: job_facts.BuildNumber
    include_lines:
      - '^{'

Test Jenkins output we're trying to parse:

Started by user Weiss, Chris
[Pipeline] node
Running on c5275fad5e1d-46ed17a9 in /var/swarm-client/workspace/Release_Engineering_Core/sandbox/test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Preparation)
[Pipeline] echo
{"job_facts.BuildNumber": "24"}
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

If we have the include_lines statement, we get no output to Elasticsearch.

If we remove it, we get both the (correctly deconstructed) JSON but also the unwanted non-JSON log entries.

Either way, the filebeat log displays "Error decoding JSON" entries for each non-JSON line in the log.

I've also tried many variations on the "include_lines" statement:

include_lines: ['^\{']
include_lines: '^{'

etc...

I should add that we're in control of the JSON structure of the content in the Jenkins logs, if the JSON is not formatted correctly for Filebeat, we can fix that.

2

Hi,

Have you tried with below syntax. Please try it with and let me know if still not working.

include_lines: ["^ERR", "^WARN"]

Regards,

3

The problem here is that JSON decoding happens before filtering, so the include_lines is useless.

One way you can make it work is by adding a drop processor to get rid of the non-JSON events, like this:

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /tmp/json.log
  json.keys_under_root: true
  fields_under_root: true
  json.message_key: job_facts.BuildNumber

processors:
    - drop_event:
          when:
            regexp:
              message: ""

This will drop all events that contain a message field, that did the trick for me.

4

Thanks, that worked for us as well!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.