Repeat: Filebeat include is not working when logs are in both json and non-json format


(Chris Weiss) #1

This is more or less a duplicate of the problem described here: Filebeat include is not working when logs are in both json and non-json format. It was closed without resolution :frowning: .

We are trying to parse JSON content in live Jenkins build logs, we only want the JSON bits, not all the extra Jenkins log text.

prospector config:

- type: log
    paths: "/var/jenkins_home/jobs/*/jobs/*/jobs/test/builds/*/log"
    json.keys_under_root: true
    fields_under_root: true
    json.message_key: job_facts.BuildNumber
    include_lines:
      - '^{'

Test Jenkins output we're trying to parse:

Started by user Weiss, Chris
[Pipeline] node
Running on c5275fad5e1d-46ed17a9 in /var/swarm-client/workspace/Release_Engineering_Core/sandbox/test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Preparation)
[Pipeline] echo
{"job_facts.BuildNumber": "24"}
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

If we have the include_lines statement, we get no output to Elasticsearch.

If we remove it, we get both the (correctly deconstructed) JSON but also the unwanted non-JSON log entries.

Either way, the filebeat log displays "Error decoding JSON" entries for each non-JSON line in the log.

I've also tried many variations on the "include_lines" statement:

include_lines: ['^\{']
include_lines: '^{'

etc...

I should add that we're in control of the JSON structure of the content in the Jenkins logs, if the JSON is not formatted correctly for Filebeat, we can fix that.


(Harsh Bajaj) #2

Hi,

Have you tried with below syntax. Please try it with and let me know if still not working.

include_lines: ["^ERR", "^WARN"]

Regards,


(Adrian Serrano) #3

The problem here is that JSON decoding happens before filtering, so the include_lines is useless.

One way you can make it work is by adding a drop processor to get rid of the non-JSON events, like this:

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /tmp/json.log
  json.keys_under_root: true
  fields_under_root: true
  json.message_key: job_facts.BuildNumber

processors:
    - drop_event:
          when:
            regexp:
              message: ""

This will drop all events that contain a message field, that did the trick for me.


(Chris Weiss) #4

Thanks, that worked for us as well!


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.