Structured logging with Filebeat

Elastic Observability Logs
1

Signed up for the elastic trial and quickly got the Filebeat up and running and getting docker statistics to Elasticsearch.

I'm really stuck in trying to add any kind of structured logging to kibana.
I've tried to add a ingest node pipeline and use a Key-Value pair processor to get some key values out of my logs but nothing has worked.

All i'm trying to do is take a log message like
"message": "2021/04/06 14:28:00.055|INFO|Process-Control: SUCCESS: processr is configured to not run, User=joe Fleet_ID=19 Fleet_Name=\"New Trucks\"

and pull out User, Fleet_ID, Fleet_Name so that I can see them in the "Available Fields" in Kibana

Any help would be appreciated

2

What you need is a way to use GROK to break the message apart into individual fields. Since you're using Filebeat I would recommend using the GROK feature on the ingest node. You can read more about using Filebeat with Ingest Node here:

and you can read about using GROK with the Ingest Node here:

3

We have 5 different docker images doing their own thing.

This post helped me get a little closer but I was hoping to find a more generic solution to parse key value pairs.

If we switched to json logging, would that be easier to parse?

4

If we switched to json logging, would that be easier to parse?

Absolutely, then you're not having to mess with GROK to break the fields apart. Here is a blog post on Structure Logging that explains how to ingest the JSON.

6

I've done some testing and I'm able to parse the json input.
For the containers that are not going to be logging json I'd like to use multi line pattern as shown below.
Can I use multiline options for some containers and the json.message_key for others.
Right now those two sections do not play nice with each other. I either get json parsing or I get nice multi line.

Here's the top section of my filebeat.yml

filebeat.inputs:
- type: container
  # Change to true to enable this input configuration.
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*.log

#THIS will break multiline
#  json.messge_key: log
#  json.keys_under_root: true
#  json.add_error_key: true

#- type: log
  multiline.type: pattern
  multiline.pattern: '^[0-9]{4}\/[0-9]{2}\/[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.{4}'
  multiline.negate: true
  multiline.match: after
7

I think you need to create 2 input definitions; one for the JSON and one for the multiline logs. I'm going to ping someone from the Filebeat team to chime in.

8

If changing the logging configuration in your application is feasible, I'd suggest to have a look at ECS logging. Some of the loggers (such as log4j2) allow for structured logging. The logs would then be formatted to JSON which makes parsing much easier.

9

So the question evolved to "can I do both json and multiline for one filebeat configuration."

After much testing, the answer looks to be yes, with "filebeat.autodiscover" .
I haven't figured out how to do multiple docker images under 1 templates sections so for now it looks like I'll have repeating code sections if the multiline or json configuration is the same.

This worked for me, app1 got the multiline, app3 did json parsing

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: app1
#doesn't work, can't add more containers like this
#             docker.container.image: app2
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              multiline.type: pattern
              multiline.pattern: '^[0-9]{4}\/[0-9]{2}\/[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.{4}'
              multiline.negate: true
              multiline.match: after

    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: app3
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              json.messge_key: log
              json.keys_under_root: true
              json.add_error_key: true

With 5 containers looks like i'll have to have 5 templates sections unless someone knows of a workaround.

10

I guess it's close to my question "Ingest mixed container logs with text and JSON":

Can we easily combine text and structured JSON logging in the same container standard output? With JSON being parsed and non-parse-able data just saved in a text field? With a single configuration? (I got 50+ containers :slightly_smiling_face:)

Maybe @felixbarny can help?

11

I've added an answer in your thread: Ingest mixed container logs with text and JSON [filebeat][docker] - #2 by felixbarny

12

Here's how you can add multiple conditions to the same template: Multiple conditions with autodiscover & docker containers - #2 by steffens

13

Another option is to add a label to your docker containers that should get the same logging config and match on docker.container.labels instead of matching on the image.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.