# Structured logging with Filebeat **URL:** \ **Category:** Logs\ **Created:** [April 6, 2021, 9:23pm UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383 "2021-04-06T21:23:40Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![vharabor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vharabor/32/86659_2.png) [@vharabor](https://discuss.elastic.co/u/vharabor)\ **Post date:** [April 6, 2021, 9:23pm UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/1 "2021-04-06T21:23:40Z")
Signed up for the elastic trial and quickly got the Filebeat up and running and getting docker statistics to Elasticsearch. > **[Filebeat quick start: installation and configuration | Filebeat Reference...](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html)** I'm really stuck in trying to add any kind of structured logging to kibana. I've tried to add a ingest node pipeline and use a Key-Value pair processor to get some key values out of my logs but nothing has worked. All i'm trying to do is take a log message like `"message": "2021/04/06 14:28:00.055|INFO|Process-Control: SUCCESS: processr is configured to not run, User=joe Fleet_ID=19 Fleet_Name=\"New Trucks\"` and pull out User, Fleet\_ID, Fleet\_Name so that I can see them in the "Available Fields" in Kibana Any help would be appreciated ---
**Author:** ![simianhacker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simianhacker/32/3383_2.png) [@simianhacker](https://discuss.elastic.co/u/simianhacker)\ **Post date:** [April 6, 2021, 10:47pm UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/2 "2021-04-06T22:47:56Z")
What you need is a way to use GROK to break the message apart into individual fields. Since you're using Filebeat I would recommend using the GROK feature on the ingest node. You can read more about using Filebeat with Ingest Node here: > **[Parse data by using ingest node | Filebeat Reference \[7.12\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ingest-node.html)** and you can read about using GROK with the Ingest Node here: > **[Grok processor | Elasticsearch Guide \[master\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html)** ---
**Author:** ![vharabor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vharabor/32/86659_2.png) [@vharabor](https://discuss.elastic.co/u/vharabor)\ **Post date:** [April 7, 2021, 4:33pm UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/3 "2021-04-07T16:33:41Z")
We have 5 different docker images doing their own thing. This post helped me get a little closer but I was hoping to find a more generic solution to parse key value pairs. > [@Using Grok with KV filter](https://discuss.elastic.co/t/using-grok-with-kv-filter/170136): > > Hi, I am pretty new to ELK stack. Currently I am trying to parse my application log using grok pattern. But since my logs are not structured i may have to write too many grok conditions, which will not scale well. Sample log: 2019-02-25 10:22:27,832 LL="INFO" field1="field value" field2="field2val" MTHD="POST" O="ipadd" PAYLOAD="{"errorResponse":{"status":403,"message":"some message.","reason":"some reason"}}" using KV filter I cant parse the above log, as the payload is not parsable using KV… If we switched to json logging, would that be easier to parse? ---
**Author:** ![simianhacker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simianhacker/32/3383_2.png) [@simianhacker](https://discuss.elastic.co/u/simianhacker)\ **Post date:** [April 7, 2021, 5:32pm UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/4 "2021-04-07T17:32:28Z")
> If we switched to json logging, would that be easier to parse? Absolutely, then you're not having to mess with GROK to break the fields apart. Here is a blog post on Structure Logging that explains how to ingest the JSON. > **[Structured logging with Filebeat](https://www.elastic.co/blog/structured-logging-filebeat)** ---
**Author:** ![vharabor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vharabor/32/86659_2.png) [@vharabor](https://discuss.elastic.co/u/vharabor)\ **Post date:** [April 8, 2021, 3:44pm UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/6 "2021-04-08T15:44:46Z")
I've done some testing and I'm able to parse the json input. For the containers that are not going to be logging json I'd like to use multi line pattern as shown below. Can I use multiline options for some containers and the json.message\_key for others. Right now those two sections do not play nice with each other. I either get json parsing or I get nice multi line. Here's the top section of my filebeat.yml ``` filebeat.inputs: - type: container # Change to true to enable this input configuration. enabled: true paths: - /var/lib/docker/containers/*/*.log #THIS will break multiline # json.messge_key: log # json.keys_under_root: true # json.add_error_key: true #- type: log multiline.type: pattern multiline.pattern: '^[0-9]{4}\/[0-9]{2}\/[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.{4}' multiline.negate: true multiline.match: after ``` ---
**Author:** ![simianhacker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simianhacker/32/3383_2.png) [@simianhacker](https://discuss.elastic.co/u/simianhacker)\ **Post date:** [April 8, 2021, 5:36pm UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/7 "2021-04-08T17:36:13Z")
I think you need to create 2 input definitions; one for the JSON and one for the multiline logs. I'm going to ping someone from the Filebeat team to chime in. ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [April 12, 2021, 1:45pm UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/8 "2021-04-12T13:45:52Z")
If changing the logging configuration in your application is feasible, I'd suggest to have a look at [ECS logging](https://www.elastic.co/guide/en/ecs-logging/overview/master/intro.html). Some of the loggers (such as log4j2) allow for structured logging. The logs would then be formatted to JSON which makes parsing much easier. ---
**Author:** ![vharabor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vharabor/32/86659_2.png) [@vharabor](https://discuss.elastic.co/u/vharabor)\ **Post date:** [April 12, 2021, 10:42pm UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/9 "2021-04-12T22:42:02Z")
So the question evolved to "can I do both json and multiline for one filebeat configuration." After much testing, the answer looks to be yes, with "filebeat.autodiscover" . I haven't figured out how to do multiple docker images under 1 templates sections so for now it looks like I'll have repeating code sections if the multiline or json configuration is the same. This worked for me, app1 got the multiline, app3 did json parsing ``` filebeat.autodiscover: providers: - type: docker templates: - condition: contains: docker.container.image: app1 #doesn't work, can't add more containers like this # docker.container.image: app2 config: - type: container paths: - /var/lib/docker/containers/${data.docker.container.id}/*.log multiline.type: pattern multiline.pattern: '^[0-9]{4}\/[0-9]{2}\/[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.{4}' multiline.negate: true multiline.match: after - type: docker templates: - condition: contains: docker.container.image: app3 config: - type: container paths: - /var/lib/docker/containers/${data.docker.container.id}/*.log json.messge_key: log json.keys_under_root: true json.add_error_key: true ``` With 5 containers looks like i'll have to have 5 templates sections unless someone knows of a workaround. ---
**Author:** ![bluepuma77](https://avatars.discourse-cdn.com/v4/letter/b/a9adbd/32.png) [@bluepuma77](https://discuss.elastic.co/u/bluepuma77)\ **Post date:** [April 28, 2021, 8:30am UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/10 "2021-04-28T08:30:59Z")
I guess it's close to my question ["Ingest mixed container logs with text and JSON"](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-filebeat-docker/271139): Can we **easily** combine text and structured JSON logging in the same container standard output? With JSON being parsed and non-parse-able data just saved in a text field? With a single configuration? (I got 50+ containers 🙂) Maybe @felixbarny can help? ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [April 28, 2021, 9:36am UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/11 "2021-04-28T09:36:41Z")
I've added an answer in your thread: [Ingest mixed container logs with text and JSON [filebeat][docker] - #2 by felixbarny](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-filebeat-docker/271139/2) ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [April 29, 2021, 6:30am UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/12 "2021-04-29T06:30:41Z")
Here's how you can add multiple conditions to the same template: [Multiple conditions with autodiscover & docker containers - #2 by steffens](https://discuss.elastic.co/t/multiple-conditions-with-autodiscover-docker-containers/153634/2) ---
**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\ **Post date:** [April 29, 2021, 6:42am UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/13 "2021-04-29T06:42:59Z")
Another option is to add a label to your docker containers that should get the same logging config and match on [`docker.container.labels`](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html#_docker_2) instead of matching on the image. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 27, 2021, 6:43am UTC](https://discuss.elastic.co/t/structured-logging-with-filebeat/269383/14 "2021-05-27T06:43:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.