Autodiscover and parsing json in a multiline log message

Elastic Stack Beats
1

I configured Filebeat autodiscovery in order to monitor Elasticsearch logs (both are running as Docker containers). The objective is to monitor any errors generated by Elasticsearch. The logs generated by Elasticsearch are in JSON format.

Using the following configuration, I can see Elasticsearch logs through Kibana; however, the JSON message was not parsed, and Filebeat adds the field 'error.message' which contains the same message and the text 'Provided Grok expressions do not match field value:' at the beginning.
Here is the configuration of Filebeat:

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      json.message_key: log
      templates:
        - condition:
            equals:
              docker.container.labels.filebeat_enable: "true"
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"

Elasticsearch configuration in docker-compose file:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.4.2
    ports:
      - "${ELASTICSEARCH_HOST_HTTP_PORT}:9200"
      - "${ELASTICSEARCH_HOST_TCP_PORT}:9300"
    volumes:
     - /etc/localtime:/etc/localtime:ro
     - ./elasticsearch:/usr/share/elasticsearch/data
    environment:
      - ES_JAVA_OPTS:"-Xmx256m -Xms256m"
      - discovery.type=single-node
    labels:
      - "co.elastic.logs/multiline.pattern='^\\{'"
      - "co.elastic.logs/multiline.negate=true"
      - "co.elastic.logs/multiline.match=after"
      - "co.elastic.logs/module=elasticsearch"
      - "co.elastic.logs/json.keys_under_root=true"
      - "co.elastic.logs/json.add_error_key=true"
      - "co.elastic.logs/json.message_key=log"
    networks:
      iot-backend:
        ipv4_address: 11.0.0.13

In Kibana, the content of the field message is ( i did not provide the full message becuase it is too long) :

{"type": "server", "timestamp": "2019-11-19T09:33:10,397Z", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "docker-cluster", "node.name": "345b8b42cb9a", "message": "[.kibana][0],.........`

The content of the field error.message:
Provided Grok expressions do not match field value: [{\"type\": \"server\", \"timestamp\": \"2019-11-19T09:33:10,397Z\", \"level\": \"DEBUG\",

What is the wrong in my configuration, and how can I get the message parsed, my objective is to extract the field "level" so I can search for the errors easily in the future.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.