MongoDB logs no more displayed in Kibana Filebeat MongoDB dashboard after 7.10 migration

Elastic Stack Kibana
#1

Dear experts,
after migration from 6.5.4 to 7.10.2, I cannot display anymore my logs collected by the default Filebeat MongoDB module in Kibana Dashboard Overview [Filebeat MongoDB].
I notice some updates in the provided pipeline filebeat-x.x.x-mongodb-log-pipeline between 6.5 and 7.10.
For instance, the field mongodb.log.severity in 6.5 is replaced by log.level in 7.10 but according to 7.10 documentation mongodb.log.severity is defined as an alias to log.level. However mongodb.log.sevirity is not displayed anymore in Kibana Discover view left pannel i.e. only mongodb.log.component and mongodb.log.context are still available.

When inspecting the request invoked from Kibana dasboard, I always get 0 hints albeit I have many log entries in my index filebeat-7.10.2-mongodb-2021.01.27-000001managed by ilm.

Please help since I do not see anymore direction to investigate further more.
Thanks & Best Regards

      "query": {
        "bool": {
          "must": [
            {
              "query_string": {
                "query": "mongodb.log.severity: *",
                "analyze_wildcard": true,
                "time_zone": "Europe/Paris"
              }
            },
            {
              "query_string": {
                "query": "*",
                "analyze_wildcard": true,
                "time_zone": "Europe/Paris"
              }
            }
          ],

image
image921×431 27.7 KB

#2

Hi @Marius_Dragomir, I noticed your gave some help on a similar case last Sept 2020 ... I just would like to add that I do face the same issue whith nginx dashboard : no data displayed ... Please note that I have a specific index per application I am collecting the logs from : mongodb, nginx, ... :

image
image997×516 50.6 KB

#3

I noticed that Kibana is offering 2 dashboards relative to MongoDB:

  • Overview [Filebeat MongoDB]
  • [Filebeat MongoDB] Overview ECS

The second one is displaying all MongoDB log information as expected because he does filter the field log.level.

I have no idea why we have 2 distinctive dashboards for MongoDB in Elastic 7.10.2 with such opposite behaviors ?

Thanks

  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "should": [
              {
                "exists": {
                  "field": "log.level"
                }
              }
            ],
            "minimum_should_match": 1
          }
        },
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2021-01-28T09:24:04.075Z",
              "lte": "2021-01-28T10:24:04.075Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
#4

Same remark for Nginx dashboard ... ECS does the job as expected :smiley:

#5

You may want to ask your beats and ECS questions in the separate Beats forum that we have, monitored by the teams that work on those products.

I can help you with your Discover question, which has a simple answer: alias type fields, and multi-mapped fields in general, are not shown in Discover. We have recently merged changes to this which will get rolled out in future versions of the stack, but for now this is expected behavior.

I can't help you understand why there are 2 similar dashboards, but it seems like one of these 2 things is true:

a. Maybe one of the dashboards was carried over from previous versions of filebeat
b. Maybe this is intentional and filebeat expects you to analyze the data in 2 different ways

#6

Thanks for your supprot @wylie . I guess option a. Maybe one of the dashboards was carried over from previous versions of filebeat is the good one. I gonna try to retag my case as Beat to get Beats experts feedbacks.

Have a good day.