Kibana - Stack Monitoring - No structured logs found

Elastic Stack Kibana
1

Hi All,

We have an issue where Kibana throws a "No structured logs found" message in the Log section when viewing the Stack Monitoring page, where it should show the last 10 log entries.

I have Filebeat running on each of the ES nodes using the standard configuration (with updated var.paths to our log file locations) as documented here https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-elasticsearch.html

The logs are being successfuly written to an index with name filebeat-7.8.0-elasticsearch. Can anyone help to identify why Kibana does not display them when viewing the Stack Monitoring page?

Elasticsearch, Beats, Logstash, and Kibana all version 7.8.0.

no_structured_logs

Thanks

2

Can you share your Elasticsearch filebeat module yml file?

You’ll need to make sure you are reading the json logs

3

Sure thing.

# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-module-elasticsearch.html

- module: elasticsearch
  # Server log
  server:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      #- /elk/elasticsearch/xyz/logs/xyz-cluster.log
      - /elk/elasticsearch/xyz/logs/xyz-cluster_server.json

  gc:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - /elk/elasticsearch/xyz/logs/gc.log.[0-9]*
      - /elk/elasticsearch/xyz/logs/gc.log

  audit:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - /elk/elasticsearch/xyz/logs/*_audit.json

  slowlog:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - /elk/elasticsearch/xyz/logs/*_index_search_slowlog.json
      - /elk/elasticsearch/xyz/logs/*_index_indexing_slowlog.json

  deprecation:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - /elk/elasticsearch/xyz/logs/*_deprecation.json
4

Can you temporarily disable this and see if it fixes the problem?

5

I tried it with the below for 4+ hours. No difference.

- module: elasticsearch
  # Server log
  server:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - /elk/elasticsearch/xyz/logs/*_server.json

  gc:
    enabled: false
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - /elk/elasticsearch/xyz/logs/gc.log.[0-9]*
      - /elk/elasticsearch/xyz/logs/gc.log
...
6

Can you try running this query against the cluster that stores the filebeat data?

POST filebeat-*/_search?filter_path=hits.hits._source.event.dataset,hits.hits._source.@timestamp,hits.hits._source.elasticsearch
{
  "size": 10,
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ],
  "collapse": {
    "field": "event.dataset"
  }
}
7

Hey Chris,

Yep, this is the output

{
  "hits" : {
    "hits" : [
      {
        "_source" : {
          "event" : {
            "dataset" : "elasticsearch.server"
          },
          "@timestamp" : "2020-07-16T22:43:01.510Z"
        }
      },
      {
        "_source" : {
          "event" : {
            "dataset" : "system.syslog"
          },
          "@timestamp" : "2020-07-16T22:42:53.423Z"
        }
      },
      {
        "_source" : {
          "@timestamp" : "2020-07-16T22:42:53.107Z"
        }
      },
      {
        "_source" : {
          "@timestamp" : "2020-07-16T11:26:47.433Z",
          "event" : {
            "dataset" : "system.auth"
          }
        }
      },
      {
        "_source" : {
          "@timestamp" : "2020-07-16T03:57:37.601Z",
          "event" : {
            "dataset" : "elasticsearch.gc"
          }
        }
      },
      {
        "_source" : {
          "@timestamp" : "2020-07-09T13:01:41.489Z",
          "event" : {
            "dataset" : "elasticsearch.deprecation"
          }
        }
      }
    ]
  }
}
8

Thanks.

It seems likely the gc collection is the problem here, as it does not have structured logs.

There are two things we can do about this:

  1. File a ticket in Kibana to ensure that the logs continue to show in the Stack Monitoring UI, even if there are some indexed logs that are collected from non structured logs

  2. File a ticket in Elasticsearch and ask if we can get the gc logs as .json documents too. I'm not sure why that doesn't exist now

9

Thanks Chris. How does the gc collection affect it though, as it still didn't work even after i disabled the gc logs for several hours?

I'm happy to file tickets, assuming you just mean issues on the git repo's?

10

That's a good point.

This is the exact query we make for that page:

POST filebeat-*/_search
{
  "size": 0,
  "sort": {
    "@timestamp": {
      "order": "desc"
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "service.type": "elasticsearch"
          }
        },
        {
          "range": {
            "@timestamp": {
              "format": "epoch_millis",
              "gte": 1595594404844,
              "lte": 1595598004844
            }
          }
        },
        {
          "term": {
            "elasticsearch.cluster.uuid": "{cluster_uuid}"
          }
        }
      ]
    }
  },
  "aggs": {
    "types": {
      "terms": {
        "field": "event.dataset"
      },
      "aggs": {
        "levels": {
          "terms": {
            "field": "log.level"
          }
        }
      }
    }
  }
}

You can adjust the time period to be relative time to test too:

"range": {
  "@timestamp": {
    "gte": "now-1h",
    "lte": "now"
  }
}

Run that for the specified period in your time picker and lemme know what it returns

11

Hey Chris, so it gives back:

{
  "took" : 126,
  "timed_out" : false,
  "_shards" : {
    "total" : 45,
    "successful" : 45,
    "skipped" : 26,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "types" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [ ]
    }
  }
}

Looking over the index in kibana none of the logs are populating the elasticsearch.cluster.uuid field.

12

It looks like the server json logs all use cluster.uuid and looking over the filebeat elasticsearch module in github it seems to expect the field to be elasticsearch.server.cluster.uuid which gets renamed to elasticsearch.cluster.uuid

13

Can you share your mappings and at least one filebeat pipeline?

GET filebeat-*/_mapping - You can just return the properties.elasticsearch json blob here
GET _ingest/pipeline/filebeat-8.0.0-elasticsearch-server-pipeline

Then let's see some full documents:

POST filebeat-*/_search
{
  "size": 20,
  "sort": {
    "@timestamp": {
      "order": "desc"
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "service.type": "elasticsearch"
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-1h",
              "lte": "now"
            }
          }
        }
      ]
    }
  },
  "collapse": {
    "field": "event.dataset"
  }
}
14

GET filebeat-*/_mapping

"elasticsearch" : {
          "properties" : {
            "audit" : {
              "properties" : {
                "action" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "event_type" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "indices" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "layer" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "message" : {
                  "type" : "text",
                  "norms" : false
                },
                "origin" : {
                  "properties" : {
                    "type" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    }
                  }
                },
                "realm" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "request" : {
                  "properties" : {
                    "id" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    },
                    "name" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    }
                  }
                },
                "url" : {
                  "properties" : {
                    "params" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    }
                  }
                },
                "user" : {
                  "properties" : {
                    "realm" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    },
                    "roles" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    }
                  }
                }
              }
            },
            "cluster" : {
              "properties" : {
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "uuid" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "component" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "deprecation" : {
              "type" : "object"
            },
            "gc" : {
              "properties" : {
                "heap" : {
                  "properties" : {
                    "size_kb" : {
                      "type" : "long"
                    },
                    "used_kb" : {
                      "type" : "long"
                    }
                  }
                },
                "jvm_runtime_sec" : {
                  "type" : "float"
                },
                "old_gen" : {
                  "properties" : {
                    "size_kb" : {
                      "type" : "long"
                    },
                    "used_kb" : {
                      "type" : "long"
                    }
                  }
                },
                "phase" : {
                  "properties" : {
                    "class_unload_time_sec" : {
                      "type" : "float"
                    },
                    "cpu_time" : {
                      "properties" : {
                        "real_sec" : {
                          "type" : "float"
                        },
                        "sys_sec" : {
                          "type" : "float"
                        },
                        "user_sec" : {
                          "type" : "float"
                        }
                      }
                    },
                    "duration_sec" : {
                      "type" : "float"
                    },
                    "name" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    },
                    "parallel_rescan_time_sec" : {
                      "type" : "float"
                    },
                    "scrub_string_table_time_sec" : {
                      "type" : "float"
                    },
                    "scrub_symbol_table_time_sec" : {
                      "type" : "float"
                    },
                    "weak_refs_processing_time_sec" : {
                      "type" : "float"
                    }
                  }
                },
                "stopping_threads_time_sec" : {
                  "type" : "float"
                },
                "tags" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "threads_total_stop_time_sec" : {
                  "type" : "float"
                },
                "young_gen" : {
                  "properties" : {
                    "size_kb" : {
                      "type" : "long"
                    },
                    "used_kb" : {
                      "type" : "long"
                    }
                  }
                }
              }
            },
            "index" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "node" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "server" : {
              "properties" : {
                "gc" : {
                  "properties" : {
                    "collection_duration" : {
                      "properties" : {
                        "ms" : {
                          "type" : "float"
                        }
                      }
                    },
                    "observation_duration" : {
                      "properties" : {
                        "ms" : {
                          "type" : "float"
                        }
                      }
                    },
                    "overhead_seq" : {
                      "type" : "long"
                    },
                    "young" : {
                      "properties" : {
                        "one" : {
                          "type" : "long"
                        },
                        "two" : {
                          "type" : "long"
                        }
                      }
                    }
                  }
                },
                "stacktrace" : {
                  "type" : "keyword",
                  "index" : false,
                  "ignore_above" : 1024
                }
              }
            },
            "shard" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "slowlog" : {
              "properties" : {
                "extra_source" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "logger" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "routing" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "search_type" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "source" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "source_query" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "stats" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "took" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "total_hits" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "total_shards" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "type" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "types" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            }
          }
        },

GET _ingest/pipeline/filebeat-8.0.0-elasticsearch-server-pipeline returns a 404

15

Search results (note i cleaned up the log content by deleting the majority of the stack trace

{
  "took" : 4544,
  "timed_out" : false,
  "_shards" : {
    "total" : 48,
    "successful" : 45,
    "skipped" : 24,
    "failed" : 3,
    "failures" : [
      {
        "shard" : 0,
        "index" : "filebeat-7.8.0",
        "node" : "njKm_18uRcugxs6ZVODAtA",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "unknown type for collapse field `event.dataset`, only keywords and numbers are accepted"
        }
      }
    ]
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : "filebeat-7.8.0-elasticsearch-000008",
        "_type" : "_doc",
        "_id" : "0OXlknMBPeMNtYkwpVEh",
        "_score" : null,
        "_source" : {
          "service" : {
            "type" : "elasticsearch"
          },
          "ls_node" : "xyz204",
          "log" : {
            "file" : {
              "path" : "/elk/elasticsearch/xyz/logs/xyz-cluster_server.json"
            },
            "flags" : [
              "multiline"
            ],
            "offset" : 61583874,
            "original" : """{"type": "server", "timestamp": "2020-07-28T10:41:40,398+10:00", "level": "INFO", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "xyz-cluster", "node.name": "xyz202", "message": "[testxyz-000002][1] mapping update rejected by primary", "cluster.uuid": "ZZKjFPMkQIqdSHpz1H-oeQ", "node.id": "X2mCgm_WTP2iyZbS2bquVg" , 
"stacktrace": ["java.lang.IllegalArgumentException: Limit of total fields [4000] in index  }"""
          },
          "@version" : "1",
          "fileset" : {
            "name" : "server"
          },
          "@timestamp" : "2020-07-28T00:41:40.572Z",
          "agent" : {
            "version" : "7.8.0",
            "hostname" : "xyz202.xyz.xyz.xyz",
            "id" : "f4f68361-80e4-4257-b08b-68490facc16a",
            "name" : "xyz202.xyz.xyz.xyz",
            "ephemeral_id" : "b8f62b1a-3d86-40e1-90b5-fa57814efc2d",
            "type" : "filebeat"
          },
          "input" : {
            "type" : "log"
          },
          "host" : {
            "containerized" : false,
            "ip" : [
              "10.",
              "10.",
              "fe80:"
            ],
            "hostname" : "xyz202.xyz.xyz.xyz",
            "id" : "d83f39e122e947898a4e895eb130f93b",
            "name" : "xyz202.xyz.xyz.xyz",
            "mac" : [
              "24:6e:9
            ],
            "os" : {
              "family" : "redhat",
              "version" : "7.8 (Maipo)",
              "kernel" : "3.10.0-1127.el7.x86_64",
              "codename" : "Maipo",
              "name" : "Red Hat Enterprise Linux Server",
              "platform" : "rhel"
            },
            "architecture" : "x86_64"
          },
          "event" : {
            "module" : "elasticsearch",
            "dataset" : "elasticsearch.server"
          },
          "tags" : [
            "elasticsearch-host",
            "beats_input_codec_plain_applied",
            "logfilename_grok_failure"
          ],
          "message" : """{"type": "server", "timestamp": "2020-07-28T10:41:40,398+10:00", "level": "INFO", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "xyz-cluster", "node.name": "xyz202", "message": "[testxyz-000002][1] mapping update rejected by primary", "cluster.uuid": "ZZKjFPMkQIqdSHpz1H-oeQ", "node.id": "X2mCgm_WTP2iyZbS2bquVg" , 
"stacktrace": ["java.lang.IllegalArgumentException: Limit of total fields [4000] in in] }""",
          "ecs" : {
            "version" : "1.5.0"
          }
        },
        "fields" : {
          "event.dataset" : [
            "elasticsearch.server"
          ]
        },
        "sort" : [
          1595896900572
        ]
      }
    ]
  }
}
16

My bad. Replace 8.0.0 with the 7.8.0. You can double check the names through GET _ingest/pipeline

17

Still a 404. This is GET _ingest/pipeline

{
  "xpack_monitoring_6" : {
    "description" : "This pipeline upgrades documents from the older version of the Monitoring API to the newer version (7) by fixing breaking changes in those older documents before they are indexed from the older version (6).",
    "version" : 7000199,
    "processors" : [
      {
        "script" : {
          "source" : "ctx._type = null"
        }
      },
      {
        "gsub" : {
          "field" : "_index",
          "pattern" : """(.monitoring-\w+-)6(-.+)""",
          "replacement" : "$17$2"
        }
      }
    ]
  },
  "xpack_monitoring_7" : {
    "description" : "This is a placeholder pipeline for Monitoring API version 7 so that future versions may fix breaking changes.",
    "version" : 7000199,
    "processors" : [ ]
  }
}
18

It looks like those pipelines don't exist for some reason.

See https://www.elastic.co/guide/en/beats/filebeat/master//load-ingest-pipelines.html for helping getting those installed.

19

Hey Chris, I've loaded the ingest pipeline for elasticsearch now, but that hasn't fixed it still it seems.

{
  "filebeat-7.8.0-elasticsearch-server-pipeline" : {
    "description" : "Pipeline for parsing elasticsearch server logs",
    "processors" : [
      {
        "rename" : {
          "field" : "@timestamp",
          "target_field" : "event.created"
        }
      },
      {
        "grok" : {
          "field" : "message",
          "patterns" : [
            "^%{CHAR:first_char}"
          ],
          "pattern_definitions" : {
            "CHAR" : "."
          }
        }
      },
      {
        "pipeline" : {
          "name" : "filebeat-7.8.0-elasticsearch-server-pipeline-plaintext",
          "if" : "ctx.first_char != '{'"
        }
      },
      {
        "pipeline" : {
          "if" : "ctx.first_char == '{'",
          "name" : "filebeat-7.8.0-elasticsearch-server-pipeline-json"
        }
      },
      {
        "script" : {
          "lang" : "painless",
          "source" : """if (ctx.elasticsearch.server.gc != null && ctx.elasticsearch.server.gc.observation_duration != null) {
  if (ctx.elasticsearch.server.gc.observation_duration.unit == params.seconds_unit) {
    ctx.elasticsearch.server.gc.observation_duration.ms = ctx.elasticsearch.server.gc.observation_duration.time * params.ms_in_one_s;
  }
  if (ctx.elasticsearch.server.gc.observation_duration.unit == params.milliseconds_unit) {
    ctx.elasticsearch.server.gc.observation_duration.ms = ctx.elasticsearch.server.gc.observation_duration.time;
  }
  if (ctx.elasticsearch.server.gc.observation_duration.unit == params.minutes_unit) {
    ctx.elasticsearch.server.gc.observation_duration.ms = ctx.elasticsearch.server.gc.observation_duration.time * params.ms_in_one_m;
  }
} if (ctx.elasticsearch.server.gc != null && ctx.elasticsearch.server.gc.collection_duration != null) {
  if (ctx.elasticsearch.server.gc.collection_duration.unit == params.seconds_unit) {
    ctx.elasticsearch.server.gc.collection_duration.ms = ctx.elasticsearch.server.gc.collection_duration.time * params.ms_in_one_s;
  }
  if (ctx.elasticsearch.server.gc.collection_duration.unit == params.milliseconds_unit) {
    ctx.elasticsearch.server.gc.collection_duration.ms = ctx.elasticsearch.server.gc.collection_duration.time;
  }
  if (ctx.elasticsearch.server.gc.collection_duration.unit == params.minutes_unit) {
    ctx.elasticsearch.server.gc.collection_duration.ms = ctx.elasticsearch.server.gc.collection_duration.time * params.ms_in_one_m;
  }
}""",
          "params" : {
            "milliseconds_unit" : "ms",
            "ms_in_one_s" : 1000,
            "ms_in_one_m" : 60000,
            "minutes_unit" : "m",
            "seconds_unit" : "s"
          }
        }
      },
      {
        "set" : {
          "field" : "event.kind",
          "value" : "event"
        }
      },
      {
        "set" : {
          "value" : "database",
          "field" : "event.category"
        }
      },
      {
        "script" : {
          "lang" : "painless",
          "source" : """def errorLevels = ['FATAL', 'ERROR']; if (ctx?.log?.level != null) {
  if (errorLevels.contains(ctx.log.level)) {
    ctx.event.type = 'error';
  } else {
    ctx.event.type = 'info';
  }
}"""
        }
      },
      {
        "set" : {
          "field" : "host.name",
          "value" : "{{elasticsearch.node.name}}",
          "if" : "ctx?.elasticsearch?.node?.name != null"
        }
      },
      {
        "set" : {
          "field" : "host.id",
          "value" : "{{elasticsearch.node.id}}",
          "if" : "ctx?.elasticsearch?.node?.id != null"
        }
      },
      {
        "remove" : {
          "field" : [
            "elasticsearch.server.gc.collection_duration.time",
            "elasticsearch.server.gc.collection_duration.unit",
            "elasticsearch.server.gc.observation_duration.time",
            "elasticsearch.server.gc.observation_duration.unit"
          ],
          "ignore_missing" : true
        }
      },
      {
        "remove" : {
          "field" : [
            "elasticsearch.server.timestamp",
            "elasticsearch.server.@timestamp"
          ],
          "ignore_missing" : true
        }
      },
      {
        "remove" : {
          "field" : [
            "first_char"
          ]
        }
      }
    ],
    "on_failure" : [
      {
        "set" : {
          "field" : "error.message",
          "value" : "{{ _ingest.on_failure_message }}"
        }
      }
    ]
  }
}
20

Does this exist?

GET _ingest/pipeline/filebeat-7.8.0-elasticsearch-server-pipeline-json