Doubt about Elasticsearch module of Filebeat on container [7.9.2]

[RdrgPorto (Rodrigo Porto)]

Hi, everyone

I have been testing with Elasticsearch module of Filebeat in order to have information about Elasticsearch.

I work with Kubernetes and I have created a deployment with Filebeat. Here are you are my Filebeat configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
      - type: kubernetes
        templates:
        - condition.equals.kubernetes.container.name: "elasticsearch"
          config:
          - module: elasticsearch
            server:
              input.type: container
              input.paths: "/var/log/containers/*-${data.kubernetes.container.id}.log"
    processors:
    - convert:
        fields:
        - from: "@metadata.pipeline"
          to: "fields.pipeline"
    logging.level: warning
    output.kafka:
     hosts: ['<kafka>:<kafka-port>']
     topic: "elasticsearch-log

It works fine, Elasticsearch's log is parsed properly. However, it creates a field host.name with the value of elasticsearch.node.name:

    "elasticsearch": {
      "server": {
        "cluster": {},
        "node": {}
      },
      "cluster": {
        "name": "<cluster-name>",
        "uuid": "<uuid>"
      },
      "node": {
        "name": "elasticsearch-0-0",
        "id": "<id>"
      },
      "component": "o.e.c.m.MetadataMappingService",
      "index": {
        "name": "<some-index>",
        "id": "<id>"
      }
    },
    "service": {
      "type": "elasticsearch"
    },
    "@version": "1",
    "host": {
      "name": "elasticsearch-0-0",
      "id": "<id>"
    }

When it comes to a container, is it right this behaviour? Should it exist host.name?

Thanks in advance,

Rodrigo

[legoguy1000 (Alex)]

Yes, see Host Fields | Elastic Common Schema (ECS) Reference [1.9] | Elastic. The host.* fields describe the host/container that generates the log/event.

[xeraa]

And just to connect the individual pieces here:

• The ingest pipeline doesn't make a difference between Docker or no-Docker setup. It sets {{elasticsearch.node.name}} for host.name.
• The Elasticsearch "node name defaults to the hostname" unless you set it explicitly.

I guess the question is then: What would you expect instead — host.name shouldn't be set at all?

[RdrgPorto (Rodrigo Porto)]

Hi, @legoguy1000 & @xeraa

My concern is about Metrics visualization (Observability --> Metrics) which is showing a hostname for which I don't have any metrics. Looking over the query I have found out it is retrieving as index_patterns or alias in both (Logs & Metrics).

query SourceQuery($sourceId: ID = "default") {  source(id: $sourceId) {    ...InfraSourceFields    configuration {      ...SourceConfigurationFields    }    status {      ...SourceStatusFields    }  }}fragment InfraSourceFields on InfraSource {  id  version  updatedAt  origin}fragment SourceConfigurationFields on InfraSourceConfiguration {  name  description  logAlias  metricAlias  inventoryDefaultView  metricsExplorerDefaultView  fields {    container    host    message    pod    tiebreaker    timestamp  }  logColumns {    ... on InfraSourceTimestampLogColumn {      timestampColumn {        id      }    }    ... on InfraSourceMessageLogColumn {      messageColumn {        id      }    }    ... on InfraSourceFieldLogColumn {      fieldColumn {        id        field      }    }  }}fragment SourceStatusFields on InfraSourceStatus {  indexFields {    name    type    searchable    aggregatable    displayable  }  logIndicesExist  metricIndicesExist}

Here you are a screenshot about my dev environment:

• labs-dev-k8s0[1-3]: I have installed a Metricbeat
• elasticsearch-0-0: It is a pod, I should only obtain its metrics in Observability --> Metrics --> Show --> Kubernetes Pods.

From my point of view, in Observability --> Metrics --> Show --> Hosts should show only hosts (virtual machines, servers and so on) with metrics.

For instance, If I have a virtual machine with PostgreSQL in which I have installed/configured Filebeat in order to send logs to Elasticsearch and I have added its index_pattern in Observability --> Logs, it should not be displayed in Metrics.

Thanks in advance,

Rodrigo

[xeraa]

That's a good point

I think [Filebeat] The host.name sent from Filebeat doesn't match the same field from Metricbeat · Issue #13589 · elastic/beats · GitHub is related though it's coming from a different angle. Can you add your specific problem to the issue (if you agree that this is the same underlying issue)?

Also there is a possible workaround in the issue:

processors:
  - add_kubernetes_metadata
  - drop_fields:
      fields: ["host.name"]
      ignore_missing: true
  - copy_fields:
      fields:
        - from: kubernetes.node.name
          to: host.name
      fail_on_error: false
      ignore_missing: true

[RdrgPorto (Rodrigo Porto)]

Hi, @xeraa

I use that method for Kubernetes metrics. However, in this case, I would like to know why logs alias is used in Metrics (Observability --> Metrics).

I come back to the example of a virtual machine where I have installed a PostgreSQL. I have installed/configured Filebeat in order to send logs to Elasticsearch and I have added its index_pattern in Observability --> Logs.

Why the hostname of the virtual machine appears in Observability --> Metrics?

Thanks in advance,

Rodrigo

[xeraa]

This one here? That generally shouldn't use the logs alias. Also this is a different question now, right?

[RdrgPorto (Rodrigo Porto)]

Hi, @xeraa

My last question has a link with main topic. As I told you, I have created a virtual machine (labs-postgresql) in which I have installed Filebeat.

Logs configuration:

Metrics configuration:

As you can see, labs-postgresql appears in Metrics, I have not installed a Metricbeat in that virtual machine.

Is it right this behaviour?

Thanks in advance,

Rodrigo

[Marius_Iversen]

If you click on any of the specific hosts you should be able to drill down to a specific document that it uses for these metrics, can you try that and see which event.module is set for that document?

[RdrgPorto (Rodrigo Porto)]

Hi, @Marius_Iversen

The only data about my machine (labs-postgresql) in that cluster belongs to Filebeat.

My question/doubt is about if Metrics should show information from other index than Metricbeat index (metricbeat-*).

Thanks in advance,

Rodrigo

[Marius_Iversen]

It shouldn't in theory, but maybe you do have a metricbeat index or a index pattern that includes your own files.

In the Kibana dev tools, try to run

GET metricbeat/_search
{
    "query": {
        "match_all": {}
    }
}

[RdrgPorto (Rodrigo Porto)]

Hi, @Marius_Iversen

I have tested with Elastic Stack 7.12.1 and it works. I think it was a bug of 7.9.2.

Thank you very much,

Rodrigo