# Doubt about Elasticsearch module of Filebeat on container \[7.9.2\]

**URL:** <https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [May 5, 2021, 1:05pm UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189 "2021-05-05T13:05:24Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\
**Post date:** [May 5, 2021, 1:05pm UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/1 "2021-05-05T13:05:24Z")

</div>

Hi, everyone

I have been testing with [Elasticsearch module of Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-elasticsearch.html) in order to have information about **Elasticsearch**.

I work with **Kubernetes** and I have created a deployment with **Filebeat**. Here are you are my **Filebeat configuration** :

```auto
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
      - type: kubernetes
        templates:
        - condition.equals.kubernetes.container.name: "elasticsearch"
          config:
          - module: elasticsearch
            server:
              input.type: container
              input.paths: "/var/log/containers/*-${data.kubernetes.container.id}.log"
    processors:
    - convert:
        fields:
        - from: "@metadata.pipeline"
          to: "fields.pipeline"
    logging.level: warning
    output.kafka:
     hosts: ['<kafka>:<kafka-port>']
     topic: "elasticsearch-log

```

It works fine, **Elasticsearch's log** is parsed properly. However, it creates a field **host.name with the value of elasticsearch.node.name** :

```auto
    "elasticsearch": {
      "server": {
        "cluster": {},
        "node": {}
      },
      "cluster": {
        "name": "<cluster-name>",
        "uuid": "<uuid>"
      },
      "node": {
        "name": "elasticsearch-0-0",
        "id": "<id>"
      },
      "component": "o.e.c.m.MetadataMappingService",
      "index": {
        "name": "<some-index>",
        "id": "<id>"
      }
    },
    "service": {
      "type": "elasticsearch"
    },
    "@version": "1",
    "host": {
      "name": "elasticsearch-0-0",
      "id": "<id>"
    }

```

When it comes to a **container** , is it right this behaviour? Should it exist **host.name**?

Thanks in advance,

Rodrigo

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [May 5, 2021, 10:22pm UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/2 "2021-05-05T22:22:58Z")

</div>

Yes, see [Host Fields | Elastic Common Schema (ECS) Reference [1.9] | Elastic](https://www.elastic.co/guide/en/ecs/current/ecs-host.html#ecs-host). The `host.*` fields describe the host/container that generates the log/event.

---

<div class="post-metadata">

**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\
**Post date:** [May 5, 2021, 10:36pm UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/3 "2021-05-05T22:36:14Z")

</div>

And just to connect the individual pieces here:

- The ingest pipeline doesn't make a difference between Docker or no-Docker setup. It [sets `{{elasticsearch.node.name}}` for `host.name`](https://github.com/elastic/beats/blob/master/filebeat/module/elasticsearch/server/ingest/pipeline.yml#L70-L73).
- The Elasticsearch ["node name defaults to the hostname"](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#node-name) unless you set it explicitly.

I guess the question is then: What would you expect instead — `host.name` shouldn't be set at all?

---

<div class="post-metadata">

**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\
**Post date:** [May 6, 2021, 6:51am UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/4 "2021-05-06T06:51:09Z")

</div>

Hi, @legoguy1000 & @xeraa

My concern is about **Metrics visualization (Observability --\> Metrics)** which is showing a **hostname for which I don't have any metrics**. Looking over the **query** I have found out it is retrieving as i **ndex\_patterns or alias** in both ( **Logs** & **Metrics** ).

```auto
query SourceQuery($sourceId: ID = "default") { source(id: $sourceId) { ...InfraSourceFields configuration { ...SourceConfigurationFields } status { ...SourceStatusFields } }}fragment InfraSourceFields on InfraSource { id version updatedAt origin}fragment SourceConfigurationFields on InfraSourceConfiguration { name description logAlias metricAlias inventoryDefaultView metricsExplorerDefaultView fields { container host message pod tiebreaker timestamp } logColumns { ... on InfraSourceTimestampLogColumn { timestampColumn { id } } ... on InfraSourceMessageLogColumn { messageColumn { id } } ... on InfraSourceFieldLogColumn { fieldColumn { id field } } }}fragment SourceStatusFields on InfraSourceStatus { indexFields { name type searchable aggregatable displayable } logIndicesExist metricIndicesExist}

```

Here you are a screenshot about my dev environment:

- **labs-dev-k8s0[1-3]**: I have installed a **Metricbeat**
- **elasticsearch-0-0** : It is a pod, I should only obtain its metrics in **Observability --\> Metrics --\> Show --\> Kubernetes Pods**.

 ![observability-metrics](https://us1.discourse-cdn.com/elastic/original/3X/c/7/c7e4080df620e21523bcf703bff5fdecb01e31e9.png)

From my point of view, in **Observability --\> Metrics --\> Show --\> Hosts** should show only hosts (virtual machines, servers and so on) with **metrics**.

For instance, If I have **a virtual machine with PostgreSQL** in which I have installed/configured **Filebeat** in order to send logs to **Elasticsearch** and I have added its **index\_pattern in Observability --\> Logs** , it should not be displayed in **Metrics**.

Thanks in advance,

Rodrigo

---

<div class="post-metadata">

**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\
**Post date:** [May 8, 2021, 12:29am UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/5 "2021-05-08T00:29:35Z")

</div>

That's a good point 🙂

I think [[Filebeat] The host.name sent from Filebeat doesn't match the same field from Metricbeat · Issue #13589 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/13589) is related though it's coming from a different angle. Can you add your specific problem to the issue (if you agree that this is the same underlying issue)?

Also there is a possible [workaround in the issue](https://github.com/elastic/beats/issues/13589#issuecomment-688741290):

```auto
processors:
  - add_kubernetes_metadata
  - drop_fields:
      fields: ["host.name"]
      ignore_missing: true
  - copy_fields:
      fields:
        - from: kubernetes.node.name
          to: host.name
      fail_on_error: false
      ignore_missing: true

```

---

<div class="post-metadata">

**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\
**Post date:** [May 10, 2021, 6:53am UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/6 "2021-05-10T06:53:54Z")

</div>

Hi, @xeraa

I use that method for **Kubernetes metrics**. However, in this case, I would like to know why **logs alias** is used in **Metrics** ( **Observability --\> Metrics** ).

I come back to the **example of a virtual machine** where I have installed a **PostgreSQL**. I have installed/configured **Filebeat** in order to send logs to **Elasticsearch** and I have added its **index\_pattern in Observability --\> Logs**.

Why the hostname of the virtual machine appears in **Observability --\> Metrics**?

Thanks in advance,

Rodrigo

---

<div class="post-metadata">

**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\
**Post date:** [May 11, 2021, 12:34am UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/7 "2021-05-11T00:34:33Z")

</div>

![Screenshot 2021-05-11 at 02.33.31](https://us1.discourse-cdn.com/elastic/original/3X/4/b/4b9772efe7e416d9014a3266f5f667704a61e2be.png)

This one here? That generally shouldn't use the logs alias. Also this is a different question now, right?

---

<div class="post-metadata">

**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\
**Post date:** [May 11, 2021, 6:14am UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/8 "2021-05-11T06:14:28Z")

</div>

Hi, @xeraa

My last question has a link with main topic. As I told you, I have created a virtual machine ( **labs-postgresql** ) in which I have installed **Filebeat**.

**Logs configuration:**

 ![logs-01](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3d0115abde744f0e29603f547a56c682b591a6a6.png)  
 ![logs-02](https://us1.discourse-cdn.com/elastic/original/3X/8/d/8d1f80565a08feca5d1916e4a2359cbec19eb19a.png)

**Metrics configuration:**

 ![metrics-01](https://us1.discourse-cdn.com/elastic/original/3X/3/7/37f3c81a6e9ae7176cdaea59596f676582dc4dd4.png)  
 ![metrics-02](https://us1.discourse-cdn.com/elastic/original/3X/f/3/f3a8b5bfe1b7f7a1e30c0e94f426da63b669bb76.png)

As you can see, **labs-postgresql** appears in **Metrics** , I have not installed a **Metricbeat** in that **virtual machine**.

Is it right this behaviour?

Thanks in advance,

Rodrigo

---

<div class="post-metadata">

**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\
**Post date:** [May 11, 2021, 7:23am UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/9 "2021-05-11T07:23:23Z")

</div>

If you click on any of the specific hosts you should be able to drill down to a specific document that it uses for these metrics, can you try that and see which event.module is set for that document?

---

<div class="post-metadata">

**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\
**Post date:** [May 11, 2021, 8:21am UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/10 "2021-05-11T08:21:47Z")

</div>

Hi, @Marius_Iversen

The only data about my machine ( **labs-postgresql** ) in that cluster belongs to **Filebeat**.

 ![postgresql-discover](https://us1.discourse-cdn.com/elastic/original/3X/3/2/32d11e272c41741556ab6dee5000b27782e8bcd0.png)

 ![metricbeat-discover](https://us1.discourse-cdn.com/elastic/original/3X/9/b/9b1925836f26af6ac633e2f3f42f058773f241af.png)

My question/doubt is about if **Metrics** should show information from other index than **Metricbeat** index ( **metricbeat-** \*).

Thanks in advance,

Rodrigo

---

<div class="post-metadata">

**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\
**Post date:** [May 11, 2021, 11:59am UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/11 "2021-05-11T11:59:53Z")

</div>

It shouldn't in theory, but maybe you do have a metricbeat index or a index pattern that includes your own files.

In the Kibana dev tools, try to run

```
GET metricbeat/_search
{
    "query": {
        "match_all": {}
    }
}
```

---

<div class="post-metadata">

**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\
**Post date:** [May 12, 2021, 8:53am UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/12 "2021-05-12T08:53:09Z")

</div>

Hi, @Marius_Iversen

I have tested with **Elastic Stack 7.12.1** and it works. I think it was a bug of **7.9.2**.

Thank you very much,

Rodrigo

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 9, 2021, 10:53am UTC](https://discuss.elastic.co/t/doubt-about-elasticsearch-module-of-filebeat-on-container-7-9-2/272189/13 "2021-06-09T10:53:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.