# \[Filebeat\] Huge mapping (nearly 5k fields) when ingesting logs

**URL:** https://discuss.elastic.co/t/filebeat-huge-mapping-nearly-5k-fields-when-ingesting-logs/252185
**Category:** Beats
**Tags:** filebeat
**Created:** [October 15, 2020, 11:31am UTC](https://discuss.elastic.co/t/filebeat-huge-mapping-nearly-5k-fields-when-ingesting-logs/252185 "2020-10-15T11:31:03Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![M-O](https://avatars.discourse-cdn.com/v4/letter/m/3be4f8/32.png) [@M-O](https://discuss.elastic.co/u/M-O)
#### Post date: [October 15, 2020, 11:31am UTC](https://discuss.elastic.co/t/filebeat-huge-mapping-nearly-5k-fields-when-ingesting-logs/252185/1 "2020-10-15T11:31:03Z")

</div>

Hello,

i am currently experimenting with filebeat and noticed that, when filebeat creates the standard indicies, it comes with a huge count of field mappings.

 ![Screenshot_2020-10-15 Create index pattern - Elastic](https://us1.discourse-cdn.com/elastic/original/3X/2/c/2c48c3409fe4f74bd3c1a1dd71307a4455abadc2.png)

 ![Screenshot_2020-10-15 filebeat - Elastic](https://us1.discourse-cdn.com/elastic/original/3X/2/6/26c7601ff6ebf87c74d7438e54fca7840a1aa98e.png)

To me it looks like the index-template specifies every possible field for every possible filebeat module.

 ![Screenshot_2020-10-15 Screenshot](https://us1.discourse-cdn.com/elastic/original/3X/d/d/ddfd659cf7480b0b2c8e23fdabf1122843e8fb0c.png)

Is this a configuration-issue on my end?  
My filebeat.yml looks like this

```auto
    filebeat.inputs:
    - type: log
      paths:
        - /usr/share/data/access.log
      pipeline: "access_combined_wcookie_parsing_pipeline"
      processors:
        - add_tags:
            tags: ["access_combined_wcookie"]

    output.elasticsearch:
      hosts: ["elasticsearch:9200"]
      username: "${ELASTIC_USERNAME}"
      password: "${ELASTIC_PASSWORD}"

```

Any ideas why this happens? Any fixes?

Best regards,  
Mo

---

<div class="post-metadata">

### Author: ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [October 16, 2020, 9:02am UTC](https://discuss.elastic.co/t/filebeat-huge-mapping-nearly-5k-fields-when-ingesting-logs/252185/2 "2020-10-16T09:02:21Z")

</div>

I suppose it should work this way. You need to setup all fields in advance (due to the solution design). If you prefer to have only relevant fields, please take a look at the Fleet/Ingest Management in Kibana. It's the agent based environment, where you automatically install only parts you really need (e.g. a subset of fields used by enabled integrations).

---

<div class="post-metadata">

### Author: ![M-O](https://avatars.discourse-cdn.com/v4/letter/m/3be4f8/32.png) [@M-O](https://discuss.elastic.co/u/M-O)
#### Post date: [October 16, 2020, 9:08am UTC](https://discuss.elastic.co/t/filebeat-huge-mapping-nearly-5k-fields-when-ingesting-logs/252185/3 "2020-10-16T09:08:34Z")

</div>

Hey thanks for the answer.  
I as thought about it for a while it makes sense.  
It is also grouped by types (mysql, apache, etc.) (i guess) so it would makes sense now.

As you just said, Filebeat has to create every possible type-mapping because all the logs flow into the same index.

Tho imagine apache-, mysql- and aws logs are all stored in the same `filebeat-7.9.1-2020-10-16` index and i would create an index-pattern in kibana, e.g. `filebeat*`. Is there a way to filter during search only for apache-logs?  
It seems quite a lot to search through all possible filebeat logs and then to filter based on the type.

Am i missing something?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 13, 2020, 11:08am UTC](https://discuss.elastic.co/t/filebeat-huge-mapping-nearly-5k-fields-when-ingesting-logs/252185/4 "2020-11-13T11:08:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.