# Filter input data from Filebeat using logstash?

**URL:** <https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078>\
**Category:** Logstash\
**Created:** [December 28, 2022, 8:34am UTC](https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078 "2022-12-28T08:34:27Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![camilovietnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/camilovietnam/32/115267_2.png) [@camilovietnam](https://discuss.elastic.co/u/camilovietnam)\
**Post date:** [December 28, 2022, 8:34am UTC](https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078/1 "2022-12-28T08:34:27Z")

</div>

Hello! I managed to set up the stack Filebeat-\>Logstash-\>Elasticsearch, but I am using journald as an input for my filebeat logs, which means that a lot of unnecessary data appears to be saved in the ES index. I thought the mappings in my logstash config would only let through the declared properties, so my template currently looks like this:

```auto
{
  "template": "logstash",
  "index_patterns": [
    "logstash-*"
  ],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "level": {
        "type": "byte"
      },
      "request-id": {
        "type": "keyword"
      },
      "app-id": {
        "type": "keyword"
      },
      "instance-id": {
        "type": "keyword"
      },
      "route": {
        "type": "text"
      },
      "ip": {
        "type": "ip"
      },
      "@timestamp": {
        "type": "date"
      }
    }
  }
}

```

However, a search against ES returns a lot of fields, so I was wondering where exactly in the stack should I write some kind of filter to trim all unnecessary data before it is being stored in ES. I'd appreciate a push in the right direction.

---

<div class="post-metadata">

**Author:** ![dadiasish](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadiasish/32/114221_2.png) [@dadiasish](https://discuss.elastic.co/u/dadiasish)\
**Post date:** [December 28, 2022, 9:24am UTC](https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078/2 "2022-12-28T09:24:56Z")

</div>

Hi,

One way of filtering out unnecessary fields from events in logstash is by using the drop/remove field filter.

This field will remove those fields and only the remaining fields will be mapped or stored in Elasticsearch.

```auto
    filter {
      drop {
        remove_field => ["foo_%{somefield}"]
      }
    }

```

You can refer the below doc if you need additional information.

> **[Drop filter plugin | Logstash Reference \[8.5\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html#plugins-filters-drop-remove_field)**

Thanks,  
Asish

---

<div class="post-metadata">

**Author:** ![camilovietnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/camilovietnam/32/115267_2.png) [@camilovietnam](https://discuss.elastic.co/u/camilovietnam)\
**Post date:** [December 28, 2022, 10:03am UTC](https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078/3 "2022-12-28T10:03:19Z")

</div>

Yes, perfect! Thank you very much. Would the `filter { drop { remove_field ... }}}` do the same as something like `filter { mutate { remove_field ...} }` ?

---

<div class="post-metadata">

**Author:** ![dadiasish](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadiasish/32/114221_2.png) [@dadiasish](https://discuss.elastic.co/u/dadiasish)\
**Post date:** [December 28, 2022, 10:05am UTC](https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078/4 "2022-12-28T10:05:15Z")

</div>

Mutate do not have a remove field filter.

See the below doc.

> **[Mutate filter plugin | Logstash Reference \[8.5\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html)**

So it's better you go with the format I've given.

---

<div class="post-metadata">

**Author:** ![camilovietnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/camilovietnam/32/115267_2.png) [@camilovietnam](https://discuss.elastic.co/u/camilovietnam)\
**Post date:** [December 28, 2022, 10:13am UTC](https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078/5 "2022-12-28T10:13:05Z")

</div>

According to your link:  
[Mutate filter plugin | Logstash Reference [8.5] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html) "The following configuration options are supported by all filter plugins" and then it lists common options, included remove\_field 😃 I actually tested it with `mutate {remove_field ... }` and it appears to be working.

 ![Screenshot 2022-12-28 at 11.12.23](https://us1.discourse-cdn.com/elastic/original/3X/1/c/1cd352c82eaca28f8779007d1ca171415c12b622.png)

Both versions seem to work, thanks again! I will continue reading about plugins and stuff

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 28, 2022, 3:54pm UTC](https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078/6 "2022-12-28T15:54:38Z")

</div>

> [@camilovietnam](#):
>
> I thought the mappings in my logstash config would only let through the declared properties

The default for elasticsearch is to enable [dynamic mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html), so that any new field on a document creates a new field in the index. You can turn that off, so that only fields in your template are created.

---

<div class="post-metadata">

**Author:** ![camilovietnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/camilovietnam/32/115267_2.png) [@camilovietnam](https://discuss.elastic.co/u/camilovietnam)\
**Post date:** [December 30, 2022, 4:48pm UTC](https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078/7 "2022-12-30T16:48:57Z")

</div>

Sounds like an useful parameter, if it just worked...I checked your link to figure out how to use it:

> **[dynamic | Elasticsearch Guide \[8.5\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html#dynamic)**

From the examples in the page it seems like I need to add this to my `mappings` object, so something like this:

```auto
{
  "template": "logstash",
  "index_patterns": [
    "logstash-*"
  ],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "dynamic": false,
    "properties": {
      "level": {
        "type": "byte"
      }
    }
  }
}

```

So this should save only documents with a "level" field, right? Of course, it doesn't work. When I check the documents in Elasticsearch, they contain the entire list of fields sent by Logstash:

 ![Screenshot 2022-12-30 at 17.38.58](https://us1.discourse-cdn.com/elastic/original/3X/6/5/651e2bc9ce829b7b62b468a3ab99511aa0a9653f.png)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 31, 2022, 1:06am UTC](https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078/9 "2022-12-31T01:06:29Z")

</div>

If you [get the mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html) what is the value for dynamic?

---

<div class="post-metadata">

**Author:** ![camilovietnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/camilovietnam/32/115267_2.png) [@camilovietnam](https://discuss.elastic.co/u/camilovietnam)\
**Post date:** [January 2, 2023, 12:26pm UTC](https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078/10 "2023-01-02T12:26:30Z")

</div>

Thanks, but after a discussion with my peers I learned that we should be switching to Opensearch, and instead of sending from Logstash directly to Elasticsearch I need to send from Logstash to Graylog using the GELF format. This opened up an entire new family of issues to figure out, so this thread should be considered closed as of now.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 2, 2023, 12:26pm UTC](https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078/11 "2023-01-02T12:26:30Z")

</div>

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 30, 2023, 12:26pm UTC](https://discuss.elastic.co/t/filter-input-data-from-filebeat-using-logstash/322078/12 "2023-01-30T12:26:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.