# Filter unstructured logs with filebeat before sending to logstash

**URL:** <https://discuss.elastic.co/t/filter-unstructured-logs-with-filebeat-before-sending-to-logstash/315960>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 6, 2022, 11:41am UTC](https://discuss.elastic.co/t/filter-unstructured-logs-with-filebeat-before-sending-to-logstash/315960 "2022-10-06T11:41:18Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ihms](https://avatars.discourse-cdn.com/v4/letter/i/3ec8ea/32.png) [@Ihms](https://discuss.elastic.co/u/Ihms)\
**Post date:** [October 6, 2022, 11:41am UTC](https://discuss.elastic.co/t/filter-unstructured-logs-with-filebeat-before-sending-to-logstash/315960/1 "2022-10-06T11:41:18Z")

</div>

Hello, I'm new to Elasticsearch and have some questions. Filebeat is fetching way too many logs and for the sake of bandwidth, I want to filter the logs at the edge before sending to logstash. The logs don't have the same pattern, so I doubt if the filbeat `dissect` can be used to achieve this.

Is there any way to achieve this and send only logs with Error and INFO log levels?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 6, 2022, 12:19pm UTC](https://discuss.elastic.co/t/filter-unstructured-logs-with-filebeat-before-sending-to-logstash/315960/2 "2022-10-06T12:19:14Z")

</div>

You need to provide more information, please share your `filebeat.yml` file and sample lines of your log file, both the ones you want to collect and the ones you do not want to collect.

You have the option to exclude lines that you do not want to send based on a regex.

---

<div class="post-metadata">

**Author:** ![Ihms](https://avatars.discourse-cdn.com/v4/letter/i/3ec8ea/32.png) [@Ihms](https://discuss.elastic.co/u/Ihms)\
**Post date:** [October 6, 2022, 3:05pm UTC](https://discuss.elastic.co/t/filter-unstructured-logs-with-filebeat-before-sending-to-logstash/315960/3 "2022-10-06T15:05:40Z")

</div>

These are samples of the logs below. I want to only send logs that have INFO or in a different case, ERROR as the log Level. I don't want logs like the last two lines. The first 3 lines of logs have different patterns, so I am not sure using `dissect` would work.

```auto
2022-10-05 17:49:24,795] {processor.py:651} INFO - DAG(s) dict_keys(['log_filtered'])

[2022-10-05 17:48:22,005: INFO/ForkPoolWorker-1] Filling up the DagBag from /opt/airflow/dags/population.py

[2022-10-05 17:48:21,662] {population.py:24} INFO - I was executed 

/opt/airflow/dags/population.py 

Stale pidfile exists - Removing it.

```

filebeat.yml

```auto

filebeat.inputs:

- type: filestream

  id: my-filestream-id

  enabled: true

  paths:

    - /home/ubuntu/logs/scheduler/**/*.log

filebeat.config.modules:

  path: /etc/filebeat/modules.d/*.yml

  reload.enabled: false

setup.template.settings:

  index.number_of_shards: 1

output.logstash:

  hosts: ["${logstash_ip}:5044"]

processors:

  - add_host_metadata:

      when.not.contains.tags: forwarded

  - add_cloud_metadata: ~

  - add_docker_metadata: ~

  - add_kubernetes_metadata: ~

  - drop_fields:

      fields: ["agent", "cloud", "ecs", "host", "input", "tags"]

      ignore_missing: true

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 3, 2022, 5:06pm UTC](https://discuss.elastic.co/t/filter-unstructured-logs-with-filebeat-before-sending-to-logstash/315960/4 "2022-11-03T17:06:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.