# Parsing multiple files with Filebeat

**URL:** <https://discuss.elastic.co/t/parsing-multiple-files-with-filebeat/271591>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 29, 2021, 6:10am UTC](https://discuss.elastic.co/t/parsing-multiple-files-with-filebeat/271591 "2021-04-29T06:10:20Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![tkkchan](https://avatars.discourse-cdn.com/v4/letter/t/8491ac/32.png) [@tkkchan](https://discuss.elastic.co/u/tkkchan)\
**Post date:** [April 29, 2021, 6:10am UTC](https://discuss.elastic.co/t/parsing-multiple-files-with-filebeat/271591/1 "2021-04-29T06:10:20Z")

</div>

Dear all,  
I have several JSON files that I wish to parse with Filebeat; and I read the following:  
[https://stackoverflow.com/questions/39983918/can-filebeat-use-multiple-config-files](https://stackoverflow.com/questions/39983918/can-filebeat-use-multiple-config-files)  
I used the code in the above solution for configuring my filebeat.yml , it looks like the following

```auto
filebeat.inputs:
- type: log
  enabled: true
  path: inputs.d/*.yml

output.elasticsearch:
  hosts: ["localhost:9200"]

```

and created a directory `inputs.d` at `/etc/filebeat/` to put in all the individual yml files for parsing each JSON file.  
However, when I run filebeat, nothing happened -- there's no output or whatsoever in ES.

I tried to bypass this issue and look for alternative solutions, just like in this page[https://blog.csdn.net/shgh\_2004/article/details/98650114](https://blog.csdn.net/shgh_2004/article/details/98650114)  
(You don't need to understand chinese to read the yml.)  
in this case, the author combined all of the yml files in the `/etc/filebeat/filebeat.yml`. However, I was concerned about how to put in the processors, as I have some fields that I wish to drop for each of the JSON files.

Each of the yml files looked like this

```auto

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - ...
 
processors:
 - decode_json_fields:
     fields: ['message']
     target: ''
     overwrite_keys: true

 - drop_fields:
     fields: [....]

filebeat.shutdown_timeout: 5s
setup.template.enabled: false
setup.ilm.enabled: false

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "..."
 

```

Thank you in advance for your attention 🙂

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 29, 2021, 6:22am UTC](https://discuss.elastic.co/t/parsing-multiple-files-with-filebeat/271591/2 "2021-04-29T06:22:08Z")

</div>

> [@tkkchan](#):
>
> and created a directory `inputs.d` at `/etc/filebeat/` to put in all the individual yml files for parsing each JSON file.

I think you're misunderstanding how that works.

[`path`](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filestream-input-paths) is;

> A list of glob-based paths that will be crawled and fetched. All patterns supported by [Go Glob](https://golang.org/pkg/path/filepath/#Glob) are also supported here.

It's literally the path of the files that you want Filebeat to process, not a list of configs to read to then process files elsewhere.

> [@tkkchan](#):
>
> in this case, the author combined all of the yml files in the `/etc/filebeat/filebeat.yml` . However, I was concerned about how to put in the processors, as I have some fields that I wish to drop for each of the JSON files.

Just to be clear, you want to have specific processing rules for specific sets of json files?

---

<div class="post-metadata">

**Author:** ![tkkchan](https://avatars.discourse-cdn.com/v4/letter/t/8491ac/32.png) [@tkkchan](https://discuss.elastic.co/u/tkkchan)\
**Post date:** [April 29, 2021, 6:26am UTC](https://discuss.elastic.co/t/parsing-multiple-files-with-filebeat/271591/3 "2021-04-29T06:26:04Z")

</div>

Dear Mark,  
Thank you for your clarification. is there a ways I can use several yml files in one filebeat run? Or, do I have to run multiple filebeat instances?

> [@warkolm](#):
>
> Just to be clear, you want to have specific processing rules for specific sets of json files?

Yes, that's what I wished to do. The JSON files have different structures, and in each of them, there're different fields I want to drop. (Though, I am not sure if I just write all of the fields I want to drop from all files together in one statement, is that going to work?...)  
Thanks again 🙂

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 29, 2021, 6:27am UTC](https://discuss.elastic.co/t/parsing-multiple-files-with-filebeat/271591/4 "2021-04-29T06:27:46Z")

</div>

Try [Load external configuration files | Filebeat Reference [7.12] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-reloading.html)

---

<div class="post-metadata">

**Author:** ![tkkchan](https://avatars.discourse-cdn.com/v4/letter/t/8491ac/32.png) [@tkkchan](https://discuss.elastic.co/u/tkkchan)\
**Post date:** [April 29, 2021, 6:36am UTC](https://discuss.elastic.co/t/parsing-multiple-files-with-filebeat/271591/5 "2021-04-29T06:36:48Z")

</div>

Dear Mark,  
I think you mean this?

```auto
filebeat.config.inputs:
  enabled: true
  path: inputs.d/*.yml

output.elasticsearch:
  hosts: ["localhost:9200"]

```

I tried it but still nothing happened. I thought it was because I wrote `filebeat.inputs:` instead of `filebeat.config.inputs:`, but it doesn't make a difference after I changed it. (now the `path` should be correct?)

I used the following to get filebeat running

```auto
#!/usr/bin/env bash

# Script to run Filebeat in foreground with the same path settings that
# the init script / systemd unit file would do.

rm -rf /var/lib/filebeat/registry/
exec /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -d "publish" \
  --path.home /usr/share/filebeat \
  --path.data /var/lib/filebeat \
  --path.logs /var/log/filebeat \
  "$@"

```

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [April 29, 2021, 5:13pm UTC](https://discuss.elastic.co/t/parsing-multiple-files-with-filebeat/271591/6 "2021-04-29T17:13:23Z")

</div>

What does your external config file look like?

---

<div class="post-metadata">

**Author:** ![tkkchan](https://avatars.discourse-cdn.com/v4/letter/t/8491ac/32.png) [@tkkchan](https://discuss.elastic.co/u/tkkchan)\
**Post date:** [April 30, 2021, 12:03am UTC](https://discuss.elastic.co/t/parsing-multiple-files-with-filebeat/271591/7 "2021-04-30T00:03:10Z")

</div>

Dear Alex,  
I listed the external yml at the bottom of original post.

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [April 30, 2021, 12:07am UTC](https://discuss.elastic.co/t/parsing-multiple-files-with-filebeat/271591/8 "2021-04-30T00:07:00Z")

</div>

Then your external config files are invalid. They can only have input configs. See the example here of what should be there, [Load external configuration files | Filebeat Reference [7.12] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-reloading.html#load-input-config). the output config and other settings must be in the main config file.

---

<div class="post-metadata">

**Author:** ![tkkchan](https://avatars.discourse-cdn.com/v4/letter/t/8491ac/32.png) [@tkkchan](https://discuss.elastic.co/u/tkkchan)\
**Post date:** [April 30, 2021, 1:20am UTC](https://discuss.elastic.co/t/parsing-multiple-files-with-filebeat/271591/9 "2021-04-30T01:20:29Z")

</div>

Dear Alex,  
I see... Thank you very much indeed.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 28, 2021, 3:21am UTC](https://discuss.elastic.co/t/parsing-multiple-files-with-filebeat/271591/10 "2021-05-28T03:21:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.