# Problem about using modules parsing log files while having JSON files

**URL:** <https://discuss.elastic.co/t/problem-about-using-modules-parsing-log-files-while-having-json-files/249050>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 18, 2020, 4:09am UTC](https://discuss.elastic.co/t/problem-about-using-modules-parsing-log-files-while-having-json-files/249050 "2020-09-18T04:09:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sougou](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sougou](https://discuss.elastic.co/u/sougou)\
**Post date:** [September 18, 2020, 4:09am UTC](https://discuss.elastic.co/t/problem-about-using-modules-parsing-log-files-while-having-json-files/249050/1 "2020-09-18T04:09:56Z")

</div>

Hi,

My architecture is as follows:  
Filebeat -\> Logstash -\> Elasticsearch

I passed a number of files to filebeat. Some of which uses some modules. These modules have been given unique var.paths in their module configuration files.

e.g. cisco.yml

> ```
> - module: cisco
> asa:
> enabled: true
> var.paths: ["/path/to/log/*.log"]
> 
> ```

However, I do also have some files that do not require the use of modules (i.e. JSON files). I have set the paths to these files in the filebeat.yml file.

e.g. filebeat.yml

> ```
> filebeat.inputs:
> - type: log
> paths:
> - /path/to/log/*.json
> enabled: true
> json.keys_under_root: true
> fields:
> log_type: json-log
> 
> ```

So my problem is, when i use the module, filebeat does not seem to have parsed the JSON files. Hence, I am wondering if i should have a separate module to manage my JSON files because i think by defining unique paths separately in the modules, the paths that are set in filebeat.input are ignore.

Also my logstash config are as follows:  
e.g. in-out.conf

> ```
> input {
> beats {
> port => 5044
> }
> }
> output {
> if [@metadata][pipeline] {
> elasticsearch {
> hosts => ["localhost:9200"]
> manage_template => false
> index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
> pipeline => "%{[@metadata][pipeline]}"
> }
> } else {
> elasticsearch {
> hosts => ["localhost:9200"]
> manage_template => false
> index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
> }
> }
> }
> 
> ```

Please correct me if I misunderstood the use of filebeat and the use of its modules.

Any help would be very much appreciated.

Regards.

---

<div class="post-metadata">

**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\
**Post date:** [September 18, 2020, 9:48am UTC](https://discuss.elastic.co/t/problem-about-using-modules-parsing-log-files-while-having-json-files/249050/2 "2020-09-18T09:48:36Z")

</div>

Please review the following thread: [Parse / ship JSON file with filebeat](https://discuss.elastic.co/t/parse-ship-json-file-with-filebeat/34540) and see if it's helpful.

---

<div class="post-metadata">

**Author:** ![sougou](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sougou](https://discuss.elastic.co/u/sougou)\
**Post date:** [September 18, 2020, 11:04am UTC](https://discuss.elastic.co/t/problem-about-using-modules-parsing-log-files-while-having-json-files/249050/3 "2020-09-18T11:04:17Z")

</div>

Hi,

Thanks for getting back to me.

The above reference, however, did not solve my problem.

I do not think filebeat is reading the json files at all. I can only find data from the modules I have enabled.

Regards.

---

<div class="post-metadata">

**Author:** ![sougou](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sougou](https://discuss.elastic.co/u/sougou)\
**Post date:** [September 24, 2020, 10:28am UTC](https://discuss.elastic.co/t/problem-about-using-modules-parsing-log-files-while-having-json-files/249050/4 "2020-09-24T10:28:05Z")

</div>

Hi,

This issue is somehow solved when I have re-did everything from the installation to the configurations.

The main difference was that I deleted the filebeat registry directory before restarting the process.

Also, refer to: https://discuss.elastic.co/t/periodically-updated-log-file/117215/7

Regards.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 22, 2020, 12:28pm UTC](https://discuss.elastic.co/t/problem-about-using-modules-parsing-log-files-while-having-json-files/249050/5 "2020-10-22T12:28:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.