Problem about using modules parsing log files while having JSON files

Hi,

My architecture is as follows:
Filebeat -> Logstash -> Elasticsearch

I passed a number of files to filebeat. Some of which uses some modules. These modules have been given unique var.paths in their module configuration files.

e.g. cisco.yml

- module: cisco
  asa:
    enabled: true
    var.paths: ["/path/to/log/*.log"]

However, I do also have some files that do not require the use of modules (i.e. JSON files). I have set the paths to these files in the filebeat.yml file.

e.g. filebeat.yml

filebeat.inputs:
- type: log
  paths:
    - /path/to/log/*.json
  enabled: true
  json.keys_under_root: true
  fields:
    log_type: json-log

So my problem is, when i use the module, filebeat does not seem to have parsed the JSON files. Hence, I am wondering if i should have a separate module to manage my JSON files because i think by defining unique paths separately in the modules, the paths that are set in filebeat.input are ignore.

Also my logstash config are as follows:
e.g. in-out.conf

input {
  beats {
    port => 5044
  }
}
output {
  if [@metadata][pipeline] {
      elasticsearch {
      hosts => ["localhost:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      pipeline => "%{[@metadata][pipeline]}"
    }
  } else {
    elasticsearch {
      hosts => ["localhost:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  }
}

Please correct me if I misunderstood the use of filebeat and the use of its modules.

Any help would be very much appreciated.

Regards.

Please review the following thread: Parse / ship JSON file with filebeat and see if it's helpful.

Hi,

Thanks for getting back to me.

The above reference, however, did not solve my problem.

I do not think filebeat is reading the json files at all. I can only find data from the modules I have enabled.

Regards.

Hi,

This issue is somehow solved when I have re-did everything from the installation to the configurations.

The main difference was that I deleted the filebeat registry directory before restarting the process.

Also, refer to: https://discuss.elastic.co/t/periodically-updated-log-file/117215/7

Regards.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.