Multiple .gz files in multiple sub directories

Elastic Stack Kibana
1

I am sure this has been asked before but nothing I have found during my research has solved my problem. I am using Windows Server 2019 Standard. I have installed Kibana and Elasticsearch via zip files.

I am trying to visualize log data (unknown format) held in multiple sub directories by year, month with each log file in gzipped format for each day of the month separately. (i.e. C:\logs\subdirectory\1\2021\01\2021-01-01.log.gz).

I am able to connect to the Kibana URL and select the "Add data" link and then the "kibana logs" link. I follow the steps to edit the filebeat.yml file to accept inputs as follows:
filebeat.inputs:
- type: log
enabled: true
paths:
- C:\Log\Files\To\Ingest\logs**.gz

I edit the "modules.d\kibana.yml file to accept the same path noted in the filebeat.yml file above.

I run ./filebeat setup and get no errors upon completion. I restart the filebeat and elasticsearch services. When I check the module "Check data" link there is no data being received.

I have been unsuccessful at every configuration I have tried and have created this topic as a last resort. If any one has any advice or guidance it will be greatly appreciated.

2

Hi @jmt605 Welcome to the community.

I think you should perhaps take a look at the file / path glob syntax here.

Example
C:\logs\subdirectory\1\2021\01\2021-01-01.log.gz

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\logs\subdirectory\**\*.log.gz

Are you trying to ship Kibana Logs? otherwise you do not need to do that? If you are just trying to ship some custom application logs just use the filebeat:inputs above

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.