Filebeat not sending specific Log Files

Elastic Stack Beats
1

I have configured filebeat 6.6 on a Windows instance. Weird thing is, it is sending logs for IIS but not for file I have specified even though the filebeat can detect it.

Filebeat.yml -->

filebeat.inputs:
- type: log
  enabled: true
  paths:
 - C:\ELK-Logger\filebeat-6.6.1-windows-x86_64\LowError.txt
- type: log
  enabled: true
  paths:
 - C:\inetpub\logs\LogFiles\*\*
 - C:\Hosting\stagingb2c\PaymentGatewayLogs\*\*
  recursive_glob: enabled 
- type: log
  enabled: true
  paths:
 - C:\Hosting\stagingb2c\ErrorLogs\*
  

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.logstash:
  hosts: ["13.234.83.186:5044"]


processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

logging:
  to_files: true
  files:
path: C:\ELK-Logger\filebeat-6.6.1-windows-x86_64\filebeat-6.6.1-windows-x86_64\LOG
  level: info

I can see logs from C:\inetpub\logs\LogFiles folder but not from C:\Hosting\stagingb2c\PaymentGatewayLogs.

I can not see any errors or warnings in filebeat.log when I started it with :slight_smile:

PS C:\ELK-Logger\filebeat-6.6.1-windows-x86_64\filebeat-6.6.1-windows-x86_64> .\filebeat.exe -e -d "*"

    |2019-03-04T21:15:51.602+0300|INFO|log/harvester.go:255|Harvester started for file: C:\Hosting\stagingb2c\PaymentGatewayLogs\CredimaxPaymentGateway_OrderId_12f1050220190810\CredimaxPayment_TransactionDetails_OrderId_12f1050220190810|
|---|---|---|---|
|2019-03-04T21:15:51.761+0300|INFO|log/harvester.go:255|Harvester started for file: C:\Hosting\stagingb2c\PaymentGatewayLogs\CredimaxPaymentGateway_OrderId_Sw2m\CredimaxPayment_PROCESS_ACS_RESULT_Response_20190213124610_OrderId_Sw2m.txt|
|2019-03-04T21:15:51.920+0300|INFO|log/harvester.go:255|Harvester started for file: C:\Hosting\stagingb2c\PaymentGatewayLogs\CredimaxPaymentGateway_OrderId__SoLx\CredimaxPayment_PAY_Request_20190205085701_OrderId__SoLx.txt|

I am not able to see these logs in logstash though I can surely see other files coming in Logstash.

2

Is there any error on the Logstash and could you share that configuration?

3

My logstash.yml file has a basic configuration

root@ip-172-31-27-22:/etc/logstash# cat logstash.yml | grep -v "#"
path.data: /var/lib/logstash
path.logs: /var/log/logstash
4

Sorry I was not clear enough in my request, I was talking about y our Logstash pipeline configuration.

5

I don't have much configured there either.

root@ip-172-31-27-22:/etc/logstash# cat pipelines.yml | grep -v  "#"

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

Also,

root@ip-172-31-27-22:/etc/logstash# cat conf.d/02-beats-input.conf 

input {

beats {

client_inactivity_timeout => 1200

port => 5044

}

}

root@ip-172-31-27-22:/etc/logstash# cat conf.d/30-elasticsearch-output.conf 

output {

elasticsearch {

hosts => ["localhost:9200"]

manage_template => false

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

}

}
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.