Filebeat not sending specific Log Files

I have configured filebeat 6.6 on a Windows instance. Weird thing is, it is sending logs for IIS but not for file I have specified even though the filebeat can detect it.

Filebeat.yml -->

- type: log
  enabled: true
 - C:\ELK-Logger\filebeat-6.6.1-windows-x86_64\LowError.txt
- type: log
  enabled: true
 - C:\inetpub\logs\LogFiles\*\*
 - C:\Hosting\stagingb2c\PaymentGatewayLogs\*\*
  recursive_glob: enabled 
- type: log
  enabled: true
 - C:\Hosting\stagingb2c\ErrorLogs\*

  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

  index.number_of_shards: 3

  hosts: [""]

  - add_host_metadata: ~
  - add_cloud_metadata: ~

  to_files: true
path: C:\ELK-Logger\filebeat-6.6.1-windows-x86_64\filebeat-6.6.1-windows-x86_64\LOG
  level: info

I can see logs from C:\inetpub\logs\LogFiles folder but not from C:\Hosting\stagingb2c\PaymentGatewayLogs.

I can not see any errors or warnings in filebeat.log when I started it with :slight_smile:

PS C:\ELK-Logger\filebeat-6.6.1-windows-x86_64\filebeat-6.6.1-windows-x86_64> .\filebeat.exe -e -d "*"

    |2019-03-04T21:15:51.602+0300|INFO|log/harvester.go:255|Harvester started for file: C:\Hosting\stagingb2c\PaymentGatewayLogs\CredimaxPaymentGateway_OrderId_12f1050220190810\CredimaxPayment_TransactionDetails_OrderId_12f1050220190810|
|2019-03-04T21:15:51.761+0300|INFO|log/harvester.go:255|Harvester started for file: C:\Hosting\stagingb2c\PaymentGatewayLogs\CredimaxPaymentGateway_OrderId_Sw2m\CredimaxPayment_PROCESS_ACS_RESULT_Response_20190213124610_OrderId_Sw2m.txt|
|2019-03-04T21:15:51.920+0300|INFO|log/harvester.go:255|Harvester started for file: C:\Hosting\stagingb2c\PaymentGatewayLogs\CredimaxPaymentGateway_OrderId__SoLx\CredimaxPayment_PAY_Request_20190205085701_OrderId__SoLx.txt|

I am not able to see these logs in logstash though I can surely see other files coming in Logstash.

Is there any error on the Logstash and could you share that configuration?

My logstash.yml file has a basic configuration

root@ip-172-31-27-22:/etc/logstash# cat logstash.yml | grep -v "#" /var/lib/logstash
path.logs: /var/log/logstash

Sorry I was not clear enough in my request, I was talking about y our Logstash pipeline configuration.

I don't have much configured there either.

root@ip-172-31-27-22:/etc/logstash# cat pipelines.yml | grep -v  "#"

- main
  path.config: "/etc/logstash/conf.d/*.conf"


root@ip-172-31-27-22:/etc/logstash# cat conf.d/02-beats-input.conf 

input {

beats {

client_inactivity_timeout => 1200

port => 5044



root@ip-172-31-27-22:/etc/logstash# cat conf.d/30-elasticsearch-output.conf 

output {

elasticsearch {

hosts => ["localhost:9200"]

manage_template => false

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"



This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.