Problem with Filebeat sending data to logstash

Elastic Stack Beats
1

After I successfully made it to connect FileBeat with Logstash and Elasticsearch I am now having troubles with it, I tried to adjust my filter in Logstash and even after getting it back to the initial state I am unable to connect them.
I'm running all as windows services. My operation system is VMWare workstation windows 2016 and here are my config files:
filebeat.yaml:
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
- type: log

	enabled: true

	paths:
		- c:\programdata\filebeat\Applogs\AxesPlus\*.txt
	
#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false
  
#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  
#================================ General =====================================

tags: ["AxesPlus"]

#============================== Kibana =====================================

setup.kibana:

  host: "localhost:5601"
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
    hosts: ["localhost:5044"]

logstash config:

input {
  beats {
   port => 5044
   type => "log"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Logstash log:
[2017-11-22T16:11:19,430][INFO ][logstash.pipeline ] Pipeline started {"pipeline.id"=>"main"}
[2017-11-22T16:11:19,444][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2017-11-22T16:11:19,546][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}

filebeat:
2017-11-22T16:32:12+02:00 INFO Home path: [C:\Program Files\Filebeat] Config path: [C:\Program Files\Filebeat] Data path: [C:\Program Files\Filebeat\data] Logs path: [C:\Program Files\Filebeat\logs]
2017-11-22T16:32:12+02:00 INFO Beat UUID: 9b0b57c2-7109-471c-9295-d0bbc9f21ab7
2017-11-22T16:32:12+02:00 INFO Metrics logging every 30s
2017-11-22T16:32:12+02:00 INFO Setup Beat: filebeat; Version: 6.0.0
2017-11-22T16:32:12+02:00 INFO Elasticsearch url: http://localhost:9200
2017-11-22T16:32:12+02:00 INFO Beat name: WIN-E56I4JOUMH9
2017-11-22T16:32:12+02:00 INFO Elasticsearch url: http://localhost:9200
2017-11-22T16:32:12+02:00 INFO Connected to Elasticsearch version 6.0.0
2017-11-22T16:32:12+02:00 INFO Template already exists and will not be overwritten.

2

Could you provide more logs from Filebeat? Is there any concrete error message?

3

I don't have much logs. Now I started using the filebeat as process not as a service. It worked for a while but now it doesn't. I run it with debug log and it sends details of the beat it is sending, yet nothing gets to elastic. It says from time to time that it has connection time out on connection. Although I don't see any error on logstash log file

4

Could you show the output of filebeat -e -d "*"?

5

Thanks for your help. Eventually, it appears all my trouble was that the file it is following wasn't change. Now I am having trouble replacing the default timestamp filed.

6

Hi Tal,

Coming back to your original question. I thought you used logstash as an output for filebeat.
Why then, in the logging output you posted, one sees such lines as

xxxxxxxxx INFO Elasticsearch xxxxxxx
xxx INFO Connected to Elasticsearch xxxxxx

I have not tried Filebeat 6.0 but it seems that maybe in the filebeat.yml the elasticsearch output was left uncommented (as per default)?
When I start my Filebeat 5.6. x (connecting to Logstash) it clearly states its output as Logstash and proceeds.
You said you had run the connection with success, at least once.
What had you changed since? Maybe you put back an original filebeat.yml with the elastic output uncommented.
As I have not followed the evolution of your tests and as I do not use Filebeat 6.0, sorry in case suggestions are not pertinent to your problems.
Cheers

Lara

7

Thank you so much for your help. My problem was that the log file that was attached to filebeat wasn't changed. Thus, it wouldn't upload anything. It took filebeat a while until it logged that line that said that the file wasn't changed.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.