Beats / Filebeat / Topbeat / packetbeat / SSL connections

Elastic Stack Beats
1

Hi,

I installed all Filebeat - Topbeat and Packetbeat on 5 servers and they all connect back to my ELK server delivering the files to logs to logstash.

Some how the connections keep droping.

For example:

Srv1 will only keep filebeat connection alive and drop Topbeat and Packetbeat...

If I restart Topbeat it will connect and drop filebeat.

Is there any solution that will allow using all of them simultaneous?

Regards,

Leo

2

which versions are you using?

Anything in logstash logs about congested pipeline?

3

Hi Steffens,

The logstash shows this error:

message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::OpenBreaker, :level=>:warn}

But I end up changing the settings of the servers to ouput the content of topbeat and packetbeat in to a file and use only filebeat to forward the logs in to the ELK server.

This seems to have sorted the initial problem, I still see the same error on logstash.log but the data now arrive from all 5 servers.

The issue I need to resolve now is a way to index the data properly.

Everything get index under filebeat index now. I need it to get index in the way it happened before.

Any suggestions? if I use the topbeat to deliver the data direct to logstash it get separated automatically, but now it arrives from filebeat and everything gets indexed under the same index...

root@ELKSERVER /etc/logstash/conf.d # cat 01-filebeat-input.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/ssl/logstash-forwarder.crt"
ssl_key => "/etc/ssl/logstash-forwarder.key"
}
}

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
root@ELKSERVER /etc/logstash/conf.d # cat 02-output.conf
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

4

##############################################################################
##############################################################################
##############################################################################

topbeat.yml
...

File as output

file:
# Path to the directory where to save the generated files. The option is mandatory.
path: "/var/log/beats"

# Name of the generated files. The default is `topbeat` and it generates files: `topbeat`, `topbeat.1`, `topbeat.2`, etc.
filename: topbeat.log

...

##############################################################################
##############################################################################
##############################################################################

packetbeat.yml

...

File as output

file:
# Path to the directory where to save the generated files. The option is mandatory.
path: "/var/log/beats/"

# Name of the generated files. The default is `packetbeat` and it generates files: `packetbeat`, `packetbeat.1`, `packetbeat.2`, etc.
filename: packetbeat.log

##############################################################################
##############################################################################
##############################################################################
filebeat.yml

############################# Filebeat ######################################
filebeat:

List of prospectors to fetch data.

prospectors:
# Each - is a prospector. Below are the prospector specific configurations
-
# Paths that should be crawled and fetched. Glob based paths.
# To fetch all ".log" files from a specific level of subdirectories
# /var/log//.log can be used.
# For each file found under this path, a harvester is started.
# Make sure not file is defined twice as this can lead to unexpected behaviour.
paths:
- /var/log/varnish/varnishncsa.log
- /var/log/syslog
- /var/log/beats/*.log

5

I forgot to post the version...

Debian 8.5

dpkg -l elasticsearch logstash filebeat packetbeat topbeat

ii elasticsearch 2.4.0 all Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.el
ii filebeat 1.3.0 amd64 Sends log files to Logstash or directly to Elasticsearch.
ii logstash 1:2.3.4-1 all An extensible logging pipeline
ii packetbeat 1.3.1 amd64 A network analyzer tool that sends data to Elasticsearch.
ii topbeat 1.3.1 amd64 Like the top command but outputing periodically Logstash, Elasticsearch or file

6

logstash uses a circuit breaker with default timeout of 5 seconds. Once pipeline in logstash get's slowed down by too many events, the circuit breaker might trigger closing connections + not allowing any new connections for some time.

One can increase to disable the circuit breaker by setting congestion_threshold to multiple days for example:

input {
    beats {
        ...
        congestion_threshold => 604800 # = 60 * 60 * 24 * 7 (7 days)
    }
}

Having this option you might still be able to process all events.

Sending via filebeat adds a json in json encoding by default. filebeat 5.0 can parse json logs. Alternatively In logstash apply the json filter to decode the original message and see if you can extract the @timestamp. On trick is to use fields setting in filebeat prospectors to add additional meta-data to events. the meta-data can be used in logstash for additional filtering.

7

This topic was automatically closed after 21 days. New replies are no longer allowed.