Syslog and filebeat assistance request

Elastic Stack Beats
1

We have been running ELK v. 6.2.x
We have a central syslog server on EL5 pushing log files with filebeat to logstash
I have been tasked with setting up a new central syslog server on EL7 (replace EL5 instance).
Filebeat-6.2.2 is the current version on the EL5 host, matched filebeat version on EL7 host.

when testing the output its all ok.
parse host... OK
dns lookup... OK
addresses.... 10.10.3.19
dial up... OK
TLS... WARN
talk to server... OK

What I am not getting are the latest syslog details displaying in Kibana.

Please tell me how I can verify the syslog data is getting to Logstash? Elasticsearch?

Any advice would be much appreciated. TYIA!

Please let me know if there are details you want to know about the environment.
Sorry, I am an elk-noob.
Brian

2

My /etc/filebeat/filebeat.yml . -- without comments
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/messages
- /var/log/asa
- /var/log/asa-nat
- /var/log/audit
- /var/log/boot.log
- /var/log/cron
- /var/log/bsp-nas
- /var/log/cups
- /var/log/eduroam
- /var/log/faillog
- /var/log/lastlog
- /var/log/mail
- /var/log/maillog
- /var/log/mcelog
- /var/log/sa
- /var/log/samba
- /var/log/spooler
- /var/log/tallylog
- /var/log/up2date
- /var/log/yum.log
- /var/log/dhcpd-wifi-radius
- /var/log/secure
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.template.overwrite: true
setup.kibana:
output.logstash:
hosts: ["logstash-host.mydomain.com:5044"]

3

Getting this in the filebeat log on host 10.1.38.21
2019-08-26T15:22:49.446-0500 ERROR logstash/async.go:256 Failed to publish events caused by: write tcp 10.1.38.21:53346->10.1.38.19:5044: write: connection reset by peer
2019-08-26T15:22:51.007-0500 ERROR pipeline/output.go:121 Failed to publish events: write tcp 10.1.38.21:53346->10.1.38.19:5044: write: connection reset by peer

10.1.38.21 is my filebeat server.
10.1.38.19 is my logstash server.

4

Is there any person on this forum that might provide some insight?

5

What does your logstash conf look like?

And are you sure logstash is running and listening?

Are you sure there is connectivity between the filebeat and logstash host?

You should be able to test it by getting on the filebeat host and running

telnet 10.1.38.19 5044

It should show connection established if it can reach logstash on port 5044

6

Thank you for your time!
Yes I get a connection established when I telnet.
when I run the filebeat test output it shows OK for connection.

There is connection established in the logs, but when it tries to publish events the connection is reset by peer.
2019-08-26T16:37:10.167-0500 INFO pipeline/output.go:105 Connection to backoff(async(tcp://slp-b3c.its.bethel.edu:5044)) established

Logstash conf looks like this
logstash.yml
---
path.data: "/var/lib/logstash"
path.logs: "/var/log/logstash"
pipeline.batch.size: 125
pipeline.batch.delay: 50
config.reload.automatic: true
config.reload.interval: 3s
queue.type: memory

7

Today I see a slightly different fail message
2019-08-27T09:53:25.316-0500 ERROR logstash/async.go:256 Failed to publish events caused by: read tcp 10.1.38.21:47796->10.1.38.19:3514: i/o timeout

8

Hey Brian,

Can you share your logstash beats input configuration? Anything in the logs for logstash?

The errors indicate an issue between filebeat and logstash so it could be an issue with the connectivity from filebeat to logstash (your logs show successful connection), the output plugin from filebeat (looks fine to me and logs show connection established), or the input plugin from logstash.

The successful connection from telnet shows that logstash is up and listening on that port, but not sure if you have more advanced configuration on the input plugin.

9

here is my syslog.yml contents.
input {
udp {
type => 'syslog'
port => 3514
id => 'data1_udp'
}

tcp {
type => 'syslog'
port => 3514
id => 'data1_tcp'
}
}

filter {
}

output {
#stdout { codec => rubydebug }
kafka {
codec => json
topic_id => syslog
bootstrap_servers => 'slp-3zp.its.bethel.edu:9092,slp-b3c.its.bethel.edu:9092'
}
}

Sorry about the formatting..

10

Log on logstash not showing any matching times for the ERROR getting on filebeat.

11

Thanks for the config info Brian, I understand your setup better now.

Did you say this configuration was working previously? I'm not an elastic team member, just a dude who setup ELK at my workplace, so apologies if I'm wrong but as far as I understand it, you want your output to match up with your input plugin from logstash.

In your filebeat configuration, you are sending log events to logstash on port 5044 which is the default port for the filebeat input plugin (for logstash):
https://www.elastic.co/guide/en/logstash/6.2/plugins-inputs-beats.html

You should have a beats input configured on logstash just like the link above if you want to receive events from filebeat on that port. More info in this link:
https://www.elastic.co/guide/en/beats/filebeat/6.2/logstash-output.html

If you follow the link above, setup a beats input on your logstash and keep your filebeat configuration the same, you should definitely be able to see events coming into logstash.

The current inputs you are using are UDP and TCP input plugins and would not affect your filebeat log events at all.
Could be part of an old setup? If you get syslog configured on your source log boxes, and configure it to send to your logstash server on that specific port, 3514, you should then see your syslog logs but I have no experience setting up syslog so I can't confirm this.

12

Thanks for your help justkind! We do have an existing syslog server sending with filebeats - I noticed the ports did not match up, so I will dig deeper on that. Certainly stood out as not correct.

13

Not sure if this helps, here is the startup log from logstash.
[2019-08-27T11:24:14,971][INFO ][org.apache.kafka.clients.producer.ProducerConfig] ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [slp-3zp.mydomain.com:9092, slp-b3c.mydomain.com:9092]
buffer.memory = 33554432
client.id =
compression.type = none
connections.max.idle.ms = 540000
enable.idempotence = false
interceptor.classes = null
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metric.reporters =
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 10
reconnect.backoff.ms = 10
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class org.apache.kafka.common.serialization.StringSerializer

[2019-08-27T11:24:15,127][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 1.0.0
[2019-08-27T11:24:15,127][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : aaa7af6d4a11b29d
[2019-08-27T11:24:15,178][INFO ][logstash.inputs.tcp ] Starting tcp input listener {:address=>"0.0.0.0:3514", :ssl_enable=>"false"}
[2019-08-27T11:24:15,856][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"syslog", :thread=>"#<Thread:0x7f016e99@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 sleep>"}
[2019-08-27T11:24:15,866][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:3514"}
[2019-08-27T11:24:15,914][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"0.0.0.0:3514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2019-08-27T11:24:16,663][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"winlogbeats", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-08-27T11:24:16,708][INFO ][org.apache.kafka.clients.producer.ProducerConfig] ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [slp-3zp.mydomain.com:9092, slp-b3c.mydomain.com:9092]
buffer.memory = 33554432
client.id =
compression.type = none
connections.max.idle.ms = 540000
enable.idempotence = false
interceptor.classes = null
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metric.reporters =
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 10
reconnect.backoff.ms = 10
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class org.apache.kafka.common.serialization.StringSerializer

[2019-08-27T11:24:16,726][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 1.0.0
[2019-08-27T11:24:16,727][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : aaa7af6d4a11b29d
[2019-08-27T11:24:16,729][INFO ][org.apache.kafka.clients.producer.ProducerConfig] ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [slp-3zp.mydomain.com:9092, slp-b3c.mydomain.com:9092]
buffer.memory = 33554432
client.id =
compression.type = none
connections.max.idle.ms = 540000
enable.idempotence = false
interceptor.classes = null
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metric.reporters =
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 10
reconnect.backoff.ms = 10
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class org.apache.kafka.common.serialization.StringSerializer

14

[2019-08-27T11:24:16,741][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 1.0.0
[2019-08-27T11:24:16,741][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : aaa7af6d4a11b29d
[2019-08-27T11:24:16,743][INFO ][org.apache.kafka.clients.producer.ProducerConfig] ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [slp-3zp.mydomain.com:9092, slp-b3c.mydomain.com:9092]
buffer.memory = 33554432
client.id =
compression.type = none
connections.max.idle.ms = 540000
enable.idempotence = false
interceptor.classes = null
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metric.reporters =
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 10
reconnect.backoff.ms = 10
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class org.apache.kafka.common.serialization.StringSerializer

[2019-08-27T11:24:16,762][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 1.0.0
[2019-08-27T11:24:16,762][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : aaa7af6d4a11b29d
[2019-08-27T11:24:17,152][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-08-27T11:24:17,163][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"winlogbeats", :thread=>"#<Thread:0x7d2dd061@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 sleep>"}
[2019-08-27T11:24:17,169][INFO ][logstash.agent ] Pipelines running {:count=>2, :pipelines=>["syslog", "winlogbeats"]}
[2019-08-27T11:24:17,170][INFO ][org.logstash.beats.Server] Starting server on port: 5044

15

Happy to help Brian, hopefully we can figure out what's going on.

We just want to line up logstash's input with whatever is being sent out from your log servers, whether it is filebeat log events (beats input), or generic events (your TCP and UDP inputs).

If you're interested, you can check out the logstash syslog input plugin which might be useful as these plugins are made to specifically handle syslog events:
https://www.elastic.co/guide/en/logstash/6.2/plugins-inputs-syslog.html

The logs for logstash you posted seem to show that it is properly setting up your TCP and UDP listeners on port 3514 and also has a beats input configured on port 5044.

If you can, double check your syslogs is sending to the correct logstash on the correct port.
For the filebeat log events, it seems there is an existing beats input configured somewhere so maybe try to look for that and see if there are any errors in the configuration.

16

my beats version changed from 6.2.2 to 6.8.2
after putting the version back, I am no longer seeing any ERROR in the log for filebeat.
Still not seeing anything show in Kibana....
I've restarted logstash
I've refreshed the index on Elastic heads with curl -XPOST 'http://localhost:9200/syslog-*/_refresh'

On my old syslog server I am using the same filebeat.yml contents for the new syslog server.

Syslog data is its own drive mounted under /var/logs

I use rsync on that partition to send data to the new syslog server.

Syslog is a different version... maybe that is my problem?

17

Hmm interesting that there were errors for the newer filebeats version, according to the compatibility matrix, all filebeat and logstash versions between 5.6.x-6.8.x should be fine.

With no errors it's going to be hard to find the issue but we need to trace the flow of traffic.

Could you clarify on the full flow? From the configuration you've shared, it appears to be:
Filebeat -> Logstash -> Kafka -> ? -> Elasticsearch -> Kibana

Do you happen to have another Logstash between Kafka and Elasticsearch?

Can you check anything on Kafka? Are indices populating in Elasticsearch? You can check on this by curl'ing your Elasticsearch with /_cat/indices?v?

We're just trying to figure out where traffic is stopping. If you have monitoring enabled, you should be able to see if logstash is processing events.

18

I might switch back to the newer 6.8.2 because it did provide more detail with error's...
[(1)Syslog -> Filebeat] -> [(2)Logstash -> Kafka -> Zookeeper] -> [(2)Elasticsearch] -> [(1)Kibana] (According to our documentation)
1 server for syslog & filebeat
2 servers for logstash, kafka and zookeeper - HA PROXY
2 servers for Elasticsearch - HA PROXY
1 server for Kibana

Please be patient with me as I am still a noob when it comes to Elastic and friends.
I can check logs in kafka, never looked at kafka before :expressionless:
when I curl Elasticsearch I do get results - after removing the v?
curl -XGET 'http://slp-pkf.my.com:9200/_cat/indices?' | grep syslog-2019.08.28

19

Thanks for the information Brian, and no worries at all about experience level with tools. There's always someone more experienced/knowledgeable and also always someone with less knowledge/experience than you on any topic :slight_smile:, I wouldn't call myself an expert either so would be great if someone on the @elastic team could chime in if they are reading this.

The ?v URL parameter just adds column names so it's not important, but you stated that you are getting results if you hit the indices endpoint and grep for the syslog index with today's date.

That should mean that you are indeed getting syslog data throughout your whole pipeline, if it's making its way throughout your entire flow to Elasticsearch.

If you are not seeing it in Kibana, can you ensure that you have the correct index pattern? It has to match the index names you are seeing from that /_cat/indices endpoint.
If you already have an existing index pattern and it is not showing up any data, maybe you need to refresh the field list.
You can do this by going to Management -> Index Patterns -> Click on your syslog index pattern -> Click on the refresh button (looks like 2 circling arrows) on the top right and check again.
Maybe delete the index pattern and create it again as a last resort.

You can also confirm your indices have data in it by curl'ing Elasticsearch in a similar way to getting the indices. Assuming your syslog index is named exactly what you put in your grep statement:

curl -XGET 'http://slp-pkf.my.com:9200/syslog-2019.08.28/_search

And you should get results back, maybe change the date to today's date 2019.08.29 to ensure it is being updated constantly.

20

Thanks again for your help @justkind Yes I am getting data from the expiring central syslog server...