Filebeat is not sending logs to Logstash, logstash is localhost

Elastic Stack Logstash
1

Hi All,

I am having trouble with my filebeat/logstash. connectivity seems to be inconsistenst. When i try it in debug mode it works fine but when i use it with systemctl start it fails . Below are my filebeat and logstash config details and error output details

Filebeat:

filebeat.inputs:

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input specific configurations.

  • type: log

    Change to true to enable this input configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • "/home/clouduser/apache-tomcat-9.0.36/logs/catalina.*.log"
      ignore_older: 24h
      #- c:\programdata\elasticsearch\logs*

============================== Filebeat modules ==============================

filebeat.config.modules:
enabled: true
path: ${path.config}/modules.d/*.yml

Set to true to enable config reloading

reload.enabled: false

Period on which files under path should be checked for changes

#reload.period: 10s

======================= Elasticsearch template setting =======================

setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false

================================== General ===================================

The name of the shipper that publishes the network data. It can be used to group

all the transactions sent by a single shipper in the web interface.

#name:

The tags of the shipper are included in their own field with each

transaction published.

tags: ["10.112.22.74"]

Optional fields that you can specify to add additional information to the

output.

fields:
env: Test_Server

------------------------------ Logstash Output -------------------------------

output.logstash:

The Logstash hosts

hosts: ["127.0.0.1:5044"]
bulk_max_size: 1024

Optional SSL. By default is off.

================================= Processors =================================

processors:

  • add_host_metadata:
    when.not.contains.tags: forwarded
  • add_cloud_metadata: ~
  • add_docker_metadata: ~
  • add_kubernetes_metadata: ~

================================== Logging ===================================

Sets log level. The default log level is info.

Available log levels are: error, warning, info, debug

#logging.level: debug

logging.level: debug
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
rotateeverybytes: 10485760
permissions: 0644

and my Logstash beats.conf file

input {
beats {

client_inactivity_timeout => 1200

port => 5044
ssl => false

ssl_certificate => "/etc/pki/tls/certs/logstash.crt"

ssl_key => "/etc/pki/tls/private/logstash.key"

}
}
output {
file {

path => "/mnt/xxxxxxxxxxxxx/Logs/Dev_Internal/VMSailPointTaskDEV0"

 path => "/mnt/xxxxxxxxxxx/Logs/Dev_Internal/Ganeshtest1TaskDev1/%{[host][name]}-catalina_logs-%{+YYYY-MM-dd}.log"
 codec => line { format => "%{message}" }

}

and logs from filebeat

2020-09-30T23:15:50.798-0400 DEBUG [transport] transport/client.go:205 handle error: write tcp 127.0.0.1:54730->127.0.0.1:5044: write: connection reset by peer
2020-09-30T23:15:50.798-0400 DEBUG [transport] transport/client.go:118 closing
2020-09-30T23:15:50.798-0400 DEBUG [logstash] logstash/async.go:172 73 events out of 73 events sent to logstash host 127.0.0.1:5044. Continue sending
2020-09-30T23:15:50.798-0400 DEBUG [logstash] logstash/async.go:128 close connection
2020-09-30T23:15:50.798-0400 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 127.0.0.1:54730->127.0.0.1:5044: write: connection reset by peer
2020-09-30T23:15:50.798-0400 DEBUG [logstash] logstash/async.go:128 close connection
2020-09-30T23:15:50.798-0400 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2020-09-30T23:15:50.798-0400 INFO [publisher] pipeline/retry.go:223 done
2020-09-30T23:15:50.798-0400 DEBUG [harvester] log/log.go:107 End of file reached: /home/clouduser/apache-tomcat-9.0.36/logs/catalina.2020-09-30.log; Backoff now.
2020-09-30T23:15:52.532-0400 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 127.0.0.1:54730->127.0.0.1:5044: write: connection reset by peer
2020-09-30T23:15:52.532-0400 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(async(tcp://127.0.0.1:5044))
2020-09-30T23:15:52.532-0400 DEBUG [logstash] logstash/async.go:120 connect
2020-09-30T23:15:52.533-0400 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2020-09-30T23:15:52.533-0400 INFO [publisher] pipeline/retry.go:223 done
2020-09-30T23:15:52.798-0400 DEBUG [harvester] log/log.go:107 End of file reached: /home/clouduser/apache-tomcat-9.0.36/logs/catalina.2020-09-30.log; Backoff now.
2020-09-30T23:15:54.767-0400 DEBUG [input] input/input.go:139 Run input
2020-09-30T23:15:54.767-0400 DEBUG [input] log/input.go:205 Start next scan
2020-09-30T23:15:54.767-0400 DEBUG [input] log/input.go:439 Check file for harvesting: /home/clouduser/apache-tomcat-9.0.36/logs/catalina.2020-09-30.log
2020-09-30T23:15:54.767-0400 DEBUG [input] log/input.go:530 Update existing file for harvesting: /home/clouduser/apache-tomcat-9.0.36/logs/catalina.2020-09-30.log, offset: 169191
2020-09-30T23:15:54.767-0400 DEBUG [input] log/input.go:582 Harvester for file is still running: /home/clouduser/apache-tomcat-9.0.36/logs/catalina.2020-09-30.log
2020-09-30T23:15:54.767-0400 DEBUG [input] log/input.go:226 input states cleaned up. Before: 1, After: 1, Pending: 0

Can some one help me , why I am facing network/connectivity issues from logstash to filebeat

2

Welcome to our community! :smiley:

Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

1 Like
3

If filebeat is seeing 'connection reset by peer' then the peer (logstash) probably logs an error. What does logstash log (formatted, please, as Mark said).

4

Hi ,

I have tried a different way

input {
file {
path => "/home/clouduser/apache-tomcat-9.0.36/logs/catalina.*.log"
start_position => "beginning"
sincedb_path => "/dev/null"

ssl_certificate => "/etc/pki/tls/certs/logstash.crt"

ssl_key => "/etc/pki/tls/private/logstash.key"

}
}
output {
file {

path => "/mnt/xxxxxx/Logs/Dev_Internal/VMSailPointTaskDEV0"

 path => "/mnt/xxxxxx/Logs/Dev_Internal/GaneshTestTaskDev1/%{[host][name]}-catalina_logs-%{+YYYY-MM-dd}.log"
 codec => line { format => "%{message}" }

}

}

all the logs are getting captured when i run this manually

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/beats.conf

but when i start the logstash with systemctl start logstash . logs are not getting captured not sure why ?

any thoughts on this ?

5

Interestingly my logstash is running when I tried ps -ef | grep logstash and when my filebeat tries to send the logs it is getting connection refused error.

When I even do telent , i get below error message
[root@GaneshTestTaskDev2TaskDev1 logstash]# telnet localhost 5044
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

I am not able to see any logstash logs its empty, any help here ?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.