Filebeat and Logstash not connecting

Elastic Stack Logstash
1

I am reaching out to seek your expertise and guidance regarding an issue I am facing with transferring logs from Filebeat to Logstash. I have a setup where Filebeat is installed on 'Server1', which sends logs to Logstash installed on 'Server2'. However, I am encountering difficulties in getting the logs on Elasticsearch which is also on 'Server2', and I am seeing an error in the Logstash logs.

Logstash Logs

io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:260)
io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:356)
io.netty.util.concurrent.AbstractEventExecutor.runTask(io/netty/util/concurrent/AbstractEventExecutor.java:174)
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:167)
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:470)
io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:569)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:997)
io.netty.util.internal.ThreadExecutorMap$2.run(io/netty/util/internal/ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30)
java.lang.Thread.run(java/lang/Thread.java:833)
[2023-07-13T14:01:48,670][INFO ][org.logstash.beats.Server][main][73907a21526e666e42a7bc377eb6bd9e68afd77900cf719e2d7f4d4bf32155f3] Starting server on port: 5044
[2023-07-13T14:01:54,687][ERROR][logstash.javapipeline    ][main][73907a21526e666e42a7bc377eb6bd9e68afd77900cf719e2d7f4d4bf32155f3] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Beats host=>"Server1", port=>5044, id=>"73907a21526e666e42a7bc377eb6bd9e68afd77900cf719e2d7f4d4bf32155f3", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_47630d6f-99bd-4d00-8349-2ae33d49c9d2", enable_metric=>true, charset=>"UTF-8">, ssl=>false, ssl_enabled=>false, ssl_client_authentication=>"none", ssl_verify_mode=>"none", ssl_peer_metadata=>false, include_codec_tag=>true, ssl_handshake_timeout=>10000, ssl_cipher_suites=>["TLS_AES_256_GCM_SHA384", "TLS_AES_128_GCM_SHA256", "TLS_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], ssl_supported_protocols=>["TLSv1.2", "TLSv1.3"], client_inactivity_timeout=>60, executor_threads=>4, add_hostname=>false, tls_min_version=>1, tls_max_version=>1.3>
  Error: Cannot assign requested address
  Exception: Java::JavaNet::BindException
  Stack: sun.nio.ch.Net.bind0(Native Method)
sun.nio.ch.Net.bind(sun/nio/ch/Net.java:555)
sun.nio.ch.ServerSocketChannelImpl.netBind(sun/nio/ch/ServerSocketChannelImpl.java:337)
sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:294)

Logstash.conf


input {
  beats {
     host => "Server1"
     port => 5044
  }
}

output {
  if [fields][log_type] == "jicofo" {
    elasticsearch {
      hosts => ["Server2:9200"]
      index => "jicofo-%{+YYYY.MM.dd}"
    }
  } else if [fields][log_type] == "jvb" {
    elasticsearch {
      hosts => ["Server2:9200"]
      index => "jvb-%{+YYYY.MM.dd}"
    }
  }
}

filebeat yml file

filebeat.inputs:
- type: log
 id: jicofo-logs
 enabled: true
paths:
    - /var/log/jitsi/jicofo.log

- type: log
  id: jvb-logs
  enabled: true
  paths:
    - /var/log/jitsi/jvb.log
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat.log
  keepfiles: 7
  permissions: 0644

filebeat logs

{"log.level":"debug","@timestamp":"2023-07-13T07:53:10.778+0530","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":223},"message":"Start next scan","service.name":"filebeat","input_id":"4aa95f1e-fbaa-451e-9a87-d69214559cf4","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-13T07:53:10.778+0530","log.logger":"input","log.origin":{"file.name":"input/input.go","file.line":137},"message":"Run input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-13T07:53:10.778+0530","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":223},"message":"Start next scan","service.name":"filebeat","input_id":"297352d3-3914-4a6a-be07-26f3457b8943","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-13T07:53:10.778+0530","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":471},"message":"Check file for harvesting: /var/log/jitsi/jvb.log","service.name":"filebeat","input_id":"4aa95f1e-fbaa-451e-9a87-d69214559cf4","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-13T07:53:10.778+0530","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":471},"message":"Check file for harvesting: /var/log/jitsi/jicofo.log","service.name":"filebeat","input_id":"297352d3-3914-4a6a-be07-26f3457b8943","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-13T07:53:10.778+0530","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":569},"message":"Update existing file for harvesting: /var/log/jitsi/jvb.log, offset: 1010749","service.name":"filebeat","input_id":"4aa95f1e-fbaa-451e-9a87-d69214559cf4","source_file":"/var/log/jitsi/jvb.log","state_id":"native::15878-64768","finished":false,"os_id":"15878-64768","old_source":"/var/log/jitsi/jvb.log","old_finished":true,"old_os_id":"15878-64768","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-13T07:53:10.778+0530","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":624},"message":"File didn't change: /var/log/jitsi/jvb.log","service.name":"filebeat","input_id":"4aa95f1e-fbaa-451e-9a87-d69214559cf4","source_file":"/var/log/jitsi/jvb.log","state_id":"native::15878-64768","finished":false,"os_id":"15878-64768","old_source":"/var/log/jitsi/jvb.log","old_finished":true,"old_os_id":"15878-64768","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-13T07:53:10.778+0530","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":287},"message":"input states cleaned up. Before: 1, After: 1, Pending: 0","service.name":"filebeat","input_id":"4aa95f1e-fbaa-451e-9a87-d69214559cf4","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-13T07:53:10.778+0530","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":569},"message":"Update existing file for harvesting: /var/log/jitsi/jicofo.log, offset: 0","service.name":"filebeat","input_id":"297352d3-3914-4a6a-be07-26f3457b8943","source_file":"/var/log/jitsi/jicofo.log","state_id":"native::79602-64768","finished":false,"os_id":"79602-64768","old_source":"/var/log/jitsi/jicofo.log","old_finished":true,"old_os_id":"79602-64768","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-13T07:53:10.778+0530","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":624},"message":"File didn't change: /var/log/jitsi/jicofo.log","service.name":"filebeat","input_id":"297352d3-3914-4a6a-be07-26f3457b8943","source_file":"/var/log/jitsi/jicofo.log","state_id":"native::79602-64768","finished":false,"os_id":"79602-64768","old_source":"/var/log/jitsi/jicofo.log","old_finished":true,"old_os_id":"79602-64768","ecs.version":"1.6.0"}
2

Hi @Priyaansh_Dwivedi

The Beats Input Host parameter is a listening parameter. It looks like you have it configured to try and connect to Server1 from Server2. That wont work.

remove the host => "Server1" from your logstash config.

from what you posted, it looks like your Filebeat config is missing an output section.
See Configure the Logstash output | Filebeat Reference [8.8] | Elastic

you will want to add something like:

output.logstash:
  hosts: ["Server2:5044"]

in order to forward the events from filebeat to the logstash that you have listening on port 5044.

3

Thanks @eMitch for your response.
I have removed the host part from the Logstash config. Also in Beats, output.logstash was mentioned.
Still I am getting errors . data not being shift.

4

Did you check the network connectivity? If there is a firewall between server 1 and server 2, you can not send logs to observation cluster (logstash, elasticsearch) on server 2.

I had the same issue and solved it by disabling firewall. But i am not sure if i got the same errors. I do not remember that part.

5 
input {
  beats {
     host => "0.0.0.0"
     port => 5044
  }
}

Set host to 0.0.0.0 and try again.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.