Unable to connect local filebeat to distant logstash

Hey everyone,

I've got an issue when i try to send some logs from A server filebeat to B server ELK.

Telnet test :

[root@A:/etc/ansible]# telnet B 5044
Trying B...
Connected to B.
Escape character is '^]'.

(same network, no firewall).

Filebeat config test :

[root@A:/etc/ansible]# /usr/share/filebeat/bin/filebeat test config --path.home /etc/filebeat/
Config OK

Filebeat config :

filebeat.inputs:

- type: log

  enabled: true

  paths:
    - /my/path/DISPONIBILITE*

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

setup.kibana:

output.logstash:
  hosts: ["B:5044"]
  
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Filebeat output test : (During logstash remote pipeline is listening)

logstash: B:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: @IP B
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

But my filebeat service says :

mai 20 10:59:39 lpansmld1 filebeat[63959]: 2020-05-20T10:59:39.383+0200 ERROR [publisher_pipeline_output] pipeline/output.go:106 Failed to connect to backoff(elasticsearch(http://B:5044)): Get http://B:5044: read tcp @IP A:50456->@IP B:5044: read: connection reset by peer

And same time my logstash pipeline says :

[INFO ] 2020-05-20 11:00:18.149 [defaultEventExecutorGroup-4-2] BeatsHandler - [local: @IP B:5044, remote: @IP A:50526] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69
[WARN ] 2020-05-20 11:00:18.149 [nioEventLoopGroup-2-7] DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 69
        at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.0.9.jar:?]
        at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.0.9.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        ... 10 more

Now, my logstash configuration : (default)

path.data: /var/lib/logstash
pipeline.ordered: auto
path.logs: /var/log/logstash

And my pipeline configuration :

input {
  beats {
    port => 5044
  }
}
filter
{
  grok {
    patterns_dir => [ "/etc/logstash/conf.d/patterns" ]
    match => { "message" => [ "%{DATA:DATE};%{WORD:ELD};%{INT:ETAT}" ] }
  }
  mutate {
    convert => { "ETAT" => "integer" }
  }
  date {
    match => [ "DATE", "yy-MM-dd HH:mm" ]
  }

}
output
{
  elasticsearch
  {
    hosts => "http://localhost:9200"
    index => "myindex"
  }
  stdout { codec => rubydebug }
}

I read that it could be a tls problem but it is ELK test server, no TLS anywhere on this install.

Do you need more informations ?

Can somebody help me ?

Thanks Elastic Team.

You error is:

Make sure you are running a compatible version of filebeat for your version of server

You can see the compatibility matrix here https://www.elastic.co/support/matrix#matrix_compatibility

Hey,

[ansible@A]$ filebeat version
filebeat version 7.7.0 (amd64), libbeat 7.7.0

Same as logstash

Can somebody help me ?

Thx