Metricbeat > Logstash errors: client is not connected

Elastic Stack Beats
1

I'm observing these errors in Metricbeat logs on the sender machine. Looks like every Publish event fails initially, then reconnects after a re-try. Why is that happening? The Logstash consumer targets are deployed behind a Load Balancer.

2019-05-22T14:49:39.457Z        ERROR   logstash/async.go:256   Failed to publish events caused by: read tcp 10.126.142.41:47298->10.111.163.109:30003: i/o timeout
2019-05-22T14:49:39.457Z        ERROR   logstash/async.go:256   Failed to publish events caused by: read tcp 10.126.142.41:47298->10.111.163.109:30003: i/o timeout
2019-05-22T14:49:39.457Z        ERROR   logstash/async.go:256   Failed to publish events caused by: read tcp 10.126.142.41:47298->10.111.163.109:30003: i/o timeout
2019-05-22T14:49:39.457Z        INFO    [publish]       pipeline/retry.go:166   retryer: send wait signal to consumer
2019-05-22T14:49:39.457Z        INFO    [publish]       pipeline/retry.go:168     done
2019-05-22T14:49:39.457Z        INFO    [publish]       pipeline/retry.go:189   retryer: send unwait-signal to consumer
2019-05-22T14:49:39.457Z        INFO    [publish]       pipeline/retry.go:191     done
2019-05-22T14:49:39.458Z        ERROR   logstash/async.go:256   Failed to publish events caused by: client is not connected
2019-05-22T14:49:39.458Z        INFO    [publish]       pipeline/retry.go:166   retryer: send wait signal to consumer
2019-05-22T14:49:39.458Z        INFO    [publish]       pipeline/retry.go:168     done
2019-05-22T14:49:40.458Z        ERROR   pipeline/output.go:121  Failed to publish events: client is not connected
2019-05-22T14:49:40.458Z        INFO    pipeline/output.go:95   Connecting to backoff(async(tcp://tcp.k8sb.cloud9.com:30003))
2019-05-22T14:49:40.458Z        INFO    [publish]       pipeline/retry.go:189   retryer: send unwait-signal to consumer
2019-05-22T14:49:40.458Z        INFO    [publish]       pipeline/retry.go:191     done
2019-05-22T14:49:40.458Z        INFO    [publish]       pipeline/retry.go:166   retryer: send wait signal to consumer
2019-05-22T14:49:40.458Z        INFO    [publish]       pipeline/retry.go:168     done
2019-05-22T14:49:40.557Z        INFO    pipeline/output.go:105  Connection to backoff(async(tcp://tcp.k8sb.cloud9.com:30003)) established
2019-05-22T14:49:40.557Z        INFO    [publish]       pipeline/retry.go:189   retryer: send unwait-signal to consumer
2019-05-22T14:49:40.557Z        INFO    [publish]       pipeline/retry.go:191     done

My Configuration file:

metricbeat.modules:
  - module: system
  metricsets:
    # CPU stats
    - cpu

    # System Load stats
    - load

    # Per CPU core stats
    #- core

    # IO stats
    #- diskio

    # Per filesystem stats
    - filesystem

    # File system summary stats
    - fsstat

    # Memory stats
    - memory

    # Network stats
    - network

    # Per process stats
    - process

    # Sockets (linux only)
    #- socket
  enabled: true
  period: 60s
  processes: ['ruby', 'java', 'python', 'rabbitmq-server', 'metricbeat', 'filebeat', 'postmaster', 'redis-server', 'node', 'nginx', 'beam.smp']

# Uncomment the field below to add your own tags for filtering
#tags: ["mytag", "tag_two"]

output.logstash:
  hosts: ["tcp://tcp.k8sb.cloud9.com:30003"]

Configuration details:

Metricbeat:
metricbeat version 6.4.3 (amd64), libbeat 6.4.3 running on Amazon;

Machine:
Linux eco-mgm 4.14.77-81.59.amzn2.x86_64 #1 SMP Mon Nov 12 21:32:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Logstash:
docker.elastic.co/logstash/logstash-oss tag: 6.6.1 running on Kubernetes

Kubernetes info:
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

2

I found a configuration parameter called "ttl", but the documentation says this:

"# Not yet supported for async connections (i.e. with the "pipelining" option set)"

And in my log I see references to "logstash/async.go:256" ...

3

Hi @Jacko-ski,

Are you receiving any event in Elasticsearch? Could you give more details on the load balancer? Also, could you share the input configuration of your logstash?

4

@jsoriano -

The attached diagram shows relevant fragments of the architecture. Yes, I am receiving events in Elasticsearch despite these errors. I think it re-establishes the connection on each and is successful with subsequent retry, I'm just not sure why the initial connection fails.

Mon%20cluster%20arch%20fragment
Mon cluster arch fragment.png1198×485 67.4 KB

My logstash config is barebone: Input, no filters, Output to ES.

5

Oh I see, did you try to disable pipelining and enable ttl?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.