# Metricbeat \> Logstash errors: client is not connected

**URL:** <https://discuss.elastic.co/t/metricbeat-logstash-errors-client-is-not-connected/182271>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [May 22, 2019, 3:40pm UTC](https://discuss.elastic.co/t/metricbeat-logstash-errors-client-is-not-connected/182271 "2019-05-22T15:40:02Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jacko-ski](https://avatars.discourse-cdn.com/v4/letter/j/d78d45/32.png) [@Jacko-ski](https://discuss.elastic.co/u/Jacko-ski)\
**Post date:** [May 22, 2019, 3:40pm UTC](https://discuss.elastic.co/t/metricbeat-logstash-errors-client-is-not-connected/182271/1 "2019-05-22T15:40:03Z")

</div>

I'm observing these errors in Metricbeat logs on the sender machine. Looks like every Publish event fails initially, then reconnects after a re-try. Why is that happening? The Logstash consumer targets are deployed behind a Load Balancer.

```
2019-05-22T14:49:39.457Z ERROR logstash/async.go:256 Failed to publish events caused by: read tcp 10.126.142.41:47298->10.111.163.109:30003: i/o timeout
2019-05-22T14:49:39.457Z ERROR logstash/async.go:256 Failed to publish events caused by: read tcp 10.126.142.41:47298->10.111.163.109:30003: i/o timeout
2019-05-22T14:49:39.457Z ERROR logstash/async.go:256 Failed to publish events caused by: read tcp 10.126.142.41:47298->10.111.163.109:30003: i/o timeout
2019-05-22T14:49:39.457Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2019-05-22T14:49:39.457Z INFO [publish] pipeline/retry.go:168 done
2019-05-22T14:49:39.457Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2019-05-22T14:49:39.457Z INFO [publish] pipeline/retry.go:191 done
2019-05-22T14:49:39.458Z ERROR logstash/async.go:256 Failed to publish events caused by: client is not connected
2019-05-22T14:49:39.458Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2019-05-22T14:49:39.458Z INFO [publish] pipeline/retry.go:168 done
2019-05-22T14:49:40.458Z ERROR pipeline/output.go:121 Failed to publish events: client is not connected
2019-05-22T14:49:40.458Z INFO pipeline/output.go:95 Connecting to backoff(async(tcp://tcp.k8sb.cloud9.com:30003))
2019-05-22T14:49:40.458Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2019-05-22T14:49:40.458Z INFO [publish] pipeline/retry.go:191 done
2019-05-22T14:49:40.458Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2019-05-22T14:49:40.458Z INFO [publish] pipeline/retry.go:168 done
2019-05-22T14:49:40.557Z INFO pipeline/output.go:105 Connection to backoff(async(tcp://tcp.k8sb.cloud9.com:30003)) established
2019-05-22T14:49:40.557Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2019-05-22T14:49:40.557Z INFO [publish] pipeline/retry.go:191 done

```

My Configuration file:

```
metricbeat.modules:
  - module: system
  metricsets:
    # CPU stats
    - cpu

    # System Load stats
    - load

    # Per CPU core stats
    #- core

    # IO stats
    #- diskio

    # Per filesystem stats
    - filesystem

    # File system summary stats
    - fsstat

    # Memory stats
    - memory

    # Network stats
    - network

    # Per process stats
    - process

    # Sockets (linux only)
    #- socket
  enabled: true
  period: 60s
  processes: ['ruby', 'java', 'python', 'rabbitmq-server', 'metricbeat', 'filebeat', 'postmaster', 'redis-server', 'node', 'nginx', 'beam.smp']

# Uncomment the field below to add your own tags for filtering
#tags: ["mytag", "tag_two"]

output.logstash:
  hosts: ["tcp://tcp.k8sb.cloud9.com:30003"]

```

# Configuration details:

Metricbeat:  
metricbeat version 6.4.3 (amd64), libbeat 6.4.3 running on Amazon;

Machine:  
Linux eco-mgm 4.14.77-81.59.amzn2.x86\_64 #1 SMP Mon Nov 12 21:32:48 UTC 2018 x86\_64 x86\_64 x86\_64 GNU/Linux

Logstash:  
[docker.elastic.co/logstash/logstash-oss](http://docker.elastic.co/logstash/logstash-oss) tag: 6.6.1 running on Kubernetes

Kubernetes info:  
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}  
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

---

<div class="post-metadata">

**Author:** ![Jacko-ski](https://avatars.discourse-cdn.com/v4/letter/j/d78d45/32.png) [@Jacko-ski](https://discuss.elastic.co/u/Jacko-ski)\
**Post date:** [May 22, 2019, 3:48pm UTC](https://discuss.elastic.co/t/metricbeat-logstash-errors-client-is-not-connected/182271/2 "2019-05-22T15:48:43Z")

</div>

I found a configuration parameter called "ttl", but the documentation says this:

"# Not yet supported for async connections (i.e. with the "pipelining" option set)"

And in my log I see references to "logstash/async.go:256" ...

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [May 22, 2019, 6:31pm UTC](https://discuss.elastic.co/t/metricbeat-logstash-errors-client-is-not-connected/182271/3 "2019-05-22T18:31:05Z")

</div>

Hi @Jacko-ski,

Are you receiving any event in Elasticsearch? Could you give more details on the load balancer? Also, could you share the input configuration of your logstash?

---

<div class="post-metadata">

**Author:** ![Jacko-ski](https://avatars.discourse-cdn.com/v4/letter/j/d78d45/32.png) [@Jacko-ski](https://discuss.elastic.co/u/Jacko-ski)\
**Post date:** [May 24, 2019, 2:08pm UTC](https://discuss.elastic.co/t/metricbeat-logstash-errors-client-is-not-connected/182271/4 "2019-05-24T14:08:43Z")

</div>

@jsoriano -

The attached diagram shows relevant fragments of the architecture. Yes, I am receiving events in Elasticsearch despite these errors. I think it re-establishes the connection on each and is successful with subsequent retry, I'm just not sure why the initial connection fails.

 ![Mon%20cluster%20arch%20fragment](https://us1.discourse-cdn.com/elastic/original/3X/6/c/6c45c4e5ad92014c8bbfc172024d8ea4bc802728.png)

My logstash config is barebone: Input, no filters, Output to ES.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [May 24, 2019, 5:41pm UTC](https://discuss.elastic.co/t/metricbeat-logstash-errors-client-is-not-connected/182271/5 "2019-05-24T17:41:54Z")

</div>

Oh I see, did you try to disable `pipelining` and enable `ttl`?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 21, 2019, 5:47pm UTC](https://discuss.elastic.co/t/metricbeat-logstash-errors-client-is-not-connected/182271/6 "2019-06-21T17:47:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
