Elasticsearch does not send logs to filebeat and/or Logstash

Our situation is as follows: We have a Kubernetes Cluster with a lot of logs piling up. We would like to get a centralized place for the logs to query through. This centralized place is a local VM, which has Elasticsearch, Kibana, Logstash and Filebeat installed in one place.

The kubernetes cluster (with IP 192.168.253.160) sends the logs to the IP of the local VM (192.168.253.150) and the port of local Elasticsearch (which is 9200). This all works fine and I can see the logs being send to Elasticsearch using Kibana.

Only the problem here is that these logs don't go through Logstash and/or Filebeat, because if I try to put a filter on the Logstash configuration, nothing happens to the logs. Even if I stop Logstash and Filebeat completely, the logs still get send to Elasticsearch no problem.

I'm pretty sure the logs don't need to be send to Elasticsearch port, but when I try to send them to the ports 9600 (which is Logstash) or 5044 (which is Filebeat) the logs don't go through and I can't see them come into Kibana anymore. Maybe I need the use of the HTTP API?

Below I'll provide the changes I made in the configs of Elasticsearch, Logstash and Filebeat:

Elasticsearch:

    nano /etc/elasticsearch/elasticsearch.yml
    # ---------------------------------- Network -----------------------------------
    #
    # Set the bind address to a specific IP (IPv4 or IPv6):
    #
    network.host: 0.0.0.0
    #
    # Set a custom port for HTTP:
    #
    #http.port: 9200
    #
    # For more information, consult the network module documentation.
    #

Filebeat:

    nano /etc/filebeat/filebeat.yml
    # ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["0.0.0.0:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

Logstash:
nano /etc/logstash/conf.d/02-beats-input.conf

input {
  beats {
    port => 5044
  }
}
output {
         stdout { codec => rubydebug }
}

Also, I'm very new to all of this and IT in general, so please make it a bit noob friendly. Thanks!

That's not a valid host to send data to. It needs to be the actual IP of the host running Logstash.

The output of filebeat needs to be sent to logstash on the local VM. When I change this to hosts: ["localhost:5044"] still same result.

Can you telnet localhost:5044 from the Logstash host?

The Logstash host is the localhost.

Ok but can you run that?

telnet localhost 5044
Trying ::1...
Connected to localhost.
Escape character is '^]'.

Connection closed by foreign host.

What do your Filebeat logs show?

Feb 16 09:47:34 test.net filebeat[2868]: 2021-02-16T03:47:34.405-0500        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp 12
Feb 16 09:47:34 test.net filebeat[2868]: 2021-02-16T03:47:34.405-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 2
Feb 16 09:47:34 test.net filebeat[2868]: 2021-02-16T03:47:34.406-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:47:34 test.net filebeat[2868]: 2021-02-16T03:47:34.406-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:47:35 test.net filebeat[2868]: 2021-02-16T03:47:35.891-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:47:43 test.net filebeat[2868]: 2021-02-16T03:47:43.044-0500        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp 12
Feb 16 09:47:43 test.net filebeat[2868]: 2021-02-16T03:47:43.044-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 3
Feb 16 09:47:43 test.net filebeat[2868]: 2021-02-16T03:47:43.048-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:47:43 test.net filebeat[2868]: 2021-02-16T03:47:43.048-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:48:05 test.net filebeat[2868]: 2021-02-16T03:48:05.893-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:48:11 test.net filebeat[2868]: 2021-02-16T03:48:11.661-0500        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp 12
Feb 16 09:48:11 test.net filebeat[2868]: 2021-02-16T03:48:11.661-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 4
Feb 16 09:48:11 test.net filebeat[2868]: 2021-02-16T03:48:11.662-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:48:11 test.net filebeat[2868]: 2021-02-16T03:48:11.662-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:48:11 test.net filebeat[2868]: 2021-02-16T03:48:11.663-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:151        Connection to backoff(async(tcp://localhost:5044)) established
Feb 16 09:48:35 test.net filebeat[2868]: 2021-02-16T03:48:35.894-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:49:05 test.net filebeat[2868]: 2021-02-16T03:49:05.894-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:49:35 test.net filebeat[2868]: 2021-02-16T03:49:35.894-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:50:05 test.net filebeat[2868]: 2021-02-16T03:50:05.893-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:50:35 test.net filebeat[2868]: 2021-02-16T03:50:35.893-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:50:39 test.net filebeat[2868]: 2021-02-16T03:50:39.977-0500        ERROR        [logstash]        logstash/async.go:280        Failed to publish events caused by: read tcp [::1]:40178->[::1]:5044: read: connection r
Feb 16 09:50:39 test.net filebeat[2868]: 2021-02-16T03:50:39.977-0500        ERROR        [logstash]        logstash/async.go:280        Failed to publish events caused by: read tcp [::1]:40178->[::1]:5044: read: connection r
Feb 16 09:50:39 test.net filebeat[2868]: 2021-02-16T03:50:39.977-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:50:39 test.net filebeat[2868]: 2021-02-16T03:50:39.977-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:50:39 test.net filebeat[2868]: 2021-02-16T03:50:39.977-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:50:39 test.net filebeat[2868]: 2021-02-16T03:50:39.977-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:50:40 test.net filebeat[2868]: 2021-02-16T03:50:40.119-0500        ERROR        [logstash]        logstash/async.go:280        Failed to publish events caused by: client is not connected
Feb 16 09:50:40 test.net filebeat[2868]: 2021-02-16T03:50:40.119-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:50:40 test.net filebeat[2868]: 2021-02-16T03:50:40.119-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:50:41 test.net filebeat[2868]: 2021-02-16T03:50:41.864-0500        ERROR        [publisher_pipeline_output]        pipeline/output.go:180        failed to publish events: client is not connected
Feb 16 09:50:41 test.net filebeat[2868]: 2021-02-16T03:50:41.864-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:143        Connecting to backoff(async(tcp://localhost:5044))
Feb 16 09:50:41 test.net filebeat[2868]: 2021-02-16T03:50:41.865-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:50:41 test.net filebeat[2868]: 2021-02-16T03:50:41.865-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:50:45 test.net filebeat[2868]: 2021-02-16T03:50:45.443-0500        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp 12
Feb 16 09:50:45 test.net filebeat[2868]: 2021-02-16T03:50:45.443-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 1
Feb 16 09:50:45 test.net filebeat[2868]: 2021-02-16T03:50:45.443-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:50:45 test.net filebeat[2868]: 2021-02-16T03:50:45.444-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:50:52 test.net filebeat[2868]: 2021-02-16T03:50:52.822-0500        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp [:
Feb 16 09:50:52 test.net filebeat[2868]: 2021-02-16T03:50:52.822-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 2
Feb 16 09:50:52 test.net filebeat[2868]: 2021-02-16T03:50:52.825-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:50:52 test.net filebeat[2868]: 2021-02-16T03:50:52.826-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:51:01 test.net filebeat[2868]: 2021-02-16T03:51:01.082-0500        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp [:
Feb 16 09:51:01 test.net filebeat[2868]: 2021-02-16T03:51:01.082-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 3
Feb 16 09:51:01 test.net filebeat[2868]: 2021-02-16T03:51:01.083-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:51:01 test.net filebeat[2868]: 2021-02-16T03:51:01.083-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:51:05 test.net filebeat[2868]: 2021-02-16T03:51:05.891-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:51:31 test.net filebeat[2868]: 2021-02-16T03:51:31.907-0500        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp [:
Feb 16 09:51:31 test.net filebeat[2868]: 2021-02-16T03:51:31.907-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 4
Feb 16 09:51:31 test.net filebeat[2868]: 2021-02-16T03:51:31.908-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:51:31 test.net filebeat[2868]: 2021-02-16T03:51:31.908-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:51:35 test.net filebeat[2868]: 2021-02-16T03:51:35.895-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:52:05 test.net filebeat[2868]: 2021-02-16T03:52:05.893-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:52:24 test.net filebeat[2868]: 2021-02-16T03:52:24.726-0500        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp [:
Feb 16 09:52:24 test.net filebeat[2868]: 2021-02-16T03:52:24.727-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 5
Feb 16 09:52:24 test.net filebeat[2868]: 2021-02-16T03:52:24.728-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:52:24 test.net filebeat[2868]: 2021-02-16T03:52:24.728-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:52:32 test.net filebeat[2868]: 2021-02-16T03:52:32.237-0500        INFO        log/harvester.go:333        File is inactive: /var/log/secure. Closing because close_inactive of 5m0s reached.
Feb 16 09:52:35 test.net filebeat[2868]: 2021-02-16T03:52:35.892-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:53:05 test.net filebeat[2868]: 2021-02-16T03:53:05.893-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:53:20 test.net filebeat[2868]: 2021-02-16T03:53:20.887-0500        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp 12
Feb 16 09:53:20 test.net filebeat[2868]: 2021-02-16T03:53:20.887-0500        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 6
Feb 16 09:53:20 test.net filebeat[2868]: 2021-02-16T03:53:20.888-0500        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Feb 16 09:53:20 test.net filebeat[2868]: 2021-02-16T03:53:20.888-0500        INFO        [publisher]        pipeline/retry.go:223          done
Feb 16 09:53:35 test.net filebeat[2868]: 2021-02-16T03:53:35.894-0500        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"
Feb 16 09:54:00 test.net filebeat[2868]: 2021-02-16T03:54:00.458-0500        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp 12

Note: I replaced the current hostname to test.net

And Filebeat is installed on the same host as Logstash right?

yes.

I've found the solution. It had to do with the input of Filebeat and to which port the logs were send to from the Kubernetes cluster. I changed the port to where to send the logs to, to 9201. I changed the filebeat.input like this, so that it listens on port 9021 over tcp protocol:

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: tcp
  hosts: ["localhost:9201"]
#- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  #paths:
    #- /var/log/*.log

Also, I had to change the output of Logstash so it got send to Elasticsearch:

input {
  beats {
    port => 5044
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "test-index"
    }
}