Logstash node not sending to elasticsearch node

[smaxxx]

Hello,

so after performance problems. we decided to split logstash and elasticsearch up in separate nodes. However, the logstash node isn't sending anything to the elasticsearch node. Here are my findings so far:

Can the logstash node ping the elasticsearch node? Yes.

CT-1241 logstash# curl -XGET XXX.XXX.XXX.XX:9200
{
  "name" : "blabla.hostname.blabla.redacted",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "venez9W1Q2O44c0wuVVSHw",
  "version" : {
    "number" : "7.4.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96",
    "build_date" : "2019-10-28T20:40:44.881551Z",
    "build_snapshot" : false,
    "lucene_version" : "8.2.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

How does my output file look like?:

output {
  elasticsearch {
    id => "output_elasticsearch_single"
    hosts => [ "${ELASTIFLOW_ES_HOST:127.0.0.1:9200}" ]
    ssl => "${ELASTIFLOW_ES_SSL_ENABLE:false}"
    ssl_certificate_verification => "${ELASTIFLOW_ES_SSL_VERIFY:false}"
    # If ssl_certificate_verification is true, uncomment cacert and set the path to the certificate.
    #cacert => "/PATH/TO/CERT"
    user => "${ELASTIFLOW_ES_USER:elastic}"
    password => "${ELASTIFLOW_ES_PASSWD:changeme}"
    index => "elastiflow-3.5.3-%{+YYYY.MM.dd}"
    template => "${ELASTIFLOW_TEMPLATE_PATH:/etc/logstash/elastiflow/templates}/elastiflow.template.json"
    template_name => "elastiflow-3.5.3"
    template_overwrite => "true"
  }
}

The environment variable "ELASTIFLOW_ES_HOST" is set:
Environment="ELASTIFLOW_ES_HOST=XXX.XXX.XXX.XX:9200"

Here are the elasticsearch.yml settings:

# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
transport.host: localhost
transport.tcp.port: 9300

Logstash log:
https://hastebin.com/puyibuhocu.cpp

I can send data to the elasticsearch node from the logstash node manually, so it's not a firewall issue:

curl -XPOST http://redacted:9200/test/doc -H "Content-Type: application/json" -d @test.json
{"_index":"test","_type":"doc","_id":"HktFVm8B9m-GyVBgEKON","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

How can I fix this? What do I have to configure to send the data from logstash to a remote elasticsearch node?

// Edit: Also yes, the logstash node is receiving data. Confirmed with tcpdump and udp. I'm using Elastiflow (https://github.com/robcowart/elastiflow)

[Christian_Dahlqvist]

If you can curl a request to the custer without specifying authentication or SSL it does not look like your cluster is secured, which you have configured for your Elasticsearch output. Seems inconsistent.

[smaxxx]

So I can see that the elasticsearch node receives data from the logstash node (tcpdump src logstashnodeIP):
https://pastebin.com/tZHmKzh6

But somehow only after the restart of the logstash service and then after that, no more data is sent / received. Weird...

//edit: Elasticsearch logs: https://pastebin.com/HFmzzBzV

[smaxxx]

I honestly have no idea anymore what could be the reason that it doesn't send the data, but "pings" when it starts

[smaxxx]

So I investigated further and looked at the HTTP Stream when logstash starts.
It really downloads the template and acknowledges the elasticsearch node, but that was it. https://pastebin.com/xBsrFx73

[smaxxx]

Fixed it. was a Firewall issue (SELinux)...