Logstash forwarder is sending a message to multiple servers

Elastic Stack Logstash
1

We have multiple ELK environments (dev, test etc) and have observed recently that the forwarder is sending messages to multiple environments.

In all environments, the forwarder config looks like this:

{

The network section covers network configuration

"network": {

"servers": [ "host123.domain.com:5043" ],

#"ssl key": "./logstash-forwarder.key",

# The path to your trusted ssl CA file. This is used
# to authenticate your downstream server.
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",

"timeout": 15

}
}

An /etc/hosts entry maps the host123 to the appropriate IP in each environment. Is there a way to enable verbosity in the forwarder logs to see why it is sending to multiple servers ?

2

@tsvivek

You can do a couple things if you run logstash forwarder manually:

sudo /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder.conf

It will output the number of events and whom and what it connected to, unless you have changed the default you can also find a logs in /var/log/logstash-forwarder/

The thing about linux is the only way it knows about anything upstream is it must be explicitly stated there is not much under the hood if you only have one host listed for your connection then I would look upstream at the Logstash host to see where it is outputting received logs.

I'm assuming that host123 is an example in reality each host has a different hostname in your "hosts" file?

Can you santize your forwarder config and post the full thing?

3

I've checked the forwarder logs at /var/log/logstash-forwarder/logstash-forwarder.err. Nothing to indicate a connection was made to a server other than the intended one.

{

"network": {

"servers": [ "elkserver.softlayer.com:5043" ],
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15

}
}
This is the complete network configuraiton. The file paths are in separate conf files.
As mentioned earlier the network configuration is exactly the same in every environment (including the server name). We did this to avoid generating a unique ssl certificate for every environment (i know not a good thing). So the ssl certificate was generated with host name "elkserver.softlayer.com"
So the /etc/hosts in each environment would have the required mapping:

158.23.115.123 elkserver.softlayer.com elkserver

The IP of elkserver speciifed above would change in every environment.