Conditional in output filter fails on Linux

[moderable]

I have the output filter below. It helps me choose between my development Elasticsearch server and the corporate one. If I set [@metadata][local_es] to false it will send the events to the corporate server otherwise it sends it to my local development server. On Windows this works beautifully and I can flip it at will and the right server gets the events. On Linux it consistently tries the local server on localhost in spite of the flag. And so , of course since I have nothing listening on localhost port 9200 it consistently fails. Can someone tells me if this is a known issue?

output {
	if ![@metadata][local_es]  {
		elasticsearch  {
			hosts => ["http://corporatehost:80"]
			document_id => "%{[@metadata][prefix]}%{[@metadata][fingerprint]}"
			index => "filebeat-log-%{logtype}-%{+YYYY.MM.dd}"
		}
	}  else {
		elasticsearch {
			hosts => ["http://localhost:9200"]
			document_id => "%{[@metadata][prefix]}%{[@metadata][fingerprint]}"
			index => "filebeat-log-%{logtype}-%{+YYYY.MM.dd}"
		}
	}
	stdout {
		codec => rubydebug { metadata => true }
	}	  
}

[Badger]

An elasticsearch output establishes a connection at startup, it does not wait for an event to arrive. If nothing is listening on localhost:9200 I would expect logstash to continuously log errors about being unable to connect.

[moderable]

Well, all that is fine, except the code tells it what server to talk to and it seems that on Linux there is a break down in communication because although there is a server ready, willing and able to receive events, Logstash latches on localhost and will not let go event its life depended on it ;forgive my humor

[moderable]

This seems to be a bug in Logstash to me. I think I need to create a bug report in Github.

[moderable]

So is this me or something else. I never seem to find an answer to my questions on this forum. I am beginning to wonder.

[rcowart (Rob Cowart)]

@moderable

all that is fine, except the code tells it what server to talk to

Yes the code tells it which server to talk to, which as @Badger explained, in your case is BOTH of them.

All that your if statements achieves is determining for each event to which of the two outputs will the event be sent. That could be different from one event to the next.

What you may want to consider is changing to a single output which is set using an environment variable, e.g. ELASTICSEARCH_HOSTS. In your elasticsearch output you would then use:

hosts => [ "${ELASTICSEARCH_HOSTS}" ]

This is especially useful if files are developed locally, where ELASTICSEARCH_HOSTS=http://localhost:9200, and then deployed onto the Linux server where ELASTICSEARCH_HOSTS=http://corporatehost:80. They will work in both environments without modification.

Environment variables are the key to making Logstash pipelines that are easily customized for multiple environments without modifying the pipelines themselves. For example... https://github.com/robcowart/elastiflow/blob/master/INSTALL.md#environment-variable-reference

Rob

How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations
What is the best storage technology for Elasticsearch?

[moderable]

Thanks Rob for your detailed answer. I am open to trying out environment variables and forgive me if I belabor the point. I am really thinking there is a bug here because of the following reasons:

1. The code behaves as expected on Windows
2. It cannot be both because I expressly set it at the beginning of each event

Perhaps you need to shed a little more light with respect to how it can be

different from one event to the next

[moderable]

I think, I am getting a little light. It tries all servers before it gets any event. Let me verify that.

[moderable]

Yeah. I have stopped my local server. I set it to send events to the corporate server. It tries repeatedly to contact localhost. I think I am all set. if an event comes it will be routed to the proper server but it is just that it will keep trying to contact the servers listed in the output filter. I took the failure to reach the unreachable server as an inability to function. I will definitely look at the environment variable approach. Thanks @rcowart ob and @Badger.