I have the output filter below. It helps me choose between my development Elasticsearch server and the corporate one. If I set [@metadata][local_es] to false it will send the events to the corporate server otherwise it sends it to my local development server. On Windows this works beautifully and I can flip it at will and the right server gets the events. On Linux it consistently tries the local server on localhost in spite of the flag. And so , of course since I have nothing listening on localhost port 9200 it consistently fails. Can someone tells me if this is a known issue?
An elasticsearch output establishes a connection at startup, it does not wait for an event to arrive. If nothing is listening on localhost:9200 I would expect logstash to continuously log errors about being unable to connect.
Well, all that is fine, except the code tells it what server to talk to and it seems that on Linux there is a break down in communication because although there is a server ready, willing and able to receive events, Logstash latches on localhost and will not let go event its life depended on it ;forgive my humor
Yes the code tells it which server to talk to, which as @Badger explained, in your case is BOTH of them.
All that your if statements achieves is determining for each event to which of the two outputs will the event be sent. That could be different from one event to the next.
What you may want to consider is changing to a single output which is set using an environment variable, e.g. ELASTICSEARCH_HOSTS. In your elasticsearch output you would then use:
hosts => [ "${ELASTICSEARCH_HOSTS}" ]
This is especially useful if files are developed locally, where ELASTICSEARCH_HOSTS=http://localhost:9200, and then deployed onto the Linux server where ELASTICSEARCH_HOSTS=http://corporatehost:80. They will work in both environments without modification.
Thanks Rob for your detailed answer. I am open to trying out environment variables and forgive me if I belabor the point. I am really thinking there is a bug here because of the following reasons:
The code behaves as expected on Windows
It cannot be both because I expressly set it at the beginning of each event
Perhaps you need to shed a little more light with respect to how it can be
Yeah. I have stopped my local server. I set it to send events to the corporate server. It tries repeatedly to contact localhost. I think I am all set. if an event comes it will be routed to the proper server but it is just that it will keep trying to contact the servers listed in the output filter. I took the failure to reach the unreachable server as an inability to function. I will definitely look at the environment variable approach. Thanks @rcowart ob and @Badger.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.