I'm testing Logstash sniffing in our environment for the Elasticsearch output. So far, it appears to resolve a few issues I've had, but I've got a few questions:
sniffing => true in my elasticsearch output appears to return all elasticsearch nodes in my cluster. I use dedicated ingest nodes. When Logstash is choosing a node to send output to, does it only choose from among the ingest nodes? If not, is there a setting I can include in sniffing_path to force it to only return ingest nodes?
How frequently does Logstash do an "update" sniff, to update the list of nodes? Or do I have to restart Logstash after adding/removing nodes to/from my cluster?
The default for the sniffing_delay option on the output is documented as 5 seconds, but looking at the code it appears to be 10. Down at line 152 you can see it just sits in a loop sniffing and then delaying.
Looking at the sniffing code it skips "master" nodes and includes everything else.
I saw that, but couldn't determine from the documentation whether it sniffed every 5s, or that was just a "sniff again in 5s if you don't get a response."
Thanks for pointing me there. Unfortunately, that's a bummer, as I use dedicated ingest nodes. However, I'm not seeing errors so far.
I know that the sniffer_path allows me to set a specific endpoint including, presumably, parameters, if accepted. Unfortunately, the documentation for the node info API in Elasticsearch makes it appear that this is not possible. And while _nodes/_master/http will return master node info, there doesn't appear to be a corresponding _nodes/_ingest/http to return ingest node info.
Is there anything you may be aware of that I'm missing?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.