I would like to set up our Elasticsearch Cluster on some hosts and it consists of 4~5 hosts. I also want to use Logstash to put data into Elasticsearch cluster. My website will run Logstash when customer requests it. Until now, I’m running all of that (Elasticsearch, Logstash and website) on single host for test. There is no problem. However, I’m trying to set up the cluster on different hosts now. The Elasticsearch cluster will have 4~5 hosts and my website will have 1 host. I wonder that where Logstash should be in? I think that Logstash should be in website host because website calls the Logstash directly. Also Logstash connects with Elasticsearch based on host.
If anybody has idea or recommendation about this, please let me know.
Having the web site's backend code fire up Logstash seems like a bad idea compared to e.g. configure Logstash to have an http input that the backend can call. If you go with the latter you could easily run Logstash on any machine you have. But yes, if you insist on letting the backend start Logstash it makes better sense to do that on the same machine.
Thank you for answering my question I know your point but I have no idea how to implement your example. If I create http connection in backendn, how Logstash detect the connection? Is the HTTP input plugin provide the function? If you have any example configuration about this, could you please share it with me?
Configuring host and port of http input plugin https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html in input section of basic logstash config should be sufficient to run logstash on any host (and make sure you have proper ports open). Then you can use some REST methods to send messages to your logstash.
Thank you I read some document about the HTTP input plugin. I also need to use JDBC input plugin because I have to get data from DB. Is it possible to implement something like this:
when Logstash get a message thru HTTP.
get data from DB
run some filtering action
index data into Elasticsearch
Can I use JDBC input after HTTP input?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
I know your point but I have no idea how to implement your example. If I create http connection in backendn, how Logstash detect the connection? Is the HTTP input plugin provide the function? If you have any example configuration about this, could you please share it with me?