Hi, I would like to setup a Kibana GUI to query some log files in HDFS. So, I'm looking at the Hadoop-ES connector if this can help to meet my requirement. However, there are few doubts I have now.
Do I need to store all these log files into ElasticSearch in order to query using Kibana? Then, I will have double copy of log files in HDFS and ElasticSearch.
How Hadoop-ES connector to connect to HDFS? I refer to the documents, it only mentions about connecting to Hive, Pig, Spark, Storm, etc, but no HDFS.
Where should I install this Hadoop-ES connector? In ES cluster, or Hadoop cluster? I have gone through the installation guide, however, it didn't state the installation steps clearly. Anyone can help?