I am doing a reasearch on Logstash+ElasticSearch+Kibana to make sure they meet my needs .Below is the usage scenario:
I want to use them for my Hadoop application logs search in a WEB UI. Since a hadoop application can be executed on any machine , so , check the application log is always cumbersome. A new application will create a directory under /home/log/hadoop/logs/userlogs/
of the machine it is running on and create three log files: syslog
、syserr
、sysout
.
Currently, to check the log of a application , user get the applicationId(for example it is 123123123-223-1), and then goes to directory /home/log/hadoop/logs/userlogs/123123123-223-1/
of each machine to check the logs files. Of cource , only the machine the application running on contains its logs.
So ,after deployed Logstash+ElasticSearch+Kibana ,I wonder if user can query the log for a specific applicaiton Id like this:
For example , user input the applicationId 123123123-223-1as the query condition, ElasticSearch will return the log content for this application, that is to see, the file content under /home/log/hadoop/logs/userlogs/123123123-223-1
which may locate on any machine of my cluster. That is to say, can the subdirectory name(the applicationId) be a index key so , user can just search its own application log?