Querying Elastic Search

[oraclept]

I want to query across all available indices on given elastic search instance and get following information.

1. List available indices
2. unique host names available across available indices
3. All source log in unique host names in available indices.
4. last update and early update available on source log.

I have my own file which will contain host name and source log information to compare with and flag host names and source log file which are not available in elastic search instance.

I see there is rest API.

https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docs.html

Some one who worked on APIs.

Should I build my own dashboard using rest API or use Kibana ?
Any pointers will be helpful.

Thanks,

[warkolm (Mark Walkom)]

Why not use Kibana, it'll make things a lot faster.

[oraclept]

Thanks Mark for prompt reply.

If using kibana . how would I compare with static file to flag if there are any hosts or source log information not available in elastic search ?

[warkolm (Mark Walkom)]

You will need to ingest that data into Elasticsearch, or do comparisons to that data during ingestion using a translation table or similar.

[oraclept]

Sorry I did not get data ingestion part.
I have to write my own script to do comparison ? Is it something elastic search provides.
We use log stash and grok logs before sending it to elastic search.

[warkolm (Mark Walkom)]

If you want to compare against a static file you have three options;

1. Do it outside of Elasticsearch, where you retrieve the events you want via and then compare them, all with a custom script
2. Put the data into Elasticsearch in it's own index, then when you ingest the events through Logstash, run an Elasticsearch filter on the IP to enhance the event
3. The same as number 2, just do it with a translate filter in Logstash