I originally posted this question on StackOverflow, but I see that this
group might be a more suitable place for it.
We are setting up logs from several related applications so the log events
are imported into Elasticsearch (via Logstash). It was straightforward
create Kibana dashboards to visualize log indexes for each application, but
since the applications are related and its activities belong to the same
pipeline, it would be great to build a dashboard that would show aggregated
information, collected from different applications. Such dashboard would be
especially useful to track failures and performance problems.
Right now I can see three main ways to implement aggregated dashboard:
- Keep separate application logs and configure Kibana dashboard that
would consume information from different applications. I am afraid this can
be a challenging task, I am not even sure Kibana fully supports it. - Revise application logging so they will all log to the same index.
What I dislike about this is that log event structure must be then unified
across applications, and they are built by different people in different
languages. I've lost my faith to centralized control over such low level
details like logging. - Keep applications log and corresponding Elastichsearch indexes as
they are now, but set up a new index which will contain aggregate
information. This article
http://www.elasticsearch.org/blog/logging-elasticsearch-events-with-logstash-and-elasticsearch/
describes how to configure Elasticsearch to dump it’s logs to Logstash
which would then insert them back into Elasticsearch for searching. At
first glance this approach may look surprising: why would you need to
re-insert log data once again into the same database? It's another index,
it adds overhead, uses more space etc. But it gives the opportunity to set
up the index in a way that will be suitable for a aggregated Kibana
dashboard.
I wonder if someone has gone through a similar dilemma and can share their
experience.
Thanks in advance
Vagif Abilov
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ac164ad2-ea7f-4b00-a9af-fc6e819949e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.