The optimal way to aggregate in Kibana information from multiple Elasticsearch indexes


(Vagif Abilov) #1

I originally posted this question on StackOverflow, but I see that this
group might be a more suitable place for it.

We are setting up logs from several related applications so the log events
are imported into Elasticsearch (via Logstash). It was straightforward
create Kibana dashboards to visualize log indexes for each application, but
since the applications are related and its activities belong to the same
pipeline, it would be great to build a dashboard that would show aggregated
information, collected from different applications. Such dashboard would be
especially useful to track failures and performance problems.

Right now I can see three main ways to implement aggregated dashboard:

  1. Keep separate application logs and configure Kibana dashboard that
    would consume information from different applications. I am afraid this can
    be a challenging task, I am not even sure Kibana fully supports it.
  2. Revise application logging so they will all log to the same index.
    What I dislike about this is that log event structure must be then unified
    across applications, and they are built by different people in different
    languages. I've lost my faith to centralized control over such low level
    details like logging.
  3. Keep applications log and corresponding Elastichsearch indexes as
    they are now, but set up a new index which will contain aggregate
    information. This article
    http://www.elasticsearch.org/blog/logging-elasticsearch-events-with-logstash-and-elasticsearch/
    describes how to configure Elasticsearch to dump it’s logs to Logstash
    which would then insert them back into Elasticsearch for searching. At
    first glance this approach may look surprising: why would you need to
    re-insert log data once again into the same database? It's another index,
    it adds overhead, uses more space etc. But it gives the opportunity to set
    up the index in a way that will be suitable for a aggregated Kibana
    dashboard.

I wonder if someone has gone through a similar dilemma and can share their
experience.

Thanks in advance

Vagif Abilov

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ac164ad2-ea7f-4b00-a9af-fc6e819949e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #2