How to parse ES json output. Scripting, re-indexing, other suggestion?

Elastic Stack Elasticsearch
1

Hi,

I am very new to the stack and need some advice.
I have been trying to make sense out of our logs, imported to ES using logstash, and generating graphs (and queries) from kibana.
I ran into some difficulties when I started to look for a way to visualize some success/failure ratio.
As I can't get a readable visualization in kibana for this problem, I decided to write the Elasticsearch query using the Bucket Script Aggregations needed to get this information.

I now have the information I was looking for, in a nice and very long JSON format, ready to be parsed and sorted.
Hundreds of blocks like this:

  ...
  {
    "key": "my_institution_name",
     "doc_count": 153255,
      "Response_success": {
        "doc_count": 116924
      },
      "failure_ratio": {
        "value": 23.706
      }
    },
    ...

My question is what are my options to use this elasticsearch output and actually make something readable out of it?
My first idea is to write a script to parse it all and create a sorted table Institutions - Failure ratio.

Do you have some alternatives that would make more sense?
Should I (/ is there a good way ) to create an index out of the result and to use it into kibana? The original idea was to make a graph, it seems a bit conterproductive to skip kibana, parse the thing by hand and plot the result... :-/

2

It does seem a bit counterproductive to just skip Kibana altogether. Your proposal about indexing the data in a format that can be easily handled with visualizations in Kibana seems like a reasonable workaround until something like https://github.com/elastic/kibana/issues/4707 can be completed. I've seen people do something similar before, so it isn't unprecedented.

Something like watcher can be used to calculate updated ratios on an interval.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.