How to parse ES json output. Scripting, re-indexing, other suggestion?

(Steph Cooper) #1


I am very new to the stack and need some advice.
I have been trying to make sense out of our logs, imported to ES using logstash, and generating graphs (and queries) from kibana.
I ran into some difficulties when I started to look for a way to visualize some success/failure ratio.
As I can't get a readable visualization in kibana for this problem, I decided to write the Elasticsearch query using the Bucket Script Aggregations needed to get this information.

I now have the information I was looking for, in a nice and very long JSON format, ready to be parsed and sorted.
Hundreds of blocks like this:

    "key": "my_institution_name",
     "doc_count": 153255,
      "Response_success": {
        "doc_count": 116924
      "failure_ratio": {
        "value": 23.706

My question is what are my options to use this elasticsearch output and actually make something readable out of it?
My first idea is to write a script to parse it all and create a sorted table Institutions - Failure ratio.

Do you have some alternatives that would make more sense?
Should I (/ is there a good way ) to create an index out of the result and to use it into kibana? The original idea was to make a graph, it seems a bit conterproductive to skip kibana, parse the thing by hand and plot the result... :-/

Trying to vizualize "top N/ordered" ratios
(Court Ewing) #2

It does seem a bit counterproductive to just skip Kibana altogether. Your proposal about indexing the data in a format that can be easily handled with visualizations in Kibana seems like a reasonable workaround until something like can be completed. I've seen people do something similar before, so it isn't unprecedented.

Something like watcher can be used to calculate updated ratios on an interval.

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.