Convert string results into failure rates

(Robert Cronk) #1

I have a string field (data.result) in my documents that can contain "SUCCESS" or "FAILURE". I'm running ~20 tests across ~50 servers and each test on a server has a "SUCCESS" or "FAILURE" result. I want to make a heatmap of the failure rates with test names on one axis and server names on the other axis. I am already doing this successfully with an average "test duration" field and it works great.

I think I just need to convert "SUCCESS" to 100 and "FAILURE" to 0 and then do the average of those to get a failure rate. I tried to convert the data.result field to numbers using the JSON Input field in the heatmap visualization like this:

{"script":"if (doc['data.result'].value = 'SUCCESS') { return 100; } else { return 0; }}

But I don't know which field I should select for the "average". I only have already existing numeric fields to choose from, not the result of this script. And it also gave me, "Courier Fetch: 25 of 34 shards failed."

I also tried to aggregate it in the script like this

        "lang": "painless",
        "inline": "int totalSuccess = 0; int totalFailure = 0; for (int i = 0; i < doc['_type'].length; ++i) { if(doc['data.result'][i] =='SUCCESS') {totalSuccess += 1;} if(doc['data.result'][i] == 'FAILURE'){totalFailure +=1;}} return totalFailure/(totalFailure+totalSuccess);"

This one gives me "Courier Fetch: 5 of 34 shards failed."

I also tried to add a scripted field to the index, but that gave me errors about too many script compilations per minute and we increased that and it just gave us similar errors to what we got above.

I'd also like to know what tools people are using to test/debug/validate their scripts before throwing them into production. It says to test the scripts out beforehand, but it doesn't say what tool to use for that.

I obviously don't know what I'm doing, so any help will be greatly appreciated. I have been a C++/Python developer for 25 years, but ELK is brand new to me.

If I can't get this working with a scripted field, I'll have to create a new field in the document that just has 0 for FAILURE's and 100 for SUCCESS's and then do an average on that numeric field. I'd really like to figure out how to get a scripted field to work in a heatmap though.

(Joe Fleming) #2

Converting the values from success and failure to 100 and 0 respectively shouldn't be too difficult with a scripted field, and logically what you have in that first example is right. In Kibana, you can add a new scripted field, set its type to number, and set the script to the following:

doc['data.result'].value == 'SUCCESS' ? 100 : 0

This will give you a new field, with whatever name you gave it, that maps that string value to a number. Then you can use the field to get the average value in the heat map.

This isn't your data, but the idea is the same. I called my field ext_val, as you can see in the metric config.

Building Visualization Table
(Robert Cronk) #3

Thanks, Joe! I created a scripted field in the index with the exact script you have above, and it shows up as a field I can select but when I do that, I get the following error:

Courier Fetch: 5 of 34 shards failed.

What might be causing that error?

(Robert Cronk) #4

I figured it out. I had to instead use the following script because apparently some records don't have that field in them:

if (!doc['data.result.keyword'].empty) {
  return doc['data.result.keyword'].value == 'SUCCESS' ? 100 : 0;
return null;

I also had to add .keyword to the end of the field name to get it working. It's working beautifully now.

(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.