Pie Chart with Filters Aggregation ..what is my path (Sorry, I really haven't got a clue)?

[ChrizK]

The Test area within my organisation has implemented ELK. The main purpose is to capture, analyse and visualise logs produced by an in-house load tool. There are also other 'applications' which feed logs.

I have mainly got by using Dashboards produced by various people, and have also ventured into Discover. My low level of knowledge is pretty much clicking on +/- to add a filter.
I have also managed some basic filters in the Search bar (KQL?), and I am now venturing into Visualize.

My initial attempt is a pie chart, but I have quickly found I need to 'split' the slices using two different fields, those being HTTP.FAIL and HTTPS.FAIL.
NB When there is a failure, these simply contain '1'.

What I want/need to do, is to show the failure reason on the outer ring (fields HTTP_REASON and HTTPS_REASON).
My inner ring is the environment, so I would like a split chart showing;
environment > HTTP/HTTPS Fail > Reason for failure.
Hopefully that makes sense.

I believe I need to aggregate my second split, ie HTTP.FAIL and HTTPS.FAIL, before I can proceed to 'reason'.
I am trying to follow

Filters Aggregation | Elasticsearch Guide [7.1] | Elastic

but the very first line of the example, PUT /logs/_bulk?refresh, illustrates I have no idea what I am doing!

/logs/_bulk looks like a path, and so my basic question is ...how do I identify the path of the data?

If I view Stack Monitoring, I can see that Logstash and Beats are running. I would guess that these have a number of scheduled tasks which have a configuration to determine where the logs are stored????

Is it a clue that within Discover, there is a drop down box that allows me to select different 'sets' of data (I don't know what to call this, as there is no prompt next to the drop down). The choices include filebeat-auditor-, filebeat-functional-, logstash-report-* etc.
I am fairly sure these directly relate to the logs for each 'function', the load test tool being logstash-report, and the functional test logging being filebeat-functional ...but how do I identify the path I need in the PUT statement?

Hopefully the question makes sense!
(I also have no idea what ?refresh does, but I expect I will need additional help with the whole example/query)

NB we are running Version 7.1.1

[warkolm (Mark Walkom)]

Welcome to our community!

Can you share an example of what one of your events looks like?

[ChrizK]

Thanks Mark, I am a little cautious posting publicly, as it may affect intellectual property. I will have a look at some examples, and see if I need to obscure anything. Unfortunately, it may take me a bit of time (I only get to 'play' with Kibana in my down time). Really appreciate you answering, will get back ASAP.

[ChrizK]

As a quick first attempt, is this sufficient, or is it too restricted?

Is it the case that my question does not make sense, and you are attempting to identify the goal and propose a different solution other than aggregation?
I had hoped that aggregating HTTP.FAIL/HTTPS.FAIL (only being value = 1), would allow me to produce two segments 'around' each environment, allowing me to illustrate the HTTP.REASON/HTTPS.REASON for each environment.

Ah ... I may need some direction on how to 'share events'. I added columns to Discover and copied to Excel to modify/obscure, then tried to paste here, but it loses the table and makes it hard to read. Also tried via Word, but also loses the table... Posting this message sorts out the table, but the MSG content is truncated (probably because it uses pipe as a delimiter). The MSG was only for additional information, and not my focus. It contains greater detail on the transaction, including a better description of the failure, which is unfortunately removed when posting.

Time	ENVIRONMENT	HTTP.FAIL	HTTP.REASON	HTTPS.FAIL	HTTPS.REASON	MSG
Aug 23, 2021 @ 21:39:30.050	env1	-	-	1	hostname resolution failure	1629751170.049916
Aug 23, 2021 @ 21:39:28.035	env1	-	-	-	-	1629751168.034796
Aug 23, 2021 @ 21:39:20.046	env1	1	Server Timeout	-	-	1629751160.046061

[warkolm (Mark Walkom)]

Ok in Kibana you will want to do this in Lens;

1. Create a pie chart
2. The first aggregation will be a terms on the HTTP.FAIL and HTTPS.FAIL field
3. Then add another terms aggregation below that on the HTTP_REASON and HTTPS_REASON field

Kibana runs the aggregation in Elasticsearch under the hood, literally using the same APIs and taking the response and graphing that for you. So you don't need to worry too much about the Elasticsearch docs t this stage.

[ChrizK]

Thanks Mark, really appreciate you sticking with me.
Lens was one of the first visualisation methods I read about, but I quickly found that it is not available in 7.1 (I understand it was made generally available in 7.10).
I am hoping it is just a more user friendly progression of the Visualize functionality available to me, and I can achieve what you are suggesting, however, I believe what you have said amounts to what I was trying to do... I cannot work out how to combine 'the first aggregation on the two terms'. I thought it was the aggregation method I linked to in initial question, but then got stuck on identifying 'the path of my data'.
Although it may be in the past for you , you may recall creating a pie from Visualize. You can then choose to Split Slices, BUT, I can only add ONE term at this level/'ring' (at least, as far as I can tell) ...if Lens provides additional capability, are you able to retrospectively tell me if I can achieve the same thing prior to lens?

NB I actually would like to have the three environments shown in the first ring, with HTTP/HTTPS.FAIL being the next layer, but I can work on that after successfully aggregating the fails.

[ChrizK]

In attempt to illustrate what I am trying to do...

(Would be nice to change the '1' value in the legend to be HTTP.FAIL or HTTPS.FAIL)

[warkolm (Mark Walkom)]

That would be the Custom Label box at the bottom left there

[ChrizK]

I think the Custom Label only changes the 'hover box' when the mouse is over the segment, not the value shown in the legend. At least, that is what it appears to do in this version.

Did you have any thoughts on how I can aggregate the two terms (HTTP.FAIL and HTTPS.FAIL)? If the article was correct 'solution', I think I need to identify the location of the 'logstash', or am I heading in the wrong direction?
(attempting to illustrate what I am trying to do also shows I need to aggregate the REASON per FAIL, but I hope that is straight forward once the method is understood)

[ChrizK]

Could someone tell me if the Filters Aggregation article is the correct method to achieve what I am trying to do? If so, how do I identify what I believe to be the 'path' of the data.
Sitting back and thinking about it, I assume the PUT is a location of the aggregation, and I should be asking how to determine the path\location for the GET.

This also leads me to further questions, does the PUT have to be in the same path (I don't understand _bulk?refresh)? Is the PUT data temporary or does it need to be cleansed ...is that the purpose of ?refresh?

I would very much appreciate any help that can be provided.