Data table Visualisation not showing correct result for numeric data

[ankitdevnalkar]

I am working with Cloudflare analytics data and trying to create a data table visualization to show the sum of the bandwidth in descending order for fields bandwidth.ssl.encrypted, bandwidth.ssl.unencrypted, bandwidth.ssl_protocol.TLSv1.2, bandwidth.ssl_protocol.TLSv1.3, bandwidth.ssl_protocol.none . As a result, I should get a total of all these mentioned fields however, I am getting the wrong data in the visualization.

Following is my logs sample data:

@timestamp
Jan 22, 2021 @ 22:13:01.915
	
@version
1
	
_id
z836KncBzlhaMMTNSjkT
	
_index
cf-analytics
	
_score
 - 
	
_type
_doc
	
bandwidth.all
721MB
	
bandwidth.cached
464.7MB
	
bandwidth.content_type.css
38.1MB
	
bandwidth.content_type.empty
31MB
	
bandwidth.content_type.gif
1.6MB
	
bandwidth.content_type.html
58.6MB
	
bandwidth.content_type.javascript
169MB
	
bandwidth.content_type.jpeg
2MB
	
bandwidth.content_type.json
180.8MB
	
bandwidth.content_type.octet-stream
17MB
	
bandwidth.content_type.other
82.1MB
	
bandwidth.content_type.plain
8.6MB
	
bandwidth.content_type.png
125.8MB
	
bandwidth.content_type.svg
13,740
	
bandwidth.content_type.webp
6,709,892
	
bandwidth.content_type.xml
41.5KB
	
bandwidth.country.AE
2,449,382
	
bandwidth.country.AU
241.4KB
	
bandwidth.country.CA
155.8KB
	
bandwidth.country.CH
1,664,329
	
bandwidth.country.DE
57.1KB
	
bandwidth.country.FR
1MB
	
bandwidth.country.GB
969.3KB
	
bandwidth.country.HK
117,644
	
bandwidth.country.IE
60KB
	
bandwidth.country.IN
604.2MB
	
bandwidth.country.IR
394
	
bandwidth.country.IT
9.4KB
	
bandwidth.country.JP
19.2MB
	
bandwidth.country.KR
678,846
	
bandwidth.country.KW
557.9KB
	
bandwidth.country.MM
32,854
	
bandwidth.country.MY
47.5KB
	
bandwidth.country.NG
30,969
	
bandwidth.country.NL
509.3KB
	
bandwidth.country.NP
23.7KB
	
bandwidth.country.OM
2.5KB
	
bandwidth.country.QA
197.6KB
	
bandwidth.country.RO
760,748
	
bandwidth.country.RS
75,160
	
bandwidth.country.SA
69.2KB
	
bandwidth.country.SE
965
	
bandwidth.country.SG
81.1MB
	
bandwidth.country.TR
881,066
	
bandwidth.country.UA
722,512
	
bandwidth.country.UNKNOWN
1.1MB
	
bandwidth.country.US
4.4MB
	
bandwidth.country.XX
82,878
	
bandwidth.ssl.encrypted
720.3MB
	
bandwidth.ssl.unencrypted
645.4KB
	
bandwidth.ssl_protocol.TLSv1.2
11.4KB
	
bandwidth.ssl_protocol.TLSv1.3
106KB
	
bandwidth.ssl_protocol.none
1.5KB
	
bandwidth.uncached
268,737,704
	
cloudflare-analytics
4026c5dc-1096-4035-8dec-eb94576dfa62
	
pageviews.all
5.03k
	
pageviews.search_engine.facebookexternalhit
4
	
pageviews.search_engine.googlebot
4
	
pageviews.search_engine.twitterbot
1
	
requests.all
121.70k
	
requests.cached
51,616
	
requests.content_type.css
4,087
	
requests.content_type.empty
35,601
	
requests.content_type.gif
119
	
requests.content_type.html
11,845
	
requests.content_type.javascript
4,195
	
requests.content_type.jpeg
188
	
requests.content_type.json
28,345
	
requests.content_type.octet-stream
567
	
requests.content_type.other
3,629
	
requests.content_type.plain
1,508
	
requests.content_type.png
31,460
	
requests.content_type.svg
12
	
requests.content_type.webp
100
	
requests.content_type.xml
42
	
requests.http_status.101
143
	
requests.http_status.200
78,485
	
requests.http_status.204
588
	
requests.http_status.206
6
	
requests.http_status.301
1,648
	
requests.http_status.302
5,656
	
requests.http_status.304
29,147
	
requests.http_status.400
18
	
requests.http_status.403
3,439
	
requests.http_status.404
1,016
	
requests.http_status.408
4
	
requests.http_status.409
1
	
requests.http_status.499
1,469
	
requests.http_status.522
78
	
requests.ip_class.monitoringService
14
	
requests.ip_class.noRecord
120,905
	
requests.ip_class.searchEngine
364
	
requests.ip_class.unknown
415
	
requests.ssl.encrypted
120,174
	
requests.ssl.unencrypted
1,524
	
requests.ssl_protocol.TLSv1.2
11,662
	
requests.ssl_protocol.TLSv1.3
108,512
	
requests.ssl_protocol.none
1,524
	
requests.uncached
70,082
	
since
Jan 22, 2021 @ 21:30:00.000
	
threats.all
3.45k
	
threats.country.IN
759
	
threats.country.JP
2,586
	
threats.country.UA
102
	
uniques.all
6,271
	
until
an hour ago

My visualization settings are as follow :

The visualization result is showing in the below screenshot:

[wylie (Wylie Conlon)]

It seems like you think that you've configured one thing, but what you've actually configured is not the same thing. You are using arrays of objects in Elasticsearch, and these don't work the way you expect. Kibana also doesn't support the "nested" mapping type which is a common workaround for this issue.

Here's what I would recommend instead. You seem to want the "total bandwidth per protocol". What you should do is create a separate document per protocol.

[ankitdevnalkar]

Do you mean a separate bucket for each protocol?

[wylie (Wylie Conlon)]

No, I'm saying that you have a choice:

1. To use the built-in Kibana visualizations, you need to change the way your documents are stored in Elasticsearch

2. If you don't change the documents, then you can only use Vega

[ankitdevnalkar]

In case I use built-in Kibana visualizations, what changes I would require to do?

[wylie (Wylie Conlon)]

The problem here is that you're using filters, which seem to match the entire document. So to use filters you need to have multiple documents that share the same field. For example:

{ id: 1, type: 'TLSv1.3', bandwidth: 10000 }
{ id: 2, type: 'TLSv1.2', bandwidth: 91992929 }

Then you could use filters to get a different value for each type

[ankitdevnalkar]

Thanks.