Hello Elastic community,
First of all, I would like to thank the ELK team for their support and rich online documentation.
But, I'm facing a performance problem that I didn't know how to solve, let me explain:
I have datatables on kibana along with other visualisations like maps, histograms etc.
I have documents of type "trip [date,rider,driver,destination,pickup,cost,distance,etc]",
I have created a datatable to aggregate trips like that:
{
"title": "Request list",
"type": "table",
"params": {
"perPage": 10,
"showMeticsAtAllLevels": false,
"showPartialRows": false,
"showTotal": false,
"sort": {
"columnIndex": 8,
"direction": "desc"
},
"totalFunc": "sum"
},
"aggs": [
{
"id": "10",
"enabled": true,
"type": "min",
"schema": "metric",
"params": {
"field": "requested_at",
"customLabel": "Date of first request"
}
},
{
"id": "1",
"enabled": true,
"type": "count",
"schema": "metric",
"params": {
"customLabel": "Number of requests"
}
},
{
"id": "11",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "rider_profile.rider_id",
"size": 10000,
"order": "desc",
"orderBy": "10",
"customLabel": "Rider ID"
}
},
{
"id": "2",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "rider_profile.full_name",
"size": 10000,
"order": "desc",
"orderBy": "1",
"customLabel": "Rider name"
}
},
{
"id": "3",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "rider_profile.email",
"size": 10000,
"order": "desc",
"orderBy": "1",
"customLabel": "Rider email"
}
},
{
"id": "4",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "rider_profile.phone",
"size": 10000,
"order": "desc",
"orderBy": "1",
"customLabel": "Rider phone"
}
},
{
"id": "5",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "pickup.formatted_address",
"size": 10000,
"order": "desc",
"orderBy": "1",
"customLabel": "Pickup location"
}
},
{
"id": "6",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "destination.formatted_address",
"size": 10000,
"order": "desc",
"orderBy": "1",
"customLabel": "Destination"
}
},
{
"id": "7",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "status",
"size": 10000,
"order": "desc",
"orderBy": "1",
"customLabel": "Status"
}
},
{
"id": "8",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "estimated_distance",
"size": 10000,
"order": "desc",
"orderBy": "1",
"customLabel": "Distance"
}
},
{
"id": "9",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "estimated_cost",
"size": 10000,
"order": "desc",
"orderBy": "1",
"customLabel": "Cost"
}
}
],
"listeners": {}
}
In my index, I have 19920 documents, but when setting a bigger date range, I get the error Bad gateway which means that my cluster is Overloaded!
My cluster consists of
- 2 master nodes
- 1 Client node
- 3 Data node
- 1 Discovery node
Do you think that I'm doing wrong visualizations (specially with an aggregation of size 10000) Or is it a problem with my cluster performance that need to be enhanced (I would to not go to this solution because I already have a limited ressources).
Thanks in advance for any clarifications or propositions.