How to optimize my ElasticSearch Query

Hello, I use a really strong request for my project. I tried to follow Elasticsearch tunes for search speed but it looks like its slower than before. Here you can find my request before I did the optimizations :

{
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "must": [
              {
                "bool": {
                  "must": [
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "must": [
                                {
                                  "term": {
                                    "countryId": {
                                      "value": "fr"
                                    }
                                  }
                                },
                                {
                                  "term": {
                                    "siteId": {
                                      "value": "vz"
                                    }
                                  }
                                },
                                {
                                  "term": {
                                    "periodicityId": {
                                      "value": 3
                                    }
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "budgetMin": {
                                            "lte": 479000
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "budgetMin"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "budgetMax": {
                                            "gte": 479000
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "budgetMax"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "roomMin": {
                                            "lte": 4
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "roomMin"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "roomMax": {
                                            "gte": 4
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "roomMax"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "bedRoomMin": {
                                            "lte": 3
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "bedRoomMin"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "bedRoomMax": {
                                            "gte": 3
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "bedRoomMax"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "surfaceMin": {
                                            "lte": 88
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "surfaceMin"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "surfaceMax": {
                                            "gte": 88
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "surfaceMax"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "term": {
                                    "propertyTypes": {
                                      "value": 6
                                    }
                                  }
                                },
                                {
                                  "term": {
                                    "indiv": {
                                      "value": false
                                    }
                                  }
                                },
                                {
                                  "term": {
                                    "withVisual3DLink": {
                                      "value": false
                                    }
                                  }
                                }
                              ],
                              "must_not": [
                                {
                                  "term": {
                                    "advertType": {
                                      "value": "rentals"
                                    }
                                  }
                                },
                                {
                                  "exists": {
                                    "field": "landSurfaceMin"
                                  }
                                },
                                {
                                  "exists": {
                                    "field": "landSurfaceMax"
                                  }
                                }
                              ]
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "bool": {
                                    "must": [
                                      {
                                        "bool": {
                                          "should": [
                                            {
                                              "term": {
                                                "cityIds": {
                                                  "value": 19270
                                                }
                                              }
                                            },
                                            {
                                              "term": {
                                                "departmentIds": {
                                                  "value": 4325
                                                }
                                              }
                                            },
                                            {
                                              "term": {
                                                "regionIds": {
                                                  "value": 979
                                                }
                                              }
                                            }
                                          ]
                                        }
                                      },
                                      {
                                        "match": {
                                          "searchMode": {
                                            "query": "default"
                                          }
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "must": [
                                      {
                                        "geo_shape": {
                                          "polygonalShapes": {
                                            "shape": {
                                              "type": "point",
                                              "coordinates": [
                                                -1.7193224,
                                                48.0779145
                                              ]
                                            }
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "should": [
                                            {
                                              "match": {
                                                "searchMode": {
                                                  "query": "polygonal"
                                                }
                                              }
                                            },
                                            {
                                              "match": {
                                                "searchMode": {
                                                  "query": "isochrone"
                                                }
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "must": [
                                      {
                                        "geo_shape": {
                                          "radius": {
                                            "shape": {
                                              "type": "point",
                                              "coordinates": [
                                                -1.7193224,
                                                48.0779145
                                              ]
                                            }
                                          }
                                        }
                                      },
                                      {
                                        "match": {
                                          "searchMode": {
                                            "query": "radius"
                                          }
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "must": [
                                      {
                                        "geo_shape": {
                                          "polygonalShapes": {
                                            "shape": {
                                              "type": "point",
                                              "coordinates": [
                                                -1.7193224,
                                                48.0779145
                                              ]
                                            },
                                            "relation": "contains"
                                          }
                                        }
                                      },
                                      {
                                        "match": {
                                          "searchMode": {
                                            "query": "aspirational_search"
                                          }
                                        }
                                      }
                                    ],
                                    "must_not": [
                                      {
                                        "geo_shape": {
                                          "polygonalHoles": {
                                            "shape": {
                                              "type": "point",
                                              "coordinates": [
                                                -1.7193224,
                                                48.0779145
                                              ]
                                            },
                                            "relation": "contains"
                                          }
                                        }
                                      }
                                    ]
                                  }
                                }
                              ]
                            }
                          },
                          {
                            "term": {
                              "reducePrice": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "withVideoLink": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "isNotOverlooked": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "isNorthFacing": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "isWestFacing": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "isEastFacing": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "isLastFloor": {
                                "value": false
                              }
                            }
                          }
                        ]
                      }
                    },
                    {
                      "bool": {
                        "should": [
                          {
                            "term": {
                              "floorPreference": {
                                "value": "no_ground_floor"
                              }
                            }
                          },
                          {
                            "term": {
                              "floorPreference": {
                                "value": "none"
                              }
                            }
                          },
                          {
                            "bool": {
                              "must_not": [
                                {
                                  "exists": {
                                    "field": "floorPreference"
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "should": [
                    {
                      "bool": {
                        "must": [
                          {
                            "term": {
                              "isFurnished": {
                                "value": false
                              }
                            }
                          }
                        ],
                        "must_not": [
                          {
                            "term": {
                              "unfurnished": {
                                "value": true
                              }
                            }
                          }
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "term": {
                              "unfurnished": {
                                "value": true
                              }
                            }
                          }
                        ],
                        "must_not": [
                          {
                            "term": {
                              "isFurnished": {
                                "value": false
                              }
                            }
                          }
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "term": {
                              "isFurnished": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "unfurnished": {
                                "value": true
                              }
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "term": {
                  "hasPool": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasCellar": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasGarage": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasHandicapAccess": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasTerrace": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasFireplace": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasClosets": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasParquet": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasBuildingManager": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasAlarm": {
                    "value": false
                  }
                }
              }
            ]
          }
        }
      ]
    }
  },
  "size": 10000
}

Sorry for the long request.

Do you have any idea what can I do to have faster request and to limit CPU/Disk usage ?

Thanks for the time you'll take for me

Which version of Elasticsearch are you using?

What is the size and specification of your cluster?

What is the size of the index/indices you are querying? How many primary and replica shards?

What latency are you experiencing?

Thanks for your quick answer.

I'm using the the version 7.17.8.

I got a cluster of 6 nodes, with 32gb of heap size for each.

I got 1.3M of docs and 5.7Gb. I got 1 primary shards and 2 replicas by node.

The problem is when I'm doing multiple request to the index in parallel I got "data too large" errors. I'm trying to optimize my mapper and my requests.
But with the tuning my request are as twice slower than before.

Another thing, I'm doing researches with geoshape. For a GeoPoint I'm looking which docs contains it. Maybe it requires a lot of CPU.

Does this mean you have 6 primary shards?

If that is the case I would recommend shrinking this index down to a single primary shard and set the number of replicas to 5 so all nodes hold a copy of this index. As your index is quite small it should fit in the page cache.

I'm not an expert in Elasticsearch.

I'm right when i'm saying that i got a primary shards by nodes ?

Looks like you have 5 primary shards and 2 replicas set up. Try with 1 primary shard and 5 replicas instead.

Thank for your recommandation. Do you know where i can find the documentation to find out more on shards and replicas configuration ?

Look in the official documentation.

Thanks for your time. Have a great day !

Just to check, your index name had today’s date in it?

Is there a new index every day/wee/whatever ?

Hello, We create a new index every months. Also, this index is updated continuously. I tried to reduce the shards number but I still get errors like "data too large".

FYI, the "data too large" exception rise when we perform parallel requests during a batch.

Did you successfully change to 1 primary shard and 5 replicas? Can you share equivalent of screenshot above ?

I struggled to parse your query (as a human just reading it). Is that query generated by a person, or some application ?

You can profile the query too, see where the time is being spent.

I changed to 1 primary and 2 replicas to test.

I use Nest in C# to generate this query.

here is a screenshot of all my indexes :

at the left you can see the modified index we're talking about

What does the profiler tell you?

I already tried to profile my query but the query is so big my Multi Elasticsearch Heads pluggin crashes when i'm trying to look at it. I can't copy and paste it.

Ok, the query generated by NEST/C# is not nice.

Did you try devtools in Kibana ?

Maybe have a think what the query is trying to find, and rebuild that manually.

Currently we don't have Kibana. I'll install it and look forward with it

Curious if you made any progress ?

I did the 1 primary and 5 replicas config on my cluster. I had less errors but still a lot (1,3k). During my batch I send 10 requests in parallel, maybe that's too many. How can I quantify the maximum number of parallel requests my cluster can handle ?

That recommendation assumed you were querying a single index. If you are querying multiple indices you may wan to reduce the number of replicas to e.g. 2.

Start testing with a single concurrent query and then gradually increase the number of concurrent queries until you start seeing errors or get excessive latencies.