How to optimize my ElasticSearch Query

Hello, I use a really strong request for my project. I tried to follow Elasticsearch tunes for search speed but it looks like its slower than before. Here you can find my request before I did the optimizations :

{
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "must": [
              {
                "bool": {
                  "must": [
                    {
                      "bool": {
                        "must": [
                          {
                            "bool": {
                              "must": [
                                {
                                  "term": {
                                    "countryId": {
                                      "value": "fr"
                                    }
                                  }
                                },
                                {
                                  "term": {
                                    "siteId": {
                                      "value": "vz"
                                    }
                                  }
                                },
                                {
                                  "term": {
                                    "periodicityId": {
                                      "value": 3
                                    }
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "budgetMin": {
                                            "lte": 479000
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "budgetMin"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "budgetMax": {
                                            "gte": 479000
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "budgetMax"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "roomMin": {
                                            "lte": 4
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "roomMin"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "roomMax": {
                                            "gte": 4
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "roomMax"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "bedRoomMin": {
                                            "lte": 3
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "bedRoomMin"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "bedRoomMax": {
                                            "gte": 3
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "bedRoomMax"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "surfaceMin": {
                                            "lte": 88
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "surfaceMin"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "range": {
                                          "surfaceMax": {
                                            "gte": 88
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "must_not": [
                                            {
                                              "exists": {
                                                "field": "surfaceMax"
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "term": {
                                    "propertyTypes": {
                                      "value": 6
                                    }
                                  }
                                },
                                {
                                  "term": {
                                    "indiv": {
                                      "value": false
                                    }
                                  }
                                },
                                {
                                  "term": {
                                    "withVisual3DLink": {
                                      "value": false
                                    }
                                  }
                                }
                              ],
                              "must_not": [
                                {
                                  "term": {
                                    "advertType": {
                                      "value": "rentals"
                                    }
                                  }
                                },
                                {
                                  "exists": {
                                    "field": "landSurfaceMin"
                                  }
                                },
                                {
                                  "exists": {
                                    "field": "landSurfaceMax"
                                  }
                                }
                              ]
                            }
                          },
                          {
                            "bool": {
                              "should": [
                                {
                                  "bool": {
                                    "must": [
                                      {
                                        "bool": {
                                          "should": [
                                            {
                                              "term": {
                                                "cityIds": {
                                                  "value": 19270
                                                }
                                              }
                                            },
                                            {
                                              "term": {
                                                "departmentIds": {
                                                  "value": 4325
                                                }
                                              }
                                            },
                                            {
                                              "term": {
                                                "regionIds": {
                                                  "value": 979
                                                }
                                              }
                                            }
                                          ]
                                        }
                                      },
                                      {
                                        "match": {
                                          "searchMode": {
                                            "query": "default"
                                          }
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "must": [
                                      {
                                        "geo_shape": {
                                          "polygonalShapes": {
                                            "shape": {
                                              "type": "point",
                                              "coordinates": [
                                                -1.7193224,
                                                48.0779145
                                              ]
                                            }
                                          }
                                        }
                                      },
                                      {
                                        "bool": {
                                          "should": [
                                            {
                                              "match": {
                                                "searchMode": {
                                                  "query": "polygonal"
                                                }
                                              }
                                            },
                                            {
                                              "match": {
                                                "searchMode": {
                                                  "query": "isochrone"
                                                }
                                              }
                                            }
                                          ]
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "must": [
                                      {
                                        "geo_shape": {
                                          "radius": {
                                            "shape": {
                                              "type": "point",
                                              "coordinates": [
                                                -1.7193224,
                                                48.0779145
                                              ]
                                            }
                                          }
                                        }
                                      },
                                      {
                                        "match": {
                                          "searchMode": {
                                            "query": "radius"
                                          }
                                        }
                                      }
                                    ]
                                  }
                                },
                                {
                                  "bool": {
                                    "must": [
                                      {
                                        "geo_shape": {
                                          "polygonalShapes": {
                                            "shape": {
                                              "type": "point",
                                              "coordinates": [
                                                -1.7193224,
                                                48.0779145
                                              ]
                                            },
                                            "relation": "contains"
                                          }
                                        }
                                      },
                                      {
                                        "match": {
                                          "searchMode": {
                                            "query": "aspirational_search"
                                          }
                                        }
                                      }
                                    ],
                                    "must_not": [
                                      {
                                        "geo_shape": {
                                          "polygonalHoles": {
                                            "shape": {
                                              "type": "point",
                                              "coordinates": [
                                                -1.7193224,
                                                48.0779145
                                              ]
                                            },
                                            "relation": "contains"
                                          }
                                        }
                                      }
                                    ]
                                  }
                                }
                              ]
                            }
                          },
                          {
                            "term": {
                              "reducePrice": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "withVideoLink": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "isNotOverlooked": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "isNorthFacing": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "isWestFacing": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "isEastFacing": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "isLastFloor": {
                                "value": false
                              }
                            }
                          }
                        ]
                      }
                    },
                    {
                      "bool": {
                        "should": [
                          {
                            "term": {
                              "floorPreference": {
                                "value": "no_ground_floor"
                              }
                            }
                          },
                          {
                            "term": {
                              "floorPreference": {
                                "value": "none"
                              }
                            }
                          },
                          {
                            "bool": {
                              "must_not": [
                                {
                                  "exists": {
                                    "field": "floorPreference"
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "bool": {
                  "should": [
                    {
                      "bool": {
                        "must": [
                          {
                            "term": {
                              "isFurnished": {
                                "value": false
                              }
                            }
                          }
                        ],
                        "must_not": [
                          {
                            "term": {
                              "unfurnished": {
                                "value": true
                              }
                            }
                          }
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "term": {
                              "unfurnished": {
                                "value": true
                              }
                            }
                          }
                        ],
                        "must_not": [
                          {
                            "term": {
                              "isFurnished": {
                                "value": false
                              }
                            }
                          }
                        ]
                      }
                    },
                    {
                      "bool": {
                        "must": [
                          {
                            "term": {
                              "isFurnished": {
                                "value": false
                              }
                            }
                          },
                          {
                            "term": {
                              "unfurnished": {
                                "value": true
                              }
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "term": {
                  "hasPool": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasCellar": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasGarage": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasHandicapAccess": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasTerrace": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasFireplace": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasClosets": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasParquet": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasBuildingManager": {
                    "value": false
                  }
                }
              },
              {
                "term": {
                  "hasAlarm": {
                    "value": false
                  }
                }
              }
            ]
          }
        }
      ]
    }
  },
  "size": 10000
}

Sorry for the long request.

Do you have any idea what can I do to have faster request and to limit CPU/Disk usage ?

Thanks for the time you'll take for me

Which version of Elasticsearch are you using?

What is the size and specification of your cluster?

What is the size of the index/indices you are querying? How many primary and replica shards?

What latency are you experiencing?

Thanks for your quick answer.

I'm using the the version 7.17.8.

I got a cluster of 6 nodes, with 32gb of heap size for each.

I got 1.3M of docs and 5.7Gb. I got 1 primary shards and 2 replicas by node.

The problem is when I'm doing multiple request to the index in parallel I got "data too large" errors. I'm trying to optimize my mapper and my requests.
But with the tuning my request are as twice slower than before.

Another thing, I'm doing researches with geoshape. For a GeoPoint I'm looking which docs contains it. Maybe it requires a lot of CPU.

Does this mean you have 6 primary shards?

If that is the case I would recommend shrinking this index down to a single primary shard and set the number of replicas to 5 so all nodes hold a copy of this index. As your index is quite small it should fit in the page cache.

I'm not an expert in Elasticsearch.

I'm right when i'm saying that i got a primary shards by nodes ?

Looks like you have 5 primary shards and 2 replicas set up. Try with 1 primary shard and 5 replicas instead.

Thank for your recommandation. Do you know where i can find the documentation to find out more on shards and replicas configuration ?

Look in the official documentation.

Thanks for your time. Have a great day !

Just to check, your index name had today’s date in it?

Is there a new index every day/wee/whatever ?

Hello, We create a new index every months. Also, this index is updated continuously. I tried to reduce the shards number but I still get errors like "data too large".

FYI, the "data too large" exception rise when we perform parallel requests during a batch.

Did you successfully change to 1 primary shard and 5 replicas? Can you share equivalent of screenshot above ?

I struggled to parse your query (as a human just reading it). Is that query generated by a person, or some application ?

You can profile the query too, see where the time is being spent.

I changed to 1 primary and 2 replicas to test.

I use Nest in C# to generate this query.

here is a screenshot of all my indexes :

at the left you can see the modified index we're talking about

What does the profiler tell you?

I already tried to profile my query but the query is so big my Multi Elasticsearch Heads pluggin crashes when i'm trying to look at it. I can't copy and paste it.

Ok, the query generated by NEST/C# is not nice.

Did you try devtools in Kibana ?

Maybe have a think what the query is trying to find, and rebuild that manually.

Currently we don't have Kibana. I'll install it and look forward with it