Node does not reclaim full memory after GC kicks at 75% and it eventually 'dies'

[itmeze (Michal)]

Hi,

I am new to ElasticSearch and at the moment i am evaluating it as a replacement for our sql server based search (in a couple of months time we will need text search for our products)

I have an index for product search for the website. Each product is quite complex with 50-60 attributes. Overall index contains 220k products (1gb). There are no text fields, yet (so far we try to swap sql server with es)

We set up a single node cluster via aws elasticsearch, es 7.4, r5.large.elasticseach instance type as really that was the quickest way for us to start - as we already use some of aws services.
This is still a tryout, to evaluate if es is the way to go for us. We direct 20% of our search to that node, leaving 80% to sql server.

At least twice a day full index is recreated: 1. create new index, 2. populate it with bulk insert, 3. swap alias, and 4. drop an old index. Number of searchable documents remain the same after process completes.

We run quite a lot of search result aggregations (group by product features).

I have no idea why, but it looks like jvm cannot reclaim enough memory, which eventually leads to increased cpu, and eventually starts to trigger circuit_breaker_exception. This happens each time - after few days node is unusable - which i don't understand why, knowing i recreate index at least once a day. Any idea why is it happening, and how to prevent it?

Previously this has run fine on sql server (on premise, machine with half of those specs) - we are investigating move to elasticsearch due to it's text search capabilities (which we do not use yet)

[warkolm (Mark Walkom)]

How many shards do you have? What sort of query rate and can you share an example aggregation and mapping?

[itmeze (Michal)]

I have one shard per index - index size is ~1GB.

mappings:

{
	"settings": {
		"index" : {
			"number_of_shards" : 1, 
			"refresh_interval" : "10s"
        }
	},
	"mappings": {
		"properties": {
			"Description": {
				"type": "object", 
				"enabled": false
			},
			"TranslatedDescriptions": {
				"type": "object", 
				"enabled": false
			},
			"Photos": {
				"type": "object", 
				"enabled": false
			},
			"AdInfo": {
				"properties": {
					"Featured": {
						"type": "boolean"
					},
					"FeaturedPremium": {
						"type": "boolean"
					}
				}
			},
			"Address": {
				"type": "keyword"
			},
			"ShowAddress": {
				"type": "keyword"
			},
			"GeoInfo": {
				"type": "keyword"
			},
			"AgentRef": {
				"type": "keyword"
			},
			"AirConditioning": {
				"type": "boolean"
			},
			"Bathrooms": {
				"type": "short"
			},
			"Bedrooms": {
				"type": "short"
			},
			"Toilets": {
				"type": "short"
			},
			"Build": {
				"type": "integer"
			},
			"Garage": {
				"type": "boolean"
			},
			"GroupType": {
				"properties": {
					"Id": {
						"type": "short"
					},
					"Term": {
						"type": "keyword"
					}
				}
			},
			"Id": {
				"type": "integer"
			},
			"IsPublic": {
				"type": "boolean"
			},
			"Latitude": {
				"type": "float"
			},
			"Login": {
				"type": "keyword"
			},
			"LoginLower": {
				"type": "keyword"
			},
			"LoginRelevance": {
				"properties": {
					"ForHolidays": {
						"type": "long"
					},
					"ForRent": {
						"type": "long"
					},
					"ForSale": {
						"type": "long"
					}
				}
			},
			"Relevance": {
				"properties": {
					"EN": {
						"type": "float"
					},
					"ES": {
						"type": "float"
					},
					"DE": {
						"type": "float"
					},
					"PL": {
						"type": "float"
					},
					"FR": {
						"type": "float"
					},
					"IT": {
						"type": "float"
					},
					"NL": {
						"type": "float"
					}
				}
			},
			"Longitude": {
				"type": "float"
			},
			"NormalizedPriceInEuro": {
				"type": "integer"
			},
			"DistanceToTown": {
				"type": "float"
			},
			"DistanceToAirport": {
				"type": "float"
			},
			"DistanceToMarina": {
				"type": "float"
			},
			"DistanceToBeach": {
				"type": "float"
			},
			"DistanceToCountry": {
				"type": "float"
			},
			"DistanceToGolf": {
				"type": "float"
			},
			"OfferType": {
				"properties": {
					"Id": {
						"type": "short"
					},
					"Name": {
						"type": "keyword"
					},
					"Term": {
						"type": "keyword"
					}
				}
			},
			"Plot": {
				"type": "integer"
			},
			"PriceHistory": {
				"properties": {
					"Currency": {
						"type": "keyword"
					},
					"DaysAgo": {
						"type": "integer"
					},
					"Price": {
						"type": "integer"
					}
				}
			},
			"PriceInEuro": {
				"type": "integer"
			},
			"PriceInPounds": {
				"type": "integer"
			},
			"Province": {
				"properties": {
				"Active": {
					"type": "boolean"
				},
				"DisplayLocationType": {
					"type": "boolean"
				},
				"Id": {
					"type": "short"
				},
				"IsIsland": {
					"type": "boolean"
				},
				"Labels": {
					"type": "object", 
					"enabled": false
				},
				"RegionId": {
					"type": "short"
				},
				"SameAsRegion": {
					"type": "boolean"
				}
				}
			},
			"Region": {
				"properties": {
				"Active": {
					"type": "boolean"
				},
				"DisplayLocationType": {
					"type": "boolean"
				},
				"Id": {
					"type": "short"
				},
				"Labels": {
					"type": "object", 
					"enabled": false
				},
				"SameAsProvince": {
					"type": "boolean"
				},
				"slug": {
					"type": "keyword"
				}
				}
			},
			"SwimmingPool": {
				"type": "boolean"
			},
			"Town": {
				"properties": {
				"CoastId": {
					"type": "short"
				},
				"DisplayLocationType": {
					"type": "boolean"
				},
				"Id": {
					"type": "short"
				},
				"Labels": {
					"type": "object", 
					"enabled": false
				},
				"Latitude": {
					"type": "float"
				},
				"Longitude": {
					"type": "float"
				},
				"ProvinceId": {
					"type": "short"
				},
				"Location": { 
					"type": "geo_point" 
				},
				"RegionId": {
					"type": "short"
				},
				"SameAsProvince": {
					"type": "boolean"
				},
				"SlugName": {
					"type": "keyword"
				},
				"Type": {
					"type": "keyword"
				}
				}
			},
			"Type": {
				"properties": {
				"English": {
					"type": "keyword"
				},
				"GroupId": {
					"type": "short"
				},
				"Id": {
					"type": "short"
				},
				"SameAsGroup": {
					"type": "boolean"
				}
				"Term": {
					"type": "keyword"
				}
				}
			}
		}
	}
}
`

Aggregations are usually by Town.Id (~5000 distinct values), Province.Id (around 50 distinct values), Region.Id (around 20 different values), SwimmingPool (true/false), Garage (true/false), Type (10 distinct values). Like most of property searches.