Why my query matching too slow ? Urgent project need help please

So i made indice like this(via python, language not important just want to see mapping scheme);

```def index_create(es, index_name):
# create an empty index with the defined mapping - no documents added
if not es.indices.exists(index_name):
	# define the mapping
	mapping = {
        "settings": {
           "number_of_shards": 500,
             "number_of_replicas": 0,
             "auto_expand_replicas": False
         },
		"mappings": {
			"properties": {
				"firstquery": {
					"type": "keyword"
				},
				"secondquery": {
					"type": "keyword"
				}
			}
		}
	}

	res = es.indices.create(
		index=index_name,
		body=mapping
	)```

And i did import all my database just like that into this indice.Size around 530gb and 3.7billion entry only firstquery&secondquery.I have only one indices with 500 shards.But searching way too slow like 30-40seconds(for first request), i thought it needs to be way faster with keyword matching but its not.

I am always sending firstquery for searching and expecting second query to match.This is my template

 ```query_body = {
	"query": {
		"match": {
			"firstquery": name,
		}
	},
}```

Can anyone tell me what I am doing wrong ? Because I already tried with mysql or even with tree algorithm & linux grep command results was max 2-5 second.

Additional Information;

  • 32gb ram, 20gb heap locked true
  • x8 core
  • shards 500
  • no replica in indice

Welcome! :smiley: Please don't PM people to ask for assistance. If your problem is important then we do have paid support and consulting options, but most of us spend our own time answering questions here.

You have way, way, way too many shards, I would reduce that to 10 and see if that helps.

1 Like

I already read documents but seems like not possible to change size of shards after everything is done.Is there any way to change shard size without re-write over 500gb json file because it took almost 5 days to write.

Also interesting with that paid support please pm me about details.

Look at the _shrink API.

https://www.elastic.co/subscriptions is the best place regarding support options :slight_smile:

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.