thks anyway , it seems so difficult to solve
Although I have not been actively trying to figure out how to improve the indexing performance I am now coming across another (related?) issue when including dynamic fields.
For 6.5 million documents with pre-mapped fields it takes about 25 minutes to index them all. If I add in an additional dynamic field the performance slows right down and eventually times out (Sometimes after retrying it works again but after timing out twice in a row my program exits):
...
[Tue Dec 06 21:38:01 UTC 2016] Progress: 4250000 documents indexed
[Tue Dec 06 21:38:11 UTC 2016] Progress: 4255000 documents indexed
IOException occurred, retry once|+
Host [http://localhost:9200] failure|+
[Tue Dec 06 21:38:21 UTC 2016] Progress: 4260000 documents indexed
[Tue Dec 06 21:38:31 UTC 2016] Progress: 4273000 documents indexed
[Tue Dec 06 21:38:41 UTC 2016] Progress: 4289000 documents indexed
[Tue Dec 06 21:38:51 UTC 2016] Progress: 4307000 documents indexed
[Tue Dec 06 21:39:01 UTC 2016] Progress: 4326000 documents indexed
[Tue Dec 06 21:39:11 UTC 2016] Progress: 4339000 documents indexed
IOException occurred, retry once|+
Host [http://localhost:9200] failure|+
[Tue Dec 06 21:39:21 UTC 2016] Progress: 4339000 documents indexed
Host [http://localhost:9200] failure|+
[Tue Dec 06 21:39:31 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:39:41 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:39:51 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:40:01 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:40:11 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:40:21 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:40:31 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:40:41 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:40:51 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:41:01 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:41:11 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:41:21 UTC 2016] Progress: 4339000 documents indexed
Problem running indexer
java.io.IOException: listener timeout after waiting for [10000] ms
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:616)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:184)
at
The dynamic field is pretty horrible (it is deeply nested JSON) and to get it to run as far as above I need to change the following defaults:
"mapping.depth.limit" : "100", "mapping.total_fields.limit" : "50000"
Although the client receives these timeouts there is no useful log on the server side e.g. Nothing saying longer GC etc..
It may be that using such deeply nested and variable JSON is not feasible but it would be good to understand why this is occuring.
Every time a new field is encountered the mappings will get updated if you are using dynamic mapping. This will cause the cluster state to get updated, which is a single-threaded operation, and will cause a slowdown and possibly also mapping explosion.
It's not just slow, it's awful.
I'm basically trying to migrate from old cluster (2.4) to new one (5.0) and that's a complete and total disaster. 5.x isn't even close production ready. It's full of bugs.
I'm using elasticsearch-dump (becuase reindex API just doesn't work, already logged a ticket), but it stalls on 50k document -- and there's no other tool to do the migrations. God, I miss mysqldump.
Seeing nightmare indexing performance and random OOMs. Why they didn't document the settings needed for BULK migrations is beyond my understanding. Cluster is unused, it's an initial set-up, I need to get IN the data. It's very hard, sadly. The thing is not production-ready at all.
1 Like
did you migrate to other machines ?
We also performed some tests comparing 2.4 with 5.0, and we got:
Improvements:
- Searches are faster and also not much
And what is bad: - Indexing throughput almost unchanged
- Heap consumption increased in ~2x times
- Although small, but increase of data size on disk
But this announcing article says otherwise:
Depending on your use case, you are likely to see somewhere between 25% - 80% improvement to indexing throughput.
Our case: own doc id, spinnin disk. But we did not get significant changes in indexing throughput.
In our benchmark, Points are 36% faster at query time, 71% faster at index time and used 66% less disk and 85% less memory
Almost all of our data are numbers. We dont see these improvements in heap and disk usage, however.
oh , mine
can you tell me how do you compare the consumption ? I want to confirm it
Hi! I think it use case specific. In described case we store and analize NAT logs, so we have a lot of IPs, counters/IDs (type: Long) and Timestamps. In 5.0.x there are new in memory structure points_memory_in_bytes so all this date migrate there and terms_in_memory decreased a bit.
I cannot provide any numbers for now, as we drop the test environment, but @ivank will provide it in near future as we will test 5.1.x again,
I can share numbers for different index type (most fields are strings with not_analyzed):
90 fields: 11 integer, 9 boolean, 9 double, 21 long and others strings (in 5.x keywords and analized text) like:
"type": "text",
"norms": false,
"analyzer": "lowercase_analyzer",
"index_options": "docs"
In 2.1.2
curl -XPOST 'http://localhost:9200/_forcemerge?max_num_segments=1
curl -XGET "http://localhost:9200/_stats?pretty"
...
"segments": {
"count": 8,
"memory_in_bytes": 262 754 683,
"terms_memory_in_bytes": 259 898 547,
"stored_fields_memory_in_bytes": 2 854 888,
"term_vectors_memory_in_bytes": 0,
"norms_memory_in_bytes": 512,
"doc_values_memory_in_bytes": 736,
"index_writer_memory_in_bytes": 0,
"index_writer_max_memory_in_bytes": 4096000,
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0
}
test-2016.01.01 - shards: 2 * 1 | docs: 15,001,000 | size: 8.18GB
test-2016.01.02 - shards: 2 * 1 | docs: 15,009,000 | size: 8.20GB
test-2016.01.03 - shards: 2 * 1 | docs: 15,000,000 | size: 8.19GB
test-2016.01.04 - shards: 2 * 1 | docs: 15,000,000 | size: 8.19GB
In 5.1.1
curl -XPOST 'http://localhost:9200/_forcemerge?max_num_segments=1'
"segments" : {
"count" : 8,
"memory_in_bytes" : 245 671 914,
"terms_memory_in_bytes" : 223 816 738,
"stored_fields_memory_in_bytes" : 2 828 344,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 2048,
"points_memory_in_bytes" : 19 022 896,
"doc_values_memory_in_bytes" : 1888,
"index_writer_memory_in_bytes" : 0,
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 0,
"max_unsafe_auto_id_timestamp" : -1,
"file_sizes" : { }
}
shards: 2 * 1 | docs: 15,050,562 | size: 9.04GB
shards: 2 * 1 | docs: 15,018,914 | size: 9.02GB
shards: 2 * 1 | docs: 15,009,907 | size: 9.02GB
shards: 2 * 1 | docs: 15,019,822 | size: 9.02GBHi again,
I found time to do own tests, and I want to share with you.
First, I have generated 12M documents (~100GB), so test data is equal for 2.x/5.x clusters (even doc ids are equal). Then I have indexed them all. Cluster consist of 2 data nodes and 1 master node. Index has 2 primary and 2 replica shards.
| ES 2.3.4 | ES 5.1.2
--------------------------------------
Indexing time | 42.9m | 33.1m
Index size | 9.42gb | 6.74gb
Heap used | 58mb | 300mb
Segment count | 106 | 89
Segment size | 63.9mb | 10.5mb
terms | 62.5mb | 4.7mb
stored fields | 1.3mb | 1.3mb
points | - | 4.3mb
doc_values | 9.5kb | 7.9kb
After merging to single segmet per shard:
| ES 2.3.4 | ES 5.1.2
--------------------------------------
Index size | 9.22gb | 6.6gb
Heap used | 48mb | 156mb
Segment size | 57.9mb | 11.1mb
terms | 56.6mb | 5.4mb
stored fields | 1.3mb | 1.3mb
points | - | 4.2mb
doc_values | 368b | 368b
In general, it is good numbers. But heap usage looks not so good... and strange when segments merged to 1 on ES 5. Why ES 5 consumes so much heap?
1 Like