Although I have not been actively trying to figure out how to improve the indexing performance I am now coming across another (related?) issue when including dynamic fields.
For 6.5 million documents with pre-mapped fields it takes about 25 minutes to index them all. If I add in an additional dynamic field the performance slows right down and eventually times out (Sometimes after retrying it works again but after timing out twice in a row my program exits):
...
[Tue Dec 06 21:38:01 UTC 2016] Progress: 4250000 documents indexed
[Tue Dec 06 21:38:11 UTC 2016] Progress: 4255000 documents indexed
IOException occurred, retry once|+
Host [http://localhost:9200] failure|+
[Tue Dec 06 21:38:21 UTC 2016] Progress: 4260000 documents indexed
[Tue Dec 06 21:38:31 UTC 2016] Progress: 4273000 documents indexed
[Tue Dec 06 21:38:41 UTC 2016] Progress: 4289000 documents indexed
[Tue Dec 06 21:38:51 UTC 2016] Progress: 4307000 documents indexed
[Tue Dec 06 21:39:01 UTC 2016] Progress: 4326000 documents indexed
[Tue Dec 06 21:39:11 UTC 2016] Progress: 4339000 documents indexed
IOException occurred, retry once|+
Host [http://localhost:9200] failure|+
[Tue Dec 06 21:39:21 UTC 2016] Progress: 4339000 documents indexed
Host [http://localhost:9200] failure|+
[Tue Dec 06 21:39:31 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:39:41 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:39:51 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:40:01 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:40:11 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:40:21 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:40:31 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:40:41 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:40:51 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:41:01 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:41:11 UTC 2016] Progress: 4339000 documents indexed
[Tue Dec 06 21:41:21 UTC 2016] Progress: 4339000 documents indexed
Problem running indexer
java.io.IOException: listener timeout after waiting for [10000] ms
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:616)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:184)
at
The dynamic field is pretty horrible (it is deeply nested JSON) and to get it to run as far as above I need to change the following defaults:
"mapping.depth.limit" : "100", "mapping.total_fields.limit" : "50000"
Although the client receives these timeouts there is no useful log on the server side e.g. Nothing saying longer GC etc..
It may be that using such deeply nested and variable JSON is not feasible but it would be good to understand why this is occuring.