We finally finished re-indexing everything we could. Below is the final shard count which has been reduced from 24+k.
{
"cluster_name": "elasticsearch",
"status": "green",
"timed_out": false,
"number_of_nodes": 52,
"number_of_data_nodes": 43,
"active_primary_shards": 1777,
"active_shards": 3554,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100.0
}
Unfortunately our indexing performance is still not good. Here is the output from one of the last re-indexing tasks I performed:
{
completed: true,
task: {
node: 'r39uxV_XSRy7zYDQY7BTTg',
id: 28904733,
type: 'transport',
action: 'indices:data/write/reindex',
status: {
total: 287796,
updated: 261406,
created: 26390,
deleted: 0,
batches: 288,
version_conflicts: 0,
noops: 0,
retries: [Object],
throttled_millis: 0,
requests_per_second: -1,
throttled_until_millis: 0
},
description: 'reindex from [88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-11-es6apparatusfireincident, 88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-12-es6apparatusfireincident, 88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-02-es6apparatusfireincident-m3, 88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-08-es6apparatusfireincident, 88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-06-es6apparatusfireincident, 88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-10-es6apparatusfireincident, 88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-07-es6apparatusfireincident, 88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-03-es6apparatusfireincident, 88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-05-es6apparatusfireincident, 88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-09-es6apparatusfireincident, 88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-04-es6apparatusfireincident, 88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-01-es6apparatusfireincident-m3] to [88403-tn-memphis_division_of_fire_services-apparatus-fire-incident-2020-es6apparatusfireincident]',
start_time_in_millis: 1609888634911,
running_time_in_nanos: 1189569144096,
cancellable: true,
headers: {}
},
response: {
took: 1189564,
timed_out: false,
total: 287796,
updated: 261406,
created: 26390,
deleted: 0,
batches: 288,
version_conflicts: 0,
noops: 0,
retries: { bulk: 0, search: 0 },
throttled_millis: 0,
requests_per_second: -1,
throttled_until_millis: 0,
failures: []
}
}
You can see it took almost 20 minutes to re-index approx 287k docs. We are looking even deeper into our cluster configuration. I will continue to post updates.