This is my first e-mail, so, if this problem is already explained, i'm
sorry, couldn't find out where it is.
I'm out of ideas. This is my question:
I had an Elasticsearch up and running with 1 replica, 5 shards, 1 master
(data false) and 10 slaves, and every index configured by day (from
logstash). Since we changed to a hourly index, after 2 weeks and a needed a
maintenance reboot, Elasticsearch wasn't able to start properly. It started
assigning unassigned shards and a lot of timeouts came to happen.
After 5 days trying to recover, we decided to change the configuration of
our cluster to 1 master (data false), 10 salves and 1 shard 2 replicas
indexes, from scratch, without any old index.
My task now is to reindex those lost indexes. This is my problem:
I have 10 backup files (up to 400gb each) and i'm looking for ways to
reindex those indexes (little by little).
Should i copy those indexes folder to the new cluster folder?
I don't need to change to a daily shard, i just need Elasticsearch
to assign those indexes.
Is there any way i'm able to differentiate replica folders from shards
folders?
We're using Elasticsearch 1.4.4 and each Elasticsearch is in an 8-core,
16gb ram dedicated machine.
1 shard per index doesn't make a lot of sense unless you have very small
amounts of data, You'd be better off going back to the default as you are
solving the wrong problem there.
What are these backup file you mention, how did you get them out of ES?
This is my first e-mail, so, if this problem is already explained, i'm
sorry, couldn't find out where it is.
I'm out of ideas. This is my question:
I had an Elasticsearch up and running with 1 replica, 5 shards, 1 master
(data false) and 10 slaves, and every index configured by day (from
logstash). Since we changed to a hourly index, after 2 weeks and a needed a
maintenance reboot, Elasticsearch wasn't able to start properly. It started
assigning unassigned shards and a lot of timeouts came to happen.
After 5 days trying to recover, we decided to change the configuration of
our cluster to 1 master (data false), 10 salves and 1 shard 2 replicas
indexes, from scratch, without any old index.
My task now is to reindex those lost indexes. This is my problem:
I have 10 backup files (up to 400gb each) and i'm looking for ways to
reindex those indexes (little by little).
Should i copy those indexes folder to the new cluster folder?
I don't need to change to a daily shard, i just need
Elasticsearch to assign those indexes.
Is there any way i'm able to differentiate replica folders from
shards folders?
We're using Elasticsearch 1.4.4 and each Elasticsearch is in an 8-core,
16gb ram dedicated machine.
I'm sorry for the long delay it took to answer.
Every index is a folder inside the data folder. I just simply compressed
those folders and sent to S3.
But, now, we just found an "answer":
we built (at another dc) another ES cluster and we've put those
folders inside the data directory
this is the part that i didn't participate:
we had a Logstash querying ES and outputting to our ES cluster
1 shard per index doesn't make a lot of sense unless you have very small
amounts of data, You'd be better off going back to the default as you are
solving the wrong problem there.
What are these backup file you mention, how did you get them out of ES?
This is my first e-mail, so, if this problem is already explained, i'm
sorry, couldn't find out where it is.
I'm out of ideas. This is my question:
I had an Elasticsearch up and running with 1 replica, 5 shards, 1 master
(data false) and 10 slaves, and every index configured by day (from
logstash). Since we changed to a hourly index, after 2 weeks and a needed a
maintenance reboot, Elasticsearch wasn't able to start properly. It started
assigning unassigned shards and a lot of timeouts came to happen.
After 5 days trying to recover, we decided to change the configuration of
our cluster to 1 master (data false), 10 salves and 1 shard 2 replicas
indexes, from scratch, without any old index.
My task now is to reindex those lost indexes. This is my problem:
I have 10 backup files (up to 400gb each) and i'm looking for ways to
reindex those indexes (little by little).
Should i copy those indexes folder to the new cluster folder?
I don't need to change to a daily shard, i just need
Elasticsearch to assign those indexes.
Is there any way i'm able to differentiate replica folders from
shards folders?
We're using Elasticsearch 1.4.4 and each Elasticsearch is in an 8-core,
16gb ram dedicated machine.
I'm sorry for the long delay it took to answer.
Every index is a folder inside the data folder. I just simply compressed
those folders and sent to S3.
But, now, we just found an "answer":
we built (at another dc) another ES cluster and we've put those
folders inside the data directory
this is the part that i didn't participate:
we had a Logstash querying ES and outputting to our ES cluster
1 shard per index doesn't make a lot of sense unless you have very small
amounts of data, You'd be better off going back to the default as you are
solving the wrong problem there.
What are these backup file you mention, how did you get them out of ES?
This is my first e-mail, so, if this problem is already explained, i'm
sorry, couldn't find out where it is.
I'm out of ideas. This is my question:
I had an Elasticsearch up and running with 1 replica, 5 shards, 1 master
(data false) and 10 slaves, and every index configured by day (from
logstash). Since we changed to a hourly index, after 2 weeks and a needed a
maintenance reboot, Elasticsearch wasn't able to start properly. It started
assigning unassigned shards and a lot of timeouts came to happen.
After 5 days trying to recover, we decided to change the configuration
of our cluster to 1 master (data false), 10 salves and 1 shard 2 replicas
indexes, from scratch, without any old index.
My task now is to reindex those lost indexes. This is my problem:
I have 10 backup files (up to 400gb each) and i'm looking for ways to
reindex those indexes (little by little).
Should i copy those indexes folder to the new cluster folder?
I don't need to change to a daily shard, i just need
Elasticsearch to assign those indexes.
Is there any way i'm able to differentiate replica folders from
shards folders?
We're using Elasticsearch 1.4.4 and each Elasticsearch is in an 8-core,
16gb ram dedicated machine.
On Thursday, May 14, 2015 at 5:50:33 PM UTC-6, Frederico Barnard wrote:
I'm sorry for the long delay it took to answer.
Every index is a folder inside the data folder. I just simply compressed
those folders and sent to S3.
But, now, we just found an "answer":
we built (at another dc) another ES cluster and we've put those
folders inside the data directory
this is the part that i didn't participate:
we had a Logstash querying ES and outputting to our ES cluster
That's our answer to what we were looking for
Att
Frederico Ferreira
(21) 98714-1445
2015-04-27 18:28 GMT-03:00 Mark Walkom <markw...@gmail.com <javascript:>>:
1 shard per index doesn't make a lot of sense unless you have very small
amounts of data, You'd be better off going back to the default as you are
solving the wrong problem there.
What are these backup file you mention, how did you get them out of ES?
On 27 April 2015 at 21:50, Frederico Ferreira <frede...@gmail.com
<javascript:>> wrote:
This is my first e-mail, so, if this problem is already explained, i'm
sorry, couldn't find out where it is.
I'm out of ideas. This is my question:
I had an Elasticsearch up and running with 1 replica, 5 shards, 1 master
(data false) and 10 slaves, and every index configured by day (from
logstash). Since we changed to a hourly index, after 2 weeks and a needed a
maintenance reboot, Elasticsearch wasn't able to start properly. It started
assigning unassigned shards and a lot of timeouts came to happen.
After 5 days trying to recover, we decided to change the configuration
of our cluster to 1 master (data false), 10 salves and 1 shard 2 replicas
indexes, from scratch, without any old index.
My task now is to reindex those lost indexes. This is my problem:
I have 10 backup files (up to 400gb each) and i'm looking for ways to
reindex those indexes (little by little).
Should i copy those indexes folder to the new cluster folder?
I don't need to change to a daily shard, i just need
Elasticsearch to assign those indexes.
Is there any way i'm able to differentiate replica folders from
shards folders?
We're using Elasticsearch 1.4.4 and each Elasticsearch is in an 8-core,
16gb ram dedicated machine.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.