Admin option to list all of the indexes?


(David Jensen-2) #1

I'm running a test where I'm creating thousands of indexes each with
~10k documents.

The script bailed on me last night and I want to figure out what index
it left off on. Unfortunately I didn't print out or log enough of the
progress info to get where it left off.

Is there an admin command to list the indexes? I can't seem to find
anything in the docs making me think the answer is no. But maybe it's
hidden in the details of another command.


(David Jensen-2) #2

Answered my own question.

It looks like you can run the "status" command on all the indexes by
doing the following
$ curl -XGET 'http://localhost:9200/_status'

I found it way at the bottom of this page ...
http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/indices/status/

On Aug 26, 12:23 pm, David Jensen djense...@gmail.com wrote:

I'm running a test where I'm creating thousands of indexes each with
~10k documents.

The script bailed on me last night and I want to figure out what index
it left off on. Unfortunately I didn't print out or log enough of the
progress info to get where it left off.

Is there an admin command to list the indexes? I can't seem to find
anything in the docs making me think the answer is no. But maybe it's
hidden in the details of another command.


(Berkay Mollamustafaoglu-2) #3

David,

How many indices were you able to create ?

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Thu, Aug 26, 2010 at 3:27 PM, David Jensen djensen47@gmail.com wrote:

Answered my own question.

It looks like you can run the "status" command on all the indexes by
doing the following
$ curl -XGET 'http://localhost:9200/_status'

I found it way at the bottom of this page ...

http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/indices/status/

On Aug 26, 12:23 pm, David Jensen djense...@gmail.com wrote:

I'm running a test where I'm creating thousands of indexes each with
~10k documents.

The script bailed on me last night and I want to figure out what index
it left off on. Unfortunately I didn't print out or log enough of the
progress info to get where it left off.

Is there an admin command to list the indexes? I can't seem to find
anything in the docs making me think the answer is no. But maybe it's
hidden in the details of another command.


(David Jensen-2) #4

Not as many as I wanted to. My script failed after about 1.4M
documents and about 320 indexes. When I ran status, however, I would
only see the first 35 indexes.

After creating this many indexes, load on the machine went up and the
CPU was pegged close to 100%. I tried to shutdown the servers and
reload form the gateway, this also failed miserably. I'm going to take
a different approach.

On Aug 26, 1:25 pm, Berkay Mollamustafaoglu mber...@gmail.com wrote:

David,

How many indices were you able to create ?

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Thu, Aug 26, 2010 at 3:27 PM, David Jensen djense...@gmail.com wrote:

Answered my own question.

It looks like you can run the "status" command on all the indexes by
doing the following
$ curl -XGET 'http://localhost:9200/_status'

I found it way at the bottom of this page ...

http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/indice...

On Aug 26, 12:23 pm, David Jensen djense...@gmail.com wrote:

I'm running a test where I'm creating thousands of indexes each with
~10k documents.

The script bailed on me last night and I want to figure out what index
it left off on. Unfortunately I didn't print out or log enough of the
progress info to get where it left off.

Is there an admin command to list the indexes? I can't seem to find
anything in the docs making me think the answer is no. But maybe it's
hidden in the details of another command.


(Shay Banon) #5

Which version were you using? In 0.9.0 the index creation was really heavy,
its much more lighter in 0.10. Just a note, each shard is a full Lucene
index, so its not very lightweight. The number of indices you can create
depends on the machine you are running on, the amount of memory allocated to
elasticsearch, and your expectations from the created system (latency,
throughput).

-shay.banon

On Fri, Aug 27, 2010 at 9:44 AM, David Jensen djensen47@gmail.com wrote:

Not as many as I wanted to. My script failed after about 1.4M
documents and about 320 indexes. When I ran status, however, I would
only see the first 35 indexes.

After creating this many indexes, load on the machine went up and the
CPU was pegged close to 100%. I tried to shutdown the servers and
reload form the gateway, this also failed miserably. I'm going to take
a different approach.

On Aug 26, 1:25 pm, Berkay Mollamustafaoglu mber...@gmail.com wrote:

David,

How many indices were you able to create ?

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Thu, Aug 26, 2010 at 3:27 PM, David Jensen djense...@gmail.com
wrote:

Answered my own question.

It looks like you can run the "status" command on all the indexes by
doing the following
$ curl -XGET 'http://localhost:9200/_status'

I found it way at the bottom of this page ...

http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/indice.
..

On Aug 26, 12:23 pm, David Jensen djense...@gmail.com wrote:

I'm running a test where I'm creating thousands of indexes each with
~10k documents.

The script bailed on me last night and I want to figure out what
index

it left off on. Unfortunately I didn't print out or log enough of the
progress info to get where it left off.

Is there an admin command to list the indexes? I can't seem to find
anything in the docs making me think the answer is no. But maybe it's
hidden in the details of another command.


(Andrei) #6

The README file hints that ES can be used for social search by
creating an index for each user and then searching on multiple indices
with different boost levels per index. Is this suggestion practical?
Any serious social product is likely to have 50,000+ users, so what
sort of hardware would be needed to support 50,000 indices?

-Andrei

On Aug 27, 3:24 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Which version were you using? In 0.9.0 the index creation was really heavy,
its much more lighter in 0.10. Just a note, each shard is a full Lucene
index, so its not very lightweight. The number of indices you can create
depends on the machine you are running on, the amount of memory allocated to
elasticsearch, and your expectations from the created system (latency,
throughput).


(Shay Banon) #7

Yea, it does. This is a suggested solution for solving one of the more
complex aspects of social graphs, but you would need serious cluster of
machines so solve that. It really depends on what you are going to run
against such an index. Most users can live with a 1 shard with 1 replica
index, How many shards can one machine sustain and still be operational
depends on what you do against it, very hard to give a number. But, even
with 1000 shards per machine, you will need 50 machines.

If you don't want to solve the social graph problem presented in that README
case, then all your users can be stored in a single index, or maybe several
indices just chunked differently.

-shay.banon

On Fri, Aug 27, 2010 at 9:35 PM, Andrei andrei@zmievski.org wrote:

The README file hints that ES can be used for social search by
creating an index for each user and then searching on multiple indices
with different boost levels per index. Is this suggestion practical?
Any serious social product is likely to have 50,000+ users, so what
sort of hardware would be needed to support 50,000 indices?

-Andrei

On Aug 27, 3:24 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Which version were you using? In 0.9.0 the index creation was really
heavy,
its much more lighter in 0.10. Just a note, each shard is a full Lucene
index, so its not very lightweight. The number of indices you can create
depends on the machine you are running on, the amount of memory allocated
to
elasticsearch, and your expectations from the created system (latency,
throughput).


(system) #8