Path.data v0.18.1


(Gustavo Maia) #1

Hello,
I'm using version 0.18.1 and configuring path.data as follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I was a little
surprised the lengths of each path were not similar. Below the size of
each path found.

gustavo @ gustavo-linux: ~ $ ls-lh
/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh
/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh
/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh
/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger than the
other paths. Is that correct? I thought that the sizes of the path
would be the same or very close.


(Shay Banon) #2

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia gustavobbmaia@gmail.comwrote:

Hello,
I'm using version 0.18.1 and configuring path.data as follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I was a little
surprised the lengths of each path were not similar. Below the size of
each path found.

gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger than the
other paths. Is that correct? I thought that the sizes of the path
would be the same or very close.


(Gustavo Maia) #3

Is the same Driver.

2011/10/27 Shay Banon kimchy@gmail.com

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia gustavobbmaia@gmail.comwrote:

Hello,
I'm using version 0.18.1 and configuring path.data as follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I was a little
surprised the lengths of each path were not similar. Below the size of
each path found.

gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger than the
other paths. Is that correct? I thought that the sizes of the path
would be the same or very close.

--
Gustavo Maia


(Gustavo Maia) #4

I'm using the same drive.

2011/10/27 Shay Banon kimchy@gmail.com

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia gustavobbmaia@gmail.comwrote:

Hello,
I'm using version 0.18.1 and configuring path.data as follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I was a little
surprised the lengths of each path were not similar. Below the size of
each path found.

gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger than the
other paths. Is that correct? I thought that the sizes of the path
would be the same or very close.

--
Gustavo Maia


(Shay Banon) #5

The check for the size of where to place the data is based on the free space
available per "drive", so having multiple data locations on the same drive
makes little sense.

On Fri, Oct 28, 2011 at 1:28 AM, Gustavo Maia gustavobbmaia@gmail.comwrote:

Is the same Driver.

2011/10/27 Shay Banon kimchy@gmail.com

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia gustavobbmaia@gmail.comwrote:

Hello,
I'm using version 0.18.1 and configuring path.data as follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I was a little
surprised the lengths of each path were not similar. Below the size of
each path found.

gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger than the
other paths. Is that correct? I thought that the sizes of the path
would be the same or very close.

--
Gustavo Maia


(Gustavo Maia) #6

OK. If I have 4 drives with the same size (150GB each drive). When
I index 1million documents all drives will have the same size?

I'm doing tests on my machine and then indexing at amazon.

2011/10/27 Shay Banon kimchy@gmail.com

The check for the size of where to place the data is based on the free
space available per "drive", so having multiple data locations on the same
drive makes little sense.

On Fri, Oct 28, 2011 at 1:28 AM, Gustavo Maia gustavobbmaia@gmail.com
wrote:

Is the same Driver.

2011/10/27 Shay Banon kimchy@gmail.com

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia gustavobbmaia@gmail.com
wrote:

Hello,
I'm using version 0.18.1 and configuring path.data as follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I was a little
surprised the lengths of each path were not similar. Below the size of
each path found.

gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger than the
other paths. Is that correct? I thought that the sizes of the path
would be the same or very close.

--
Gustavo Maia

--
Gustavo Maia


(Gustavo Maia) #7

The following test was done using the 0.18.0 version of the ES.
Was set up multiple path, in multiple drives.
Below is the sizes of the data path.

/dev/sdb 414G 199M 393G 1% /mnt
2)
/dev/sdc 414G 199M 393G 1% /media/dev1
3)
/dev/sdd 414G 15G 378G 4% /media/dev2
4)
/dev/sde 414G 199M 393G 1% /media/dev3

This distribution of indices that correct or is it a bug that was fixed in
version0.18.1?

Earlier I was talking about a test with the version 0.18.1, using the same
drive with multiple path.

2011/10/27 Gustavo Maia gustavobbmaia@gmail.com

OK. If I have 4 drives with the same size (150GB each drive). When
I index 1million documents all drives will have the same size?

I'm doing tests on my machine and then indexing at amazon.

2011/10/27 Shay Banon kimchy@gmail.com

The check for the size of where to place the data is based on the free
space available per "drive", so having multiple data locations on the same
drive makes little sense.

On Fri, Oct 28, 2011 at 1:28 AM, Gustavo Maia gustavobbmaia@gmail.com
wrote:

Is the same Driver.

2011/10/27 Shay Banon kimchy@gmail.com

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia <
gustavobbmaia@gmail.com> wrote:

Hello,
I'm using version 0.18.1 and configuring path.data as follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I was a
little

surprised the lengths of each path were not similar. Below the size
of

each path found.

gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger than the
other paths. Is that correct? I thought that the sizes of the path
would be the same or very close.

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia


(Shay Banon) #8

Strange..., nothing really fixed. I added some logging to 0.18 branch that
might help, can you try with a custom build of it, and set: env to TRACE?

On Fri, Oct 28, 2011 at 4:48 AM, Gustavo Maia gustavobbmaia@gmail.comwrote:

The following test was done using the 0.18.0 version of the ES.
Was set up multiple path, in multiple drives.
Below is the sizes of the data path.

/dev/sdb 414G 199M 393G 1% /mnt
2)
/dev/sdc 414G 199M 393G 1% /media/dev1
3)
/dev/sdd 414G 15G 378G 4% /media/dev2
4)
/dev/sde 414G 199M 393G 1% /media/dev3

This distribution of indices that correct or is it a bug that was fixed in
version0.18.1?

Earlier I was talking about a test with the version 0.18.1, using the same
drive with multiple path.

2011/10/27 Gustavo Maia gustavobbmaia@gmail.com

OK. If I have 4 drives with the same size (150GB each drive). When
I index 1million documents all drives will have the same size?

I'm doing tests on my machine and then indexing at amazon.

2011/10/27 Shay Banon kimchy@gmail.com

The check for the size of where to place the data is based on the free
space available per "drive", so having multiple data locations on the same
drive makes little sense.

On Fri, Oct 28, 2011 at 1:28 AM, Gustavo Maia gustavobbmaia@gmail.com
wrote:

Is the same Driver.

2011/10/27 Shay Banon kimchy@gmail.com

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia <
gustavobbmaia@gmail.com> wrote:

Hello,
I'm using version 0.18.1 and configuring path.data as follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I was a
little

surprised the lengths of each path were not similar. Below the size
of

each path found.

gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger than
the

other paths. Is that correct? I thought that the sizes of the path
would be the same or very close.

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia


(Gustavo Maia) #9

hi, Shay

Yes i can do it.

gateway: DEBUG
org.apache: WARN
discovery: TRACE

Only put it in the config/logging.yml ??

2011/10/31 Shay Banon kimchy@gmail.com

Strange..., nothing really fixed. I added some logging to 0.18 branch that might help, can you try with a custom build of it, and set: env to TRACE?

On Fri, Oct 28, 2011 at 4:48 AM, Gustavo Maia gustavobbmaia@gmail.com wrote:

The following test was done using the 0.18.0 version of the ES.
Was set up multiple path, in multiple drives.
Below is the sizes of the data path.

/dev/sdb 414G 199M 393G 1% /mnt
2)
/dev/sdc 414G 199M 393G 1% /media/dev1
3)
/dev/sdd 414G 15G 378G 4% /media/dev2
4)
/dev/sde 414G 199M 393G 1% /media/dev3

This distribution of indices that correct or is it a bug that was fixed in version0.18.1?

Earlier I was talking about a test with the version 0.18.1, using the same drive with multiple path.

2011/10/27 Gustavo Maia gustavobbmaia@gmail.com

OK. If I have 4 drives with the same size (150GB each drive). When I index 1million documents all drives will have the same size?

I'm doing tests on my machine and then indexing at amazon.

2011/10/27 Shay Banon kimchy@gmail.com

The check for the size of where to place the data is based on the free space available per "drive", so having multiple data locations on the same drive makes little sense.

On Fri, Oct 28, 2011 at 1:28 AM, Gustavo Maia gustavobbmaia@gmail.com wrote:

Is the same Driver.

2011/10/27 Shay Banon kimchy@gmail.com

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia gustavobbmaia@gmail.com wrote:

Hello,
I'm using version 0.18.1 and configuring path.data as follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I was a little
surprised the lengths of each path were not similar. Below the size of
each path found.

gustavo @ gustavo-linux: ~ $ ls-lh
/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh
/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh
/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh
/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger than the
other paths. Is that correct? I thought that the sizes of the path
would be the same or very close.

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia


(Shay Banon) #10

yes, but you will need to build a version of elasticsearch... . Should be
simple, download / clone the 0.18 branch, and run "gradlew release".

On Mon, Oct 31, 2011 at 9:53 PM, Gustavo Maia gustavobbmaia@gmail.comwrote:

hi, Shay

Yes i can do it.

gateway: DEBUG
org.apache: WARN
discovery: TRACE

Only put it in the config/logging.yml ??

2011/10/31 Shay Banon kimchy@gmail.com

Strange..., nothing really fixed. I added some logging to 0.18 branch
that might help, can you try with a custom build of it, and set: env to
TRACE?

On Fri, Oct 28, 2011 at 4:48 AM, Gustavo Maia gustavobbmaia@gmail.com
wrote:

The following test was done using the 0.18.0 version of the ES.
Was set up multiple path, in multiple drives.
Below is the sizes of the data path.

/dev/sdb 414G 199M 393G 1% /mnt
2)
/dev/sdc 414G 199M 393G 1% /media/dev1
3)
/dev/sdd 414G 15G 378G 4% /media/dev2
4)
/dev/sde 414G 199M 393G 1% /media/dev3

This distribution of indices that correct or is it a bug that was fixed
in version0.18.1?

Earlier I was talking about a test with the version 0.18.1, using the
same drive with multiple path.

2011/10/27 Gustavo Maia gustavobbmaia@gmail.com

OK. If I have 4 drives with the same size (150GB each drive). When
I index 1million documents all drives will have the same size?

I'm doing tests on my machine and then indexing at amazon.

2011/10/27 Shay Banon kimchy@gmail.com

The check for the size of where to place the data is based on the
free space available per "drive", so having multiple data locations on the
same drive makes little sense.

On Fri, Oct 28, 2011 at 1:28 AM, Gustavo Maia <
gustavobbmaia@gmail.com> wrote:

Is the same Driver.

2011/10/27 Shay Banon kimchy@gmail.com

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia <
gustavobbmaia@gmail.com> wrote:

Hello,
I'm using version 0.18.1 and configuring path.data as follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I was a
little

surprised the lengths of each path were not similar. Below the
size of

each path found.

gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger
than the

other paths. Is that correct? I thought that the sizes of the
path

would be the same or very close.

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia


(Gustavo Maia) #11

Please check, if it is correct.

My logging.yml file:

rootLogger: INFO, console, file
logger:

log action execution errors for easier debugging

action: DEBUG

reduce the logging for aws, too much is logged under the default INFO

com.amazonaws: WARN

gateway

gateway: DEBUG
index.gateway: DEBUG

peer shard recovery

indices.recovery: DEBUG

discovery

discovery: TRACE

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

Below my log, but i only see log about save in the s3.

[2011-11-01 18:43:51,141][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [335ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [42ms]
translog : id [1319482375161], number_of_operations [649], took [72ms]
[2011-11-01 18:43:59,124][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:43:59,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [591ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [57ms]
translog : id [1319482375057], number_of_operations [555], took [286ms]
[2011-11-01 18:44:01,141][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:01,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [574ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [51ms]
translog : id [1319482375161], number_of_operations [517], took [95ms]
[2011-11-01 18:44:09,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:10,293][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [577ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [49ms]
translog : id [1319482375057], number_of_operations [344], took [74ms]
[2011-11-01 18:44:11,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:12,039][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [323ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [52ms]
translog : id [1319482375161], number_of_operations [293], took [48ms]
[2011-11-01 18:44:20,294][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:20,865][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [571ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [276ms]
translog : id [1319482375057], number_of_operations [524], took [117ms]
[2011-11-01 18:44:22,039][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:22,354][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [313ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [32ms]
translog : id [1319482375161], number_of_operations [610], took [76ms]
[2011-11-01 18:44:30,865][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:31,204][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [338ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [38ms]
translog : id [1319482375057], number_of_operations [655], took [109ms]
[2011-11-01 18:44:32,354][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:35,822][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [3.4s]
index : version [1319483349493], number_of_files [98] with
total_size [5.8mb], took [1.3s]
translog : id [1319482375162], number_of_operations [49], took [61ms]
[2011-11-01 18:44:41,205][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:41,694][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [489ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [45ms]
translog : id [1319482375057], number_of_operations [421], took [112ms]
[2011-11-01 18:44:45,822][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:46,438][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [616ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [117ms]
translog : id [1319482375162], number_of_operations [500], took [63ms]
[2011-11-01 18:44:51,695][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:53,314][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [1.6s]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [38ms]
translog : id [1319482375057], number_of_operations [413], took [64ms]
[2011-11-01 18:44:56,438][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:56,812][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [373ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [41ms]
translog : id [1319482375162], number_of_operations [551], took [94ms]
[2011-11-01 18:45:03,315][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:06,813][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:07,186][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [373ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [42ms]
translog : id [1319482375162], number_of_operations [450], took [74ms]
[2011-11-01 18:45:11,812][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [8.4s]
index : version [1319483347731], number_of_files [114] with
total_size [17.9mb], took [2s]
translog : id [1319482375058], number_of_operations [284], took [90ms]
[2011-11-01 18:45:17,186][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:17,637][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [450ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [44ms]
translog : id [1319482375162], number_of_operations [458], took [158ms]
[2011-11-01 18:45:21,813][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:22,795][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [982ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [48ms]
translog : id [1319482375058], number_of_operations [743], took [90ms]
[2011-11-01 18:45:27,638][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:28,053][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [415ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [76ms]
translog : id [1319482375162], number_of_operations [392], took [98ms]
[2011-11-01 18:45:32,796][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:33,144][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [348ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [39ms]
translog : id [1319482375058], number_of_operations [510], took [91ms]
[2011-11-01 18:45:38,054][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:39,410][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [1.3s]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [42ms]
translog : id [1319482375162], number_of_operations [564], took [88ms]
[2011-11-01 18:45:43,145][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:43,666][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [521ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [59ms]
translog : id [1319482375058], number_of_operations [568], took [83ms]
[2011-11-01 18:45:49,410][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:50,025][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [615ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [55ms]
translog : id [1319482375162], number_of_operations [587], took [308ms]
[2011-11-01 18:45:53,666][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:54,098][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [431ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [41ms]
translog : id [1319482375058], number_of_operations [545], took [88ms]
[2011-11-01 18:46:00,025][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:46:00,535][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [509ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [85ms]
translog : id [1319482375162], number_of_operations [498], took [81ms]
[2011-11-01 18:46:04,098][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:46:04,424][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [325ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [38ms]
translog : id [1319482375058], number_of_operations [379], took [83ms]
[2011-11-01 18:46:10,535][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:46:11,168][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [632ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [40ms]
translog : id [1319482375162], number_of_operations [302], took [188ms]
[2011-11-01 18:46:14,424][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:46:15,062][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [637ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [139ms]
translog : id [1319482375058], number_of_operations [399], took [81ms]

2011/11/1 Shay Banon kimchy@gmail.com:

yes, but you will need to build a version of elasticsearch... . Should be
simple, download / clone the 0.18 branch, and run "gradlew release".

On Mon, Oct 31, 2011 at 9:53 PM, Gustavo Maia gustavobbmaia@gmail.com
wrote:

hi, Shay

Yes i can do it.

gateway: DEBUG
org.apache: WARN
discovery: TRACE

Only put it in the config/logging.yml ??

2011/10/31 Shay Banon kimchy@gmail.com

Strange..., nothing really fixed. I added some logging to 0.18 branch
that might help, can you try with a custom build of it, and set: env to
TRACE?

On Fri, Oct 28, 2011 at 4:48 AM, Gustavo Maia gustavobbmaia@gmail.com
wrote:

The following test was done using the 0.18.0 version of the ES.
Was set up multiple path, in multiple drives.
Below is the sizes of the data path.

/dev/sdb 414G 199M 393G 1% /mnt
2)
/dev/sdc 414G 199M 393G 1% /media/dev1
3)
/dev/sdd 414G 15G 378G 4% /media/dev2
4)
/dev/sde 414G 199M 393G 1% /media/dev3

This distribution of indices that correct or is it a bug that was fixed
in version0.18.1?

Earlier I was talking about a test with the version 0.18.1, using the
same drive with multiple path.

2011/10/27 Gustavo Maia gustavobbmaia@gmail.com

OK. If I have 4 drives with the same size (150GB each drive). When
I index 1million documents all drives will have the same size?

I'm doing tests on my machine and then indexing at amazon.

2011/10/27 Shay Banon kimchy@gmail.com

The check for the size of where to place the data is based on the
free space available per "drive", so having multiple data locations on the
same drive makes little sense.

On Fri, Oct 28, 2011 at 1:28 AM, Gustavo Maia
gustavobbmaia@gmail.com wrote:

Is the same Driver.

2011/10/27 Shay Banon kimchy@gmail.com

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia
gustavobbmaia@gmail.com wrote:

Hello,
I'm using version 0.18.1 and configuring path.data as follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I was a
little
surprised the lengths of each path were not similar. Below the
size of
each path found.

gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger
than the
other paths. Is that correct? I thought that the sizes of the
path
would be the same or very close.

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia


(Shay Banon) #12

CAn you set in the logging.yml the "env" component to TRACE, and leave all
the rest with teh default values, and then gist hte log output?

On Tue, Nov 1, 2011 at 10:48 PM, Gustavo Maia gustavobbmaia@gmail.comwrote:

Please check, if it is correct.

My logging.yml file:

rootLogger: INFO, console, file
logger:

log action execution errors for easier debugging

action: DEBUG

reduce the logging for aws, too much is logged under the default INFO

com.amazonaws: WARN

gateway

gateway: DEBUG
index.gateway: DEBUG

peer shard recovery

indices.recovery: DEBUG

discovery

discovery: TRACE

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

Below my log, but i only see log about save in the s3.

[2011-11-01 18:43:51,141][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [335ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [42ms]
translog : id [1319482375161], number_of_operations [649], took
[72ms]
[2011-11-01 18:43:59,124][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:43:59,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [591ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [57ms]
translog : id [1319482375057], number_of_operations [555], took
[286ms]
[2011-11-01 18:44:01,141][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:01,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [574ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [51ms]
translog : id [1319482375161], number_of_operations [517], took
[95ms]
[2011-11-01 18:44:09,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:10,293][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [577ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [49ms]
translog : id [1319482375057], number_of_operations [344], took
[74ms]
[2011-11-01 18:44:11,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:12,039][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [323ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [52ms]
translog : id [1319482375161], number_of_operations [293], took
[48ms]
[2011-11-01 18:44:20,294][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:20,865][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [571ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [276ms]
translog : id [1319482375057], number_of_operations [524], took
[117ms]
[2011-11-01 18:44:22,039][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:22,354][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [313ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [32ms]
translog : id [1319482375161], number_of_operations [610], took
[76ms]
[2011-11-01 18:44:30,865][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:31,204][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [338ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [38ms]
translog : id [1319482375057], number_of_operations [655], took
[109ms]
[2011-11-01 18:44:32,354][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:35,822][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [3.4s]
index : version [1319483349493], number_of_files [98] with
total_size [5.8mb], took [1.3s]
translog : id [1319482375162], number_of_operations [49], took
[61ms]
[2011-11-01 18:44:41,205][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:41,694][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [489ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [45ms]
translog : id [1319482375057], number_of_operations [421], took
[112ms]
[2011-11-01 18:44:45,822][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:46,438][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [616ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [117ms]
translog : id [1319482375162], number_of_operations [500], took
[63ms]
[2011-11-01 18:44:51,695][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:53,314][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [1.6s]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [38ms]
translog : id [1319482375057], number_of_operations [413], took
[64ms]
[2011-11-01 18:44:56,438][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:56,812][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [373ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [41ms]
translog : id [1319482375162], number_of_operations [551], took
[94ms]
[2011-11-01 18:45:03,315][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:06,813][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:07,186][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [373ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [42ms]
translog : id [1319482375162], number_of_operations [450], took
[74ms]
[2011-11-01 18:45:11,812][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [8.4s]
index : version [1319483347731], number_of_files [114] with
total_size [17.9mb], took [2s]
translog : id [1319482375058], number_of_operations [284], took
[90ms]
[2011-11-01 18:45:17,186][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:17,637][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [450ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [44ms]
translog : id [1319482375162], number_of_operations [458], took
[158ms]
[2011-11-01 18:45:21,813][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:22,795][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [982ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [48ms]
translog : id [1319482375058], number_of_operations [743], took
[90ms]
[2011-11-01 18:45:27,638][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:28,053][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [415ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [76ms]
translog : id [1319482375162], number_of_operations [392], took
[98ms]
[2011-11-01 18:45:32,796][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:33,144][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [348ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [39ms]
translog : id [1319482375058], number_of_operations [510], took
[91ms]
[2011-11-01 18:45:38,054][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:39,410][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [1.3s]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [42ms]
translog : id [1319482375162], number_of_operations [564], took
[88ms]
[2011-11-01 18:45:43,145][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:43,666][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [521ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [59ms]
translog : id [1319482375058], number_of_operations [568], took
[83ms]
[2011-11-01 18:45:49,410][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:50,025][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [615ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [55ms]
translog : id [1319482375162], number_of_operations [587], took
[308ms]
[2011-11-01 18:45:53,666][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:54,098][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [431ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [41ms]
translog : id [1319482375058], number_of_operations [545], took
[88ms]
[2011-11-01 18:46:00,025][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:46:00,535][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [509ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [85ms]
translog : id [1319482375162], number_of_operations [498], took
[81ms]
[2011-11-01 18:46:04,098][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:46:04,424][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [325ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [38ms]
translog : id [1319482375058], number_of_operations [379], took
[83ms]
[2011-11-01 18:46:10,535][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:46:11,168][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [632ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [40ms]
translog : id [1319482375162], number_of_operations [302], took
[188ms]
[2011-11-01 18:46:14,424][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:46:15,062][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [637ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [139ms]
translog : id [1319482375058], number_of_operations [399], took
[81ms]

2011/11/1 Shay Banon kimchy@gmail.com:

yes, but you will need to build a version of elasticsearch... . Should be
simple, download / clone the 0.18 branch, and run "gradlew release".

On Mon, Oct 31, 2011 at 9:53 PM, Gustavo Maia gustavobbmaia@gmail.com
wrote:

hi, Shay

Yes i can do it.

gateway: DEBUG
org.apache: WARN
discovery: TRACE

Only put it in the config/logging.yml ??

2011/10/31 Shay Banon kimchy@gmail.com

Strange..., nothing really fixed. I added some logging to 0.18 branch
that might help, can you try with a custom build of it, and set: env
to

TRACE?

On Fri, Oct 28, 2011 at 4:48 AM, Gustavo Maia <
gustavobbmaia@gmail.com>

wrote:

The following test was done using the 0.18.0 version of the ES.
Was set up multiple path, in multiple drives.
Below is the sizes of the data path.

/dev/sdb 414G 199M 393G 1% /mnt
2)
/dev/sdc 414G 199M 393G 1% /media/dev1
3)
/dev/sdd 414G 15G 378G 4% /media/dev2
4)
/dev/sde 414G 199M 393G 1% /media/dev3

This distribution of indices that correct or is it a bug
that was fixed

in version0.18.1?

Earlier I was talking about a test with the version 0.18.1, using the
same drive with multiple path.

2011/10/27 Gustavo Maia gustavobbmaia@gmail.com

OK. If I have 4 drives with the same size (150GB each drive). When
I index 1million documents all drives will have the same size?

I'm doing tests on my machine and then indexing at amazon.

2011/10/27 Shay Banon kimchy@gmail.com

The check for the size of where to place the data is based on the
free space available per "drive", so having multiple data
locations on the

same drive makes little sense.

On Fri, Oct 28, 2011 at 1:28 AM, Gustavo Maia
gustavobbmaia@gmail.com wrote:

Is the same Driver.

2011/10/27 Shay Banon kimchy@gmail.com

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia
gustavobbmaia@gmail.com wrote:

Hello,
I'm using version 0.18.1 and configuring path.data as follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I was
a

little
surprised the lengths of each path were not similar. Below the
size of
each path found.

gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger
than the
other paths. Is that correct? I thought that the sizes of the
path
would be the same or very close.

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia


(Shay Banon) #13

Heya, found the problem and fixed it in 0.18 and master branch. Will
release 0.18.3 sometime early next week with the fix.

On Wed, Nov 2, 2011 at 12:06 AM, Shay Banon kimchy@gmail.com wrote:

CAn you set in the logging.yml the "env" component to TRACE, and leave all
the rest with teh default values, and then gist hte log output?

On Tue, Nov 1, 2011 at 10:48 PM, Gustavo Maia gustavobbmaia@gmail.comwrote:

Please check, if it is correct.

My logging.yml file:

rootLogger: INFO, console, file
logger:

log action execution errors for easier debugging

action: DEBUG

reduce the logging for aws, too much is logged under the default INFO

com.amazonaws: WARN

gateway

gateway: DEBUG
index.gateway: DEBUG

peer shard recovery

indices.recovery: DEBUG

discovery

discovery: TRACE

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

Below my log, but i only see log about save in the s3.

[2011-11-01 18:43:51,141][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [335ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [42ms]
translog : id [1319482375161], number_of_operations [649], took
[72ms]
[2011-11-01 18:43:59,124][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:43:59,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [591ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [57ms]
translog : id [1319482375057], number_of_operations [555], took
[286ms]
[2011-11-01 18:44:01,141][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:01,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [574ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [51ms]
translog : id [1319482375161], number_of_operations [517], took
[95ms]
[2011-11-01 18:44:09,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:10,293][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [577ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [49ms]
translog : id [1319482375057], number_of_operations [344], took
[74ms]
[2011-11-01 18:44:11,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:12,039][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [323ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [52ms]
translog : id [1319482375161], number_of_operations [293], took
[48ms]
[2011-11-01 18:44:20,294][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:20,865][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [571ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [276ms]
translog : id [1319482375057], number_of_operations [524], took
[117ms]
[2011-11-01 18:44:22,039][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:22,354][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [313ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [32ms]
translog : id [1319482375161], number_of_operations [610], took
[76ms]
[2011-11-01 18:44:30,865][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:31,204][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [338ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [38ms]
translog : id [1319482375057], number_of_operations [655], took
[109ms]
[2011-11-01 18:44:32,354][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:35,822][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [3.4s]
index : version [1319483349493], number_of_files [98] with
total_size [5.8mb], took [1.3s]
translog : id [1319482375162], number_of_operations [49], took
[61ms]
[2011-11-01 18:44:41,205][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:41,694][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [489ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [45ms]
translog : id [1319482375057], number_of_operations [421], took
[112ms]
[2011-11-01 18:44:45,822][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:46,438][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [616ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [117ms]
translog : id [1319482375162], number_of_operations [500], took
[63ms]
[2011-11-01 18:44:51,695][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:53,314][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [1.6s]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [38ms]
translog : id [1319482375057], number_of_operations [413], took
[64ms]
[2011-11-01 18:44:56,438][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:56,812][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [373ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [41ms]
translog : id [1319482375162], number_of_operations [551], took
[94ms]
[2011-11-01 18:45:03,315][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:06,813][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:07,186][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [373ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [42ms]
translog : id [1319482375162], number_of_operations [450], took
[74ms]
[2011-11-01 18:45:11,812][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [8.4s]
index : version [1319483347731], number_of_files [114] with
total_size [17.9mb], took [2s]
translog : id [1319482375058], number_of_operations [284], took
[90ms]
[2011-11-01 18:45:17,186][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:17,637][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [450ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [44ms]
translog : id [1319482375162], number_of_operations [458], took
[158ms]
[2011-11-01 18:45:21,813][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:22,795][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [982ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [48ms]
translog : id [1319482375058], number_of_operations [743], took
[90ms]
[2011-11-01 18:45:27,638][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:28,053][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [415ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [76ms]
translog : id [1319482375162], number_of_operations [392], took
[98ms]
[2011-11-01 18:45:32,796][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:33,144][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [348ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [39ms]
translog : id [1319482375058], number_of_operations [510], took
[91ms]
[2011-11-01 18:45:38,054][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:39,410][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [1.3s]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [42ms]
translog : id [1319482375162], number_of_operations [564], took
[88ms]
[2011-11-01 18:45:43,145][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:43,666][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [521ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [59ms]
translog : id [1319482375058], number_of_operations [568], took
[83ms]
[2011-11-01 18:45:49,410][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:50,025][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [615ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [55ms]
translog : id [1319482375162], number_of_operations [587], took
[308ms]
[2011-11-01 18:45:53,666][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:54,098][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [431ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [41ms]
translog : id [1319482375058], number_of_operations [545], took
[88ms]
[2011-11-01 18:46:00,025][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:46:00,535][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [509ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [85ms]
translog : id [1319482375162], number_of_operations [498], took
[81ms]
[2011-11-01 18:46:04,098][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:46:04,424][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [325ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [38ms]
translog : id [1319482375058], number_of_operations [379], took
[83ms]
[2011-11-01 18:46:10,535][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:46:11,168][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [632ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [40ms]
translog : id [1319482375162], number_of_operations [302], took
[188ms]
[2011-11-01 18:46:14,424][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:46:15,062][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [637ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [139ms]
translog : id [1319482375058], number_of_operations [399], took
[81ms]

2011/11/1 Shay Banon kimchy@gmail.com:

yes, but you will need to build a version of elasticsearch... . Should
be
simple, download / clone the 0.18 branch, and run "gradlew release".

On Mon, Oct 31, 2011 at 9:53 PM, Gustavo Maia gustavobbmaia@gmail.com
wrote:

hi, Shay

Yes i can do it.

gateway: DEBUG
org.apache: WARN
discovery: TRACE

Only put it in the config/logging.yml ??

2011/10/31 Shay Banon kimchy@gmail.com

Strange..., nothing really fixed. I added some logging to 0.18 branch
that might help, can you try with a custom build of it, and set: env
to

TRACE?

On Fri, Oct 28, 2011 at 4:48 AM, Gustavo Maia <
gustavobbmaia@gmail.com>

wrote:

The following test was done using the 0.18.0 version of the ES.
Was set up multiple path, in multiple drives.
Below is the sizes of the data path.

/dev/sdb 414G 199M 393G 1% /mnt
2)
/dev/sdc 414G 199M 393G 1% /media/dev1
3)
/dev/sdd 414G 15G 378G 4% /media/dev2
4)
/dev/sde 414G 199M 393G 1% /media/dev3

This distribution of indices that correct or is it a bug
that was fixed

in version0.18.1?

Earlier I was talking about a test with
the version 0.18.1, using the

same drive with multiple path.

2011/10/27 Gustavo Maia gustavobbmaia@gmail.com

OK. If I have 4 drives with the same size (150GB each drive). When
I index 1million documents all drives will have the same size?

I'm doing tests on my machine and then indexing at amazon.

2011/10/27 Shay Banon kimchy@gmail.com

The check for the size of where to place the data is based on
the

free space available per "drive", so having multiple data
locations on the

same drive makes little sense.

On Fri, Oct 28, 2011 at 1:28 AM, Gustavo Maia
gustavobbmaia@gmail.com wrote:

Is the same Driver.

2011/10/27 Shay Banon kimchy@gmail.com

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia
gustavobbmaia@gmail.com wrote:

Hello,
I'm using version 0.18.1 and configuring path.data as
follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I
was a

little
surprised the lengths of each path were not similar. Below
the

size of
each path found.

gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index

/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger
than the
other paths. Is that correct? I thought that the sizes of
the

path
would be the same or very close.

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia


(Gustavo Maia) #14

Great..... thanks so much.
I will download and test, if i have some problem i tell you.

\thanks...

2011/11/2 Shay Banon kimchy@gmail.com:

Heya, found the problem and fixed it in 0.18 and master branch. Will release
0.18.3 sometime early next week with the fix.

On Wed, Nov 2, 2011 at 12:06 AM, Shay Banon kimchy@gmail.com wrote:

CAn you set in the logging.yml the "env" component to TRACE, and leave all
the rest with teh default values, and then gist hte log output?

On Tue, Nov 1, 2011 at 10:48 PM, Gustavo Maia gustavobbmaia@gmail.com
wrote:

Please check, if it is correct.

My logging.yml file:

rootLogger: INFO, console, file
logger:

log action execution errors for easier debugging

action: DEBUG

reduce the logging for aws, too much is logged under the default INFO

com.amazonaws: WARN

gateway

gateway: DEBUG
index.gateway: DEBUG

peer shard recovery

indices.recovery: DEBUG

discovery

discovery: TRACE

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

Below my log, but i only see log about save in the s3.

[2011-11-01 18:43:51,141][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [335ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [42ms]
translog : id [1319482375161], number_of_operations [649], took
[72ms]
[2011-11-01 18:43:59,124][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:43:59,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [591ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [57ms]
translog : id [1319482375057], number_of_operations [555], took
[286ms]
[2011-11-01 18:44:01,141][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:01,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [574ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [51ms]
translog : id [1319482375161], number_of_operations [517], took
[95ms]
[2011-11-01 18:44:09,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:10,293][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [577ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [49ms]
translog : id [1319482375057], number_of_operations [344], took
[74ms]
[2011-11-01 18:44:11,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:12,039][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [323ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [52ms]
translog : id [1319482375161], number_of_operations [293], took
[48ms]
[2011-11-01 18:44:20,294][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:20,865][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [571ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [276ms]
translog : id [1319482375057], number_of_operations [524], took
[117ms]
[2011-11-01 18:44:22,039][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:22,354][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [313ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [32ms]
translog : id [1319482375161], number_of_operations [610], took
[76ms]
[2011-11-01 18:44:30,865][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:31,204][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [338ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [38ms]
translog : id [1319482375057], number_of_operations [655], took
[109ms]
[2011-11-01 18:44:32,354][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:35,822][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [3.4s]
index : version [1319483349493], number_of_files [98] with
total_size [5.8mb], took [1.3s]
translog : id [1319482375162], number_of_operations [49], took
[61ms]
[2011-11-01 18:44:41,205][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:41,694][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [489ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [45ms]
translog : id [1319482375057], number_of_operations [421], took
[112ms]
[2011-11-01 18:44:45,822][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:46,438][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [616ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [117ms]
translog : id [1319482375162], number_of_operations [500], took
[63ms]
[2011-11-01 18:44:51,695][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:53,314][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [1.6s]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [38ms]
translog : id [1319482375057], number_of_operations [413], took
[64ms]
[2011-11-01 18:44:56,438][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:56,812][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [373ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [41ms]
translog : id [1319482375162], number_of_operations [551], took
[94ms]
[2011-11-01 18:45:03,315][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:06,813][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:07,186][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [373ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [42ms]
translog : id [1319482375162], number_of_operations [450], took
[74ms]
[2011-11-01 18:45:11,812][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [8.4s]
index : version [1319483347731], number_of_files [114] with
total_size [17.9mb], took [2s]
translog : id [1319482375058], number_of_operations [284], took
[90ms]
[2011-11-01 18:45:17,186][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:17,637][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [450ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [44ms]
translog : id [1319482375162], number_of_operations [458], took
[158ms]
[2011-11-01 18:45:21,813][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:22,795][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [982ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [48ms]
translog : id [1319482375058], number_of_operations [743], took
[90ms]
[2011-11-01 18:45:27,638][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:28,053][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [415ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [76ms]
translog : id [1319482375162], number_of_operations [392], took
[98ms]
[2011-11-01 18:45:32,796][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:33,144][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [348ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [39ms]
translog : id [1319482375058], number_of_operations [510], took
[91ms]
[2011-11-01 18:45:38,054][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:39,410][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [1.3s]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [42ms]
translog : id [1319482375162], number_of_operations [564], took
[88ms]
[2011-11-01 18:45:43,145][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:43,666][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [521ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [59ms]
translog : id [1319482375058], number_of_operations [568], took
[83ms]
[2011-11-01 18:45:49,410][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:50,025][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [615ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [55ms]
translog : id [1319482375162], number_of_operations [587], took
[308ms]
[2011-11-01 18:45:53,666][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:54,098][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [431ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [41ms]
translog : id [1319482375058], number_of_operations [545], took
[88ms]
[2011-11-01 18:46:00,025][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:46:00,535][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [509ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [85ms]
translog : id [1319482375162], number_of_operations [498], took
[81ms]
[2011-11-01 18:46:04,098][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:46:04,424][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [325ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [38ms]
translog : id [1319482375058], number_of_operations [379], took
[83ms]
[2011-11-01 18:46:10,535][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:46:11,168][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [632ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [40ms]
translog : id [1319482375162], number_of_operations [302], took
[188ms]
[2011-11-01 18:46:14,424][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:46:15,062][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [637ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [139ms]
translog : id [1319482375058], number_of_operations [399], took
[81ms]

2011/11/1 Shay Banon kimchy@gmail.com:

yes, but you will need to build a version of elasticsearch... . Should
be
simple, download / clone the 0.18 branch, and run "gradlew release".

On Mon, Oct 31, 2011 at 9:53 PM, Gustavo Maia gustavobbmaia@gmail.com
wrote:

hi, Shay

Yes i can do it.

gateway: DEBUG
org.apache: WARN
discovery: TRACE

Only put it in the config/logging.yml ??

2011/10/31 Shay Banon kimchy@gmail.com

Strange..., nothing really fixed. I added some logging to 0.18
branch
that might help, can you try with a custom build of it, and set: env
to
TRACE?

On Fri, Oct 28, 2011 at 4:48 AM, Gustavo Maia
gustavobbmaia@gmail.com
wrote:

The following test was done using the 0.18.0 version of the ES.
Was set up multiple path, in multiple drives.
Below is the sizes of the data path.

/dev/sdb 414G 199M 393G 1% /mnt
2)
/dev/sdc 414G 199M 393G 1% /media/dev1
3)
/dev/sdd 414G 15G 378G 4% /media/dev2
4)
/dev/sde 414G 199M 393G 1% /media/dev3

This distribution of indices that correct or is it a bug
that was fixed
in version0.18.1?

Earlier I was talking about a test with
the version 0.18.1, using the
same drive with multiple path.

2011/10/27 Gustavo Maia gustavobbmaia@gmail.com

OK. If I have 4 drives with the same size (150GB each
drive). When
I index 1million documents all drives will have the same size?

I'm doing tests on my machine and then indexing at amazon.

2011/10/27 Shay Banon kimchy@gmail.com

The check for the size of where to place the data is based on
the
free space available per "drive", so having multiple data
locations on the
same drive makes little sense.

On Fri, Oct 28, 2011 at 1:28 AM, Gustavo Maia
gustavobbmaia@gmail.com wrote:

Is the same Driver.

2011/10/27 Shay Banon kimchy@gmail.com

Is that on the same drive, or different drives?

On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia
gustavobbmaia@gmail.com wrote:

Hello,
I'm using version 0.18.1 and configuring path.data as
follows:

path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4

After 65,000 index doc, I see the size of each path and I
was a
little
surprised the lengths of each path were not similar. Below
the
size of
each path found.

gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh

/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 35M

You can observe that the size of this data4 path much larger
than the
other paths. Is that correct? I thought that the sizes of
the
path
would be the same or very close.

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia

--
Gustavo Maia


(system) #15