Please check, if it is correct.
My logging.yml file:
rootLogger: INFO, console, file
logger:
log action execution errors for easier debugging
action: DEBUG
reduce the logging for aws, too much is logged under the default INFO
com.amazonaws: WARN
gateway
gateway: DEBUG
index.gateway: DEBUG
peer shard recovery
indices.recovery: DEBUG
discovery
discovery: TRACE
appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
Below my log, but i only see log about save in the s3.
[2011-11-01 18:43:51,141][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [335ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [42ms]
translog : id [1319482375161], number_of_operations [649], took [72ms]
[2011-11-01 18:43:59,124][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:43:59,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [591ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [57ms]
translog : id [1319482375057], number_of_operations [555], took [286ms]
[2011-11-01 18:44:01,141][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:01,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [574ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [51ms]
translog : id [1319482375161], number_of_operations [517], took [95ms]
[2011-11-01 18:44:09,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:10,293][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [577ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [49ms]
translog : id [1319482375057], number_of_operations [344], took [74ms]
[2011-11-01 18:44:11,715][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:12,039][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [323ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [52ms]
translog : id [1319482375161], number_of_operations [293], took [48ms]
[2011-11-01 18:44:20,294][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:20,865][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [571ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [276ms]
translog : id [1319482375057], number_of_operations [524], took [117ms]
[2011-11-01 18:44:22,039][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:22,354][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [313ms]
index : version [1319483349133], number_of_files [1] with
total_size [5.4kb], took [32ms]
translog : id [1319482375161], number_of_operations [610], took [76ms]
[2011-11-01 18:44:30,865][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:31,204][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [338ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [38ms]
translog : id [1319482375057], number_of_operations [655], took [109ms]
[2011-11-01 18:44:32,354][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:35,822][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [3.4s]
index : version [1319483349493], number_of_files [98] with
total_size [5.8mb], took [1.3s]
translog : id [1319482375162], number_of_operations [49], took [61ms]
[2011-11-01 18:44:41,205][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:41,694][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [489ms]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [45ms]
translog : id [1319482375057], number_of_operations [421], took [112ms]
[2011-11-01 18:44:45,822][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:46,438][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [616ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [117ms]
translog : id [1319482375162], number_of_operations [500], took [63ms]
[2011-11-01 18:44:51,695][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:44:53,314][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [1.6s]
index : version [1319483347414], number_of_files [1] with
total_size [6kb], took [38ms]
translog : id [1319482375057], number_of_operations [413], took [64ms]
[2011-11-01 18:44:56,438][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:44:56,812][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [373ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [41ms]
translog : id [1319482375162], number_of_operations [551], took [94ms]
[2011-11-01 18:45:03,315][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:06,813][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:07,186][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [373ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [42ms]
translog : id [1319482375162], number_of_operations [450], took [74ms]
[2011-11-01 18:45:11,812][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [8.4s]
index : version [1319483347731], number_of_files [114] with
total_size [17.9mb], took [2s]
translog : id [1319482375058], number_of_operations [284], took [90ms]
[2011-11-01 18:45:17,186][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:17,637][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [450ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [44ms]
translog : id [1319482375162], number_of_operations [458], took [158ms]
[2011-11-01 18:45:21,813][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:22,795][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [982ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [48ms]
translog : id [1319482375058], number_of_operations [743], took [90ms]
[2011-11-01 18:45:27,638][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:28,053][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [415ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [76ms]
translog : id [1319482375162], number_of_operations [392], took [98ms]
[2011-11-01 18:45:32,796][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:33,144][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [348ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [39ms]
translog : id [1319482375058], number_of_operations [510], took [91ms]
[2011-11-01 18:45:38,054][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:39,410][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [1.3s]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [42ms]
translog : id [1319482375162], number_of_operations [564], took [88ms]
[2011-11-01 18:45:43,145][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:43,666][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [521ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [59ms]
translog : id [1319482375058], number_of_operations [568], took [83ms]
[2011-11-01 18:45:49,410][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:45:50,025][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [615ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [55ms]
translog : id [1319482375162], number_of_operations [587], took [308ms]
[2011-11-01 18:45:53,666][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:45:54,098][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [431ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [41ms]
translog : id [1319482375058], number_of_operations [545], took [88ms]
[2011-11-01 18:46:00,025][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:46:00,535][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [509ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [85ms]
translog : id [1319482375162], number_of_operations [498], took [81ms]
[2011-11-01 18:46:04,098][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:46:04,424][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [325ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [38ms]
translog : id [1319482375058], number_of_operations [379], took [83ms]
[2011-11-01 18:46:10,535][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1]
...
[2011-11-01 18:46:11,168][DEBUG][index.gateway ] [Black
Fox] [topicoindex][1] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][1],
took [632ms]
index : version [1319483349493], number_of_files [1] with
total_size [6.3kb], took [40ms]
translog : id [1319482375162], number_of_operations [302], took [188ms]
[2011-11-01 18:46:14,424][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0]
...
[2011-11-01 18:46:15,062][DEBUG][index.gateway ] [Black
Fox] [topicoindex][0] snapshot (scheduled) completed to
s3://jusbrasil-index/[jusbrasil-search-cluster][indices][topicoindex][0],
took [637ms]
index : version [1319483347731], number_of_files [1] with
total_size [5.9kb], took [139ms]
translog : id [1319482375058], number_of_operations [399], took [81ms]
2011/11/1 Shay Banon kimchy@gmail.com:
yes, but you will need to build a version of elasticsearch... . Should be
simple, download / clone the 0.18 branch, and run "gradlew release".
On Mon, Oct 31, 2011 at 9:53 PM, Gustavo Maia gustavobbmaia@gmail.com
wrote:
hi, Shay
Yes i can do it.
gateway: DEBUG
org.apache: WARN
discovery: TRACE
Only put it in the config/logging.yml ??
2011/10/31 Shay Banon kimchy@gmail.com
Strange..., nothing really fixed. I added some logging to 0.18 branch
that might help, can you try with a custom build of it, and set: env to
TRACE?
On Fri, Oct 28, 2011 at 4:48 AM, Gustavo Maia gustavobbmaia@gmail.com
wrote:
The following test was done using the 0.18.0 version of the ES.
Was set up multiple path, in multiple drives.
Below is the sizes of the data path.
/dev/sdb 414G 199M 393G 1% /mnt
2)
/dev/sdc 414G 199M 393G 1% /media/dev1
3)
/dev/sdd 414G 15G 378G 4% /media/dev2
4)
/dev/sde 414G 199M 393G 1% /media/dev3
This distribution of indices that correct or is it a bug that was fixed
in version0.18.1?
Earlier I was talking about a test with the version 0.18.1, using the
same drive with multiple path.
2011/10/27 Gustavo Maia gustavobbmaia@gmail.com
OK. If I have 4 drives with the same size (150GB each drive). When
I index 1million documents all drives will have the same size?
I'm doing tests on my machine and then indexing at amazon.
2011/10/27 Shay Banon kimchy@gmail.com
The check for the size of where to place the data is based on the
free space available per "drive", so having multiple data locations on the
same drive makes little sense.
On Fri, Oct 28, 2011 at 1:28 AM, Gustavo Maia
gustavobbmaia@gmail.com wrote:
Is the same Driver.
2011/10/27 Shay Banon kimchy@gmail.com
Is that on the same drive, or different drives?
On Fri, Oct 28, 2011 at 12:08 AM, Gustavo Maia
gustavobbmaia@gmail.com wrote:
Hello,
I'm using version 0.18.1 and configuring path.data as follows:
path.data:/apps/elasticsearch-0.18.1/data/data1,/apps/elasticsearch-0.18.1/data/data2,/apps/elasticsearch-0.18.1/data/data3,/apps/elasticsearch-0.18.1/data/data4
After 65,000 index doc, I see the size of each path and I was a
little
surprised the lengths of each path were not similar. Below the
size of
each path found.
gustavo @ gustavo-linux: ~ $ ls-lh
/apps/elasticsearch-0.18.1/data/data4/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 350M
gustavo @ gustavo-linux: ~ $ ls-lh
/apps/elasticsearch-0.18.1/data/data3/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
34m total
gustavo @ gustavo-linux: ~ $ ls-lh
/apps/elasticsearch-0.18.1/data/data2/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 60M
gustavo @ gustavo-linux: ~ $ ls-lh
/apps/elasticsearch-0.18.1/data/data1/gustavo-index/nodes/0/indices/noticiaindex/0/index
/ | grep "total"
total 35M
You can observe that the size of this data4 path much larger
than the
other paths. Is that correct? I thought that the sizes of the
path
would be the same or very close.
--
Gustavo Maia
--
Gustavo Maia
--
Gustavo Maia
--
Gustavo Maia
--
Gustavo Maia