Hi Paul,
Yea, that exception helps a lot, though very very very strange... . This
is where its coming from:
File[] files = path.listFiles();
if (files == null || files.length == 0) {
return ImmutableMap.of();
}
ImmutableMap.Builder<String, BlobMetaData> builder =
ImmutableMap.builder();
for (File file : files) {
builder.put(file.getName(), new
PlainBlobMetaData(file.getName(), file.length()));
}
return builder.build();
Basically, as you can see, I like the files in a directory, and then build
an immutable map from them. The strange thing is that it complains that
basically the listFiles returned duplicate File... . I will fix this, but
how bizar!.
-shay.banon
On Thu, Sep 23, 2010 at 7:06 PM, Paul ppearcy@gmail.com wrote:
FYI, bumped up gateway logging (required a node restart, which cleared
the issue), so hopefully will have more data next time around. Also,
when I shut the node down, I got a stack trace that may be of more
use.
gist:593991 · GitHub
Thanks,
Paul
On Sep 23, 10:36 am, Paul ppea...@gmail.com wrote:
Hey Shay,
Hitting the snapshot failed exception, at the moment.
I tried increasing the log level, but it doesn't appear the
logging.yml file dynamically updates the log level.
Will probably start restarting nodes and playing around in a few
minutes. Let me know what I should have in place to get the necessary
information to track this down next time around. I'm on 0.10.0 and not
against moving to master, if that would help.
Thanks,
Paul
On Sep 21, 2:33 am, Shay Banon shay.ba...@elasticsearch.com wrote:
The REST interface uses the Java Client to do the operations, so I
don't
think its related. I will go over the exceptions and see that at least
they
are properly logged.
On Tue, Sep 21, 2010 at 6:59 AM, Paul ppea...@gmail.com wrote:
Btw, I was unable to reproduce the search exception via curl. Does
the
rest interface have internal retries? I am using the Java Node
client. Are there any retries available via that interface?
Thanks,
Paul
On Sep 20, 6:09 pm, Paul ppea...@gmail.com wrote:
Hey Shay,
Scoured the logs and, unfortunately, that is all I have. If I see
either of these again, will enable more detailed logging and see
what
I capture.
Thanks,
Paul
On Sep 20, 5:48 pm, Shay Banon shay.ba...@elasticsearch.com
wrote:
Hi Paul,
Both are strange. Are there by any chance more detailed
exceptions in
the
logs?
-shay.banon
On Tue, Sep 21, 2010 at 1:45 AM, Paul ppea...@gmail.com wrote:
Hi Shay,
Experienced some weird behavior over the weekend on 0.10 that
I
haven't seen before. Running a 2 node mirrored cluster.
- Searching a certain shard on certain node fails. Here is the
exception I was getting:
RemoteTransportException[[DM-ADSEARCHD102.dev.local][inet[/
10.2.20.160:9301]][search/phase/query/id]]; nested:
QueryPhaseExecutionException[[newsmedia_20100917150044][0]:
query[filtered(+(+feedid:753 +wsodissue:44874)
+__documentdate:[* TO
1285023084000])-
FilterCacheFilterWrapper(QueryWrapperFilter((+indexid:genericnews2 +
(feedid:753 feedid:1236)) (+indexid:newsmedia
+providersubgroup:ap)))],from[0],size[500],sort[<custom:"__documentdate":
org.elasticsearch.index.field.data.FieldData$Type
$4$1@63ab3977>!,<custom:"documentkey":
org.elasticsearch.index.field.data.FieldData$Type$1$1@7e49e6bf
]:
Query Failed [Failed to execute main query]]; nested:
The search is valid and would work every other time (as the
working
time, it went to a good server). To confirm, I shutdown the
good node
and it would fail every time. I then brought up the good node,
shutdown the bad one and it would work every time. After
bringing the
bad node back up, it was still failing the query. I was able to
resolve this by clearing the work directory on the bad node.
- Snapshot error. I have snapshot interval disabled and am
snapshotting based on content received. I started receiving
this
exception:
ERROR > Shapshot failed, index: djnf_20100917150037, shard: 0,
reason:
BroadcastShardOperationFailedException[[djnf_20100917150037][0]
];
nested:
RemoteTransportException[[dm-adsearchd103.dev.local][inet[/
10.2.20.164:9300]][indices/gateway/snapshot/shard]]; nested:
IndexShardGatewaySnapshotFailedException[[djnf_20100917150037][0]
duplicate key: __2tf]; nested:
IllegalArgumentException[duplicate
key:
__2tf]; (Timer-0)
This was resolved by restarting the cluster.
I have only seen both these issues once and will keep an eye
out for
them again, but wanted to give a heads up, as they seem like
potential
issues.
Thanks,
Paul