IOException[Read past EOF]

Elastic Stack Elasticsearch
1

I'm periodically getting this exception when searching my indices. I'm
using 0.15.2. The full error I see is:

QueryPhaseExecutionException[[transaction_index_uvvasrpetuucrfnbbvsqbcbevvwqucat]
[0]: query[+_entityFormGuid:qbvavqropbtqrdvpwedowscqwequaqdf
+string._value:happyjack],from[0],size[10]: Query Failed [Failed to
execute main query]]; nested: IOException[Read past EOF];

I don't see any reported errors anywhere else. I'm searching across 2
indices, each index has 3 shards and no replicas. 5 of the shards are
apparently searched without error. I'm at a loss as to where to even
begin trying to diagnose the issue since it only happens periodically
for the same test case.

Any suggestions on what to look at would be appreciated.

Thanks,
Bob

2

You should see an exception for it in the logs, you sure its not there? Are you embedding elasticsearch in your java app?
On Monday, April 4, 2011 at 6:02 PM, Bob wrote:

I'm periodically getting this exception when searching my indices. I'm
using 0.15.2. The full error I see is:

QueryPhaseExecutionException[[transaction_index_uvvasrpetuucrfnbbvsqbcbevvwqucat]
[0]: query[+_entityFormGuid:qbvavqropbtqrdvpwedowscqwequaqdf
+string._value:happyjack],from[0],size[10]: Query Failed [Failed to
execute main query]]; nested: IOException[Read past EOF];

I don't see any reported errors anywhere else. I'm searching across 2
indices, each index has 3 shards and no replicas. 5 of the shards are
apparently searched without error. I'm at a loss as to where to even
begin trying to diagnose the issue since it only happens periodically
for the same test case.

Any suggestions on what to look at would be appreciated.

Thanks,
Bob

3

Thanks, Shay. Yes, I'm embedding it in my java app.

"you sure its not there"
Unfortunately/Sadly, no. Not sure at all. The "logs" I see are just
the output through the standard console output when I run my test
case. I didn't set up any Elasticsearch specific settings for logging
and when I look in the home directory I set for Elasticsearch I see
the directories for the indices/shards, but no log file to look at. I
see the translogs in each shard, but nothing with an exception in it.

I also should have provided more info from the getgo. The index that
has shards failing is configured (I hope) to be a memory only index
via:

    Builder settings = ImmutableSettings.settingsBuilder();
    settings.put("index.number_of_shards", 3);
    settings.put("index.number_of_replicas", 0);
    settings.put("index.store.type", "memory");
    CreateIndexRequest createIndexRequest = new

CreateIndexRequest(indexName, settings.build());
node.client().admin().indices().create(createIndexRequest);

By the way, is there a single location that specifies all the
different setting names? Some modules (in this case
Elasticsearch Platform — Find real-time answers at scale | Elastic)
talk about, but don't seem to actually specify, the actual parameter
name (e.g. - index.store.type).

Thanks,
Bob

On Apr 4, 12:13 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

You should see an exception for it in the logs, you sure its not there? Are you embedding elasticsearch in your java app?

On Monday, April 4, 2011 at 6:02 PM, Bob wrote:

I'm periodically getting this exception when searching my indices. I'm
using 0.15.2. The full error I see is:

QueryPhaseExecutionException[[transaction_index_uvvasrpetuucrfnbbvsqbcbevvw qucat]
[0]: query[+_entityFormGuid:qbvavqropbtqrdvpwedowscqwequaqdf
+string._value:happyjack],from[0],size[10]: Query Failed [Failed to
execute main query]]; nested: IOException[Read past EOF];

I don't see any reported errors anywhere else. I'm searching across 2
indices, each index has 3 shards and no replicas. 5 of the shards are
apparently searched without error. I'm at a loss as to where to even
begin trying to diagnose the issue since it only happens periodically
for the same test case.

Any suggestions on what to look at would be appreciated.

Thanks,
Bob

4

You might have a hit a problem with the memory based index implementation. Can you try and put "ram" instead of "memory" for the index type. It won't use direct buffers, but plain in memory lucene directory.
On Monday, April 4, 2011 at 8:53 PM, Bob wrote:

Thanks, Shay. Yes, I'm embedding it in my java app.

"you sure its not there"
Unfortunately/Sadly, no. Not sure at all. The "logs" I see are just
the output through the standard console output when I run my test
case. I didn't set up any Elasticsearch specific settings for logging
and when I look in the home directory I set for Elasticsearch I see
the directories for the indices/shards, but no log file to look at. I
see the translogs in each shard, but nothing with an exception in it.

I also should have provided more info from the getgo. The index that
has shards failing is configured (I hope) to be a memory only index
via:

Builder settings = ImmutableSettings.settingsBuilder();
settings.put("index.number_of_shards", 3);
settings.put("index.number_of_replicas", 0);
settings.put("index.store.type", "memory");
CreateIndexRequest createIndexRequest = new
CreateIndexRequest(indexName, settings.build());
node.client().admin().indices().create(createIndexRequest);

By the way, is there a single location that specifies all the
different setting names? Some modules (in this case
Elasticsearch Platform — Find real-time answers at scale | Elastic)
talk about, but don't seem to actually specify, the actual parameter
name (e.g. - index.store.type).

Thanks,
Bob

On Apr 4, 12:13 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

You should see an exception for it in the logs, you sure its not there? Are you embedding elasticsearch in your java app?

On Monday, April 4, 2011 at 6:02 PM, Bob wrote:

I'm periodically getting this exception when searching my indices. I'm
using 0.15.2. The full error I see is:

QueryPhaseExecutionException[[transaction_index_uvvasrpetuucrfnbbvsqbcbevvw qucat]
[0]: query[+_entityFormGuid:qbvavqropbtqrdvpwedowscqwequaqdf
+string._value:happyjack],from[0],size[10]: Query Failed [Failed to
execute main query]]; nested: IOException[Read past EOF];

I don't see any reported errors anywhere else. I'm searching across 2
indices, each index has 3 shards and no replicas. 5 of the shards are
apparently searched without error. I'm at a loss as to where to even
begin trying to diagnose the issue since it only happens periodically
for the same test case.

Any suggestions on what to look at would be appreciated.

Thanks,
Bob

5

By teh way, I suggest starting with flle system based storage, its, most times, fast enough. If not, you can also try mmapfs, which uses memory mapped files. The benefit of using file system based index is the fact that you can recover from it after restart.
On Tuesday, April 5, 2011 at 12:55 AM, Shay Banon wrote:

You might have a hit a problem with the memory based index implementation. Can you try and put "ram" instead of "memory" for the index type. It won't use direct buffers, but plain in memory lucene directory.
On Monday, April 4, 2011 at 8:53 PM, Bob wrote:

Thanks, Shay. Yes, I'm embedding it in my java app.

"you sure its not there"
Unfortunately/Sadly, no. Not sure at all. The "logs" I see are just
the output through the standard console output when I run my test
case. I didn't set up any Elasticsearch specific settings for logging
and when I look in the home directory I set for Elasticsearch I see
the directories for the indices/shards, but no log file to look at. I
see the translogs in each shard, but nothing with an exception in it.

I also should have provided more info from the getgo. The index that
has shards failing is configured (I hope) to be a memory only index
via:

Builder settings = ImmutableSettings.settingsBuilder();
settings.put("index.number_of_shards", 3);
settings.put("index.number_of_replicas", 0);
settings.put("index.store.type", "memory");
CreateIndexRequest createIndexRequest = new
CreateIndexRequest(indexName, settings.build());
node.client().admin().indices().create(createIndexRequest);

By thee way, is there a single location that specifies all the
different setting names? Some modules (in this case
Elasticsearch Platform — Find real-time answers at scale | Elastic)
talk about, but don't seem to actually specify, the actual parameter
name (e.g. - index.store.type).

Thanks,
Bob

On Apr 4, 12:13 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

You should see an exception for it in the logs, you sure its not there? Are you embedding elasticsearch in your java app?

On Monday, April 4, 2011 at 6:02 PM, Bob wrote:

I'm periodically getting this exception when searching my indices. I'm
using 0.15.2. The full error I see is:

QueryPhaseExecutionException[[transaction_index_uvvasrpetuucrfnbbvsqbcbevvw qucat]
[0]: query[+_entityFormGuid:qbvavqropbtqrdvpwedowscqwequaqdf
+string._value:happyjack],from[0],size[10]: Query Failed [Failed to
execute main query]]; nested: IOException[Read past EOF];

I don't see any reported errors anywhere else. I'm searching across 2
indices, each index has 3 shards and no replicas. 5 of the shards are
apparently searched without error. I'm at a loss as to where to even
begin trying to diagnose the issue since it only happens periodically
for the same test case.

Any suggestions on what to look at would be appreciated.

Thanks,
Bob

6

Thanks, Shay. I will try the "ram" setting and see how it goes. Of
course, I assumed I was doing something wrong so switched up how I was
doing indexing quite a bit and I haven't hit the problem again yet.
Time will tell. I'm using a memory store for my 'working' indices
because they are short lived, will be constantly created/destroyed
(one for each transaction in my system), be small, and do not have to
survive restarts. It seemed to me to be the best fit.

Bob

On Apr 4, 4:57 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

By teh way, I suggest starting with flle system based storage, its, most times, fast enough. If not, you can also try mmapfs, which uses memory mapped files. The benefit of using file system based index is the fact that you can recover from it after restart.

On Tuesday, April 5, 2011 at 12:55 AM, Shay Banon wrote:

You might have a hit a problem with the memory based index implementation. Can you try and put "ram" instead of "memory" for the index type. It won't use direct buffers, but plain in memory lucene directory.
On Monday, April 4, 2011 at 8:53 PM, Bob wrote:

Thanks, Shay. Yes, I'm embedding it in my java app.

"you sure its not there"
Unfortunately/Sadly, no. Not sure at all. The "logs" I see are just
the output through the standard console output when I run my test
case. I didn't set up any Elasticsearch specific settings for logging
and when I look in the home directory I set for Elasticsearch I see
the directories for the indices/shards, but no log file to look at. I
see the translogs in each shard, but nothing with an exception in it.

I also should have provided more info from the getgo. The index that
has shards failing is configured (I hope) to be a memory only index
via:

Builder settings = ImmutableSettings.settingsBuilder();
settings.put("index.number_of_shards", 3);
settings.put("index.number_of_replicas", 0);
settings.put("index.store.type", "memory");
CreateIndexRequest createIndexRequest = new
CreateIndexRequest(indexName, settings.build());
node.client().admin().indices().create(createIndexRequest);

By thee way, is there a single location that specifies all the
different setting names? Some modules (in this case
Elasticsearch Platform — Find real-time answers at scale | Elastic)
talk about, but don't seem to actually specify, the actual parameter
name (e.g. - index.store.type).

Thanks,
Bob

On Apr 4, 12:13 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

You should see an exception for it in the logs, you sure its not there? Are you embedding elasticsearch in your java app?

On Monday, April 4, 2011 at 6:02 PM, Bob wrote:

I'm periodically getting this exception when searching my indices. I'm
using 0.15.2. The full error I see is:

QueryPhaseExecutionException[[transaction_index_uvvasrpetuucrfnbbvsqbcbevvw qucat]
[0]: query[+_entityFormGuid:qbvavqropbtqrdvpwedowscqwequaqdf
+string._value:happyjack],from[0],size[10]: Query Failed [Failed to
execute main query]]; nested: IOException[Read past EOF];

I don't see any reported errors anywhere else. I'm searching across 2
indices, each index has 3 shards and no replicas. 5 of the shards are
apparently searched without error. I'm at a loss as to where to even
begin trying to diagnose the issue since it only happens periodically
for the same test case.

Any suggestions on what to look at would be appreciated.

Thanks,
Bob