Multi-threaded update crashes elasticsearch


(Zmicier) #1

I have the following multi-threaded setup.

Several threads run in parallel and may check if the same document
exists (using get). If it does not exist, a thread will try to create
it (sometimes DocumentAlreadyExistsException is thrown because another
thread has just created a document with the same id, but that's OK).

If the document exists, a thread will try to update it using a script.
Again, there is a version conflict sometimes (I
use .setRetryOnConflict(20) as part of the update), and that's also
expected.

But sometimes, and this is not deterministic, a really weird error
shows on both client and server side. Here is the error log from the
server side:

java.lang.IllegalArgumentException: No type mapped for [122]
at org.elasticsearch.index.translog.Translog$Operation
$Type.fromId(Translog.java:214)
at
org.elasticsearch.index.translog.TranslogStreams.readSource(TranslogStreams.java:
59)
at
org.elasticsearch.index.engine.robin.RobinEngine.get(RobinEngine.java:
313)
at
org.elasticsearch.index.shard.service.InternalIndexShard.get(InternalIndexShard.java:
391)
at
org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:
126)
at
org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:
93)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:
102)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:
42)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:156)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On the client side, I see

org.elasticsearch.common.util.concurrent.UncategorizedExecutionException:
Failed execution
at
org.elasticsearch.action.support.AdapterActionFuture.rethrowExecutionException(AdapterActionFuture.java:
88)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:
49)
.....
Caused by: java.lang.IllegalArgumentException: No type mapped for
[122]
.....

There may be different numbers #N in "No type mapped for [#N]".
Usually a block of these errors would follow, and then everything
would return to normal. But sometimes after a block of these errors,
elastic actually crashes.

Any ideas what is going on here?


(Shay Banon) #2

Can you use the code you have to create a standalone testcase that we can
easily run locally?

On Tue, Apr 17, 2012 at 10:54 PM, Zmicier zialonka@gmail.com wrote:

I have the following multi-threaded setup.

Several threads run in parallel and may check if the same document
exists (using get). If it does not exist, a thread will try to create
it (sometimes DocumentAlreadyExistsException is thrown because another
thread has just created a document with the same id, but that's OK).

If the document exists, a thread will try to update it using a script.
Again, there is a version conflict sometimes (I
use .setRetryOnConflict(20) as part of the update), and that's also
expected.

But sometimes, and this is not deterministic, a really weird error
shows on both client and server side. Here is the error log from the
server side:

java.lang.IllegalArgumentException: No type mapped for [122]
at org.elasticsearch.index.translog.Translog$Operation
$Type.fromId(Translog.java:214)
at

org.elasticsearch.index.translog.TranslogStreams.readSource(TranslogStreams.java:
59)
at
org.elasticsearch.index.engine.robin.RobinEngine.get(RobinEngine.java:
313)
at

org.elasticsearch.index.shard.service.InternalIndexShard.get(InternalIndexShard.java:
391)
at
org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:
126)
at
org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:
93)
at

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:
102)
at

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:
42)
at

org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:156)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On the client side, I see

org.elasticsearch.common.util.concurrent.UncategorizedExecutionException:
Failed execution
at

org.elasticsearch.action.support.AdapterActionFuture.rethrowExecutionException(AdapterActionFuture.java:
88)
at

org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:
49)
.....
Caused by: java.lang.IllegalArgumentException: No type mapped for
[122]
.....

There may be different numbers #N in "No type mapped for [#N]".
Usually a block of these errors would follow, and then everything
would return to normal. But sometimes after a block of these errors,
elastic actually crashes.

Any ideas what is going on here?


(Zmicier) #3

It will take me a while to get a standalone test case. The threading
and the elastic code are in very different parts of the system.

By the way, is the Client code synchronized, that is, can multiple
threads do get/index/update simultaneously using the same Client
instance?

On Apr 18, 4:22 am, Shay Banon kim...@gmail.com wrote:

Can you use the code you have to create a standalone testcase that we can
easily run locally?

On Tue, Apr 17, 2012 at 10:54 PM, Zmicier zialo...@gmail.com wrote:

I have the following multi-threaded setup.

Several threads run in parallel and may check if the same document
exists (using get). If it does not exist, a thread will try to create
it (sometimes DocumentAlreadyExistsException is thrown because another
thread has just created a document with the same id, but that's OK).

If the document exists, a thread will try to update it using a script.
Again, there is a version conflict sometimes (I
use .setRetryOnConflict(20) as part of the update), and that's also
expected.

But sometimes, and this is not deterministic, a really weird error
shows on both client and server side. Here is the error log from the
server side:

java.lang.IllegalArgumentException: No type mapped for [122]
at org.elasticsearch.index.translog.Translog$Operation
$Type.fromId(Translog.java:214)
at

org.elasticsearch.index.translog.TranslogStreams.readSource(TranslogStreams .java:
59)
at
org.elasticsearch.index.engine.robin.RobinEngine.get(RobinEngine.java:
313)
at

org.elasticsearch.index.shard.service.InternalIndexShard.get(InternalIndexS hard.java:
391)
at
org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:
126)
at
org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:
93)
at

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
102)
at

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
42)
at

org.elasticsearch.action.support.single.shard.TransportShardSingleOperation Action
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:156)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On the client side, I see

org.elasticsearch.common.util.concurrent.UncategorizedExecutionException:
Failed execution
at

org.elasticsearch.action.support.AdapterActionFuture.rethrowExecutionExcept ion(AdapterActionFuture.java:
88)
at

org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio nFuture.java:
49)
.....
Caused by: java.lang.IllegalArgumentException: No type mapped for
[122]
.....

There may be different numbers #N in "No type mapped for [#N]".
Usually a block of these errors would follow, and then everything
would return to normal. But sometimes after a block of these errors,
elastic actually crashes.

Any ideas what is going on here?


(Shay Banon) #4

The Client code is thread safe, yes. It would be great if a
self sufficient test case can be extracted, thanks!

On Wed, Apr 18, 2012 at 6:09 PM, Zmicier zialonka@gmail.com wrote:

It will take me a while to get a standalone test case. The threading
and the elastic code are in very different parts of the system.

By the way, is the Client code synchronized, that is, can multiple
threads do get/index/update simultaneously using the same Client
instance?

On Apr 18, 4:22 am, Shay Banon kim...@gmail.com wrote:

Can you use the code you have to create a standalone testcase that we can
easily run locally?

On Tue, Apr 17, 2012 at 10:54 PM, Zmicier zialo...@gmail.com wrote:

I have the following multi-threaded setup.

Several threads run in parallel and may check if the same document
exists (using get). If it does not exist, a thread will try to create
it (sometimes DocumentAlreadyExistsException is thrown because another
thread has just created a document with the same id, but that's OK).

If the document exists, a thread will try to update it using a script.
Again, there is a version conflict sometimes (I
use .setRetryOnConflict(20) as part of the update), and that's also
expected.

But sometimes, and this is not deterministic, a really weird error
shows on both client and server side. Here is the error log from the
server side:

java.lang.IllegalArgumentException: No type mapped for [122]
at org.elasticsearch.index.translog.Translog$Operation
$Type.fromId(Translog.java:214)
at

org.elasticsearch.index.translog.TranslogStreams.readSource(TranslogStreams
.java:

  1. at
    org.elasticsearch.index.engine.robin.RobinEngine.get(RobinEngine.java:
  2. at

org.elasticsearch.index.shard.service.InternalIndexShard.get(InternalIndexS
hard.java:

  1. at

org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:

  1. at
    org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:
  2. at

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet
Action.java:

  1. at

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet
Action.java:

  1. at

org.elasticsearch.action.support.single.shard.TransportShardSingleOperation
Action

$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:156)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On the client side, I see

org.elasticsearch.common.util.concurrent.UncategorizedExecutionException:

Failed execution
at

org.elasticsearch.action.support.AdapterActionFuture.rethrowExecutionExcept
ion(AdapterActionFuture.java:

  1. at

org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActio
nFuture.java:

  1. .....
    Caused by: java.lang.IllegalArgumentException: No type mapped for
    [122]
    .....

There may be different numbers #N in "No type mapped for [#N]".
Usually a block of these errors would follow, and then everything
would return to normal. But sometimes after a block of these errors,
elastic actually crashes.

Any ideas what is going on here?


(Zmicier) #5

OK, here is the smallest setup that I have that replicates my problem:

Index definition:

curl -XPOST "http://localhost:9200/sample-index" -d '
{
"settings" : { "number_of_shards" : 1, "number_of_replicas" : 0 },

"mappings" : {
"sample" : {
"properties" : {
"names": {
"properties" : {
"value": { "type" : "string", "index" : "no" },
"count": { "type" : "integer", "index" : "no" }
}
}
}
}
}
}
'
And here is the self-contained java code:

import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.transport.InetSocketTransportAddress;

public class ElasticUpdateTest {

static final String updateScript =
"int i=0; "+
"found=false; "+
"while (!found && i < ctx._source.names.size()) { "+
"if (ctx._source.names.get(i).value == name) { " +
"found = true; " +
"} " +
"i++; " +
"} "+
"if (found) { " +
"ctx._source.names.get(i-1).count += 1; " +
"} " +
"else { " +
"ctx._source.names.add({'value' : name, 'count' : 1 }); " +
"} ";

static final String indexName = "sample-index";

static final String indexType = "sample";

private static class ElasticUpdate
    implements Runnable {
    public void run() {
      for(int i = 0; i<100000;i++) {
        String name = "name"+(i % 100);
        String key = "key"+ (i % 11);
        //System.out.println("Thread 

"+Thread.currentThread().getName()+" updating "+key+" with name "+name);
GetResponse getResponse = client.prepareGet(indexName,
indexType, key).execute().actionGet();
if (!getResponse.exists()) {
client.prepareIndex(indexName, indexType, key).setSource("{
"names" : [] }")
.execute()
.actionGet();
}
client.prepareUpdate(indexName, indexType, key)
.setRetryOnConflict(20)
.setScript(updateScript)
.addScriptParam("name", name)
.execute()
.actionGet();
}
}

    Client client;

    ElasticUpdate (Client client) { 
      this.client = client;
    }
}

public static void main(String args[]) throws InterruptedException {
    Client client = new TransportClient()
      .addTransportAddress(new InetSocketTransportAddress("localhost", 

9300));

    int nthreads = 4;
    if (args.length > 0) {
        try {
            nthreads = Integer.parseInt(args[0]);
        } catch (NumberFormatException e) {
            System.err.println("Argument must be an integer.");
            System.exit(1);
        }
    }
    System.out.println("Starting "+nthreads+" threads");
    for(int i=0;i<nthreads;i++) {
      Thread t = new Thread(new ElasticUpdate(client));
      t.start();
    }
}

}

I have the standard (out-of-box) cluster setup. When I run the above
program,
at some point get an error like this:

Exception in thread "Thread-4"
org.elasticsearch.common.util.concurrent.UncategorizedExecutionException:
Failed execution
at
org.elasticsearch.action.support.AdapterActionFuture.rethrowExecutionException(AdapterActionFuture.java:88)
at
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:49)
at ElasticUpdateTest$ElasticUpdate.run(ElasticUpdateTest.java:45)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException: No type mapped for [5]
at
org.elasticsearch.index.translog.Translog$Operation$Type.fromId(Translog.java:214)
at
org.elasticsearch.index.translog.TranslogStreams.readSource(TranslogStreams.java:59)
at
org.elasticsearch.index.engine.robin.RobinEngine.get(RobinEngine.java:313)
at
org.elasticsearch.index.shard.service.InternalIndexShard.get(InternalIndexShard.java:391)

    at 

org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:126)
at
org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:93)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:102)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:42)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:156)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
... 1 more

On Tuesday, April 17, 2012 3:54:42 PM UTC-4, Zmicier wrote:

I have the following multi-threaded setup.

Several threads run in parallel and may check if the same document
exists (using get). If it does not exist, a thread will try to create
it (sometimes DocumentAlreadyExistsException is thrown because another
thread has just created a document with the same id, but that's OK).

If the document exists, a thread will try to update it using a script.
Again, there is a version conflict sometimes (I
use .setRetryOnConflict(20) as part of the update), and that's also
expected.

But sometimes, and this is not deterministic, a really weird error
shows on both client and server side. Here is the error log from the
server side:

java.lang.IllegalArgumentException: No type mapped for [122]
at org.elasticsearch.index.translog.Translog$Operation
$Type.fromId(Translog.java:214)
at
org.elasticsearch.index.translog.TranslogStreams.readSource(TranslogStreams.java:

  1. at 
    

org.elasticsearch.index.engine.robin.RobinEngine.get(RobinEngine.java:
313)
at
org.elasticsearch.index.shard.service.InternalIndexShard.get(InternalIndexShard.java:

  1. at
    org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:
  2. at
    org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:
  3. at 
    

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:

  1. at
    org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:

  2. at 
    

org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction

$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:156)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On the client side, I see

org.elasticsearch.common.util.concurrent.UncategorizedExecutionException:
Failed execution
at
org.elasticsearch.action.support.AdapterActionFuture.rethrowExecutionException(AdapterActionFuture.java:

  1. at 
    

org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:

  1. ..... 
    

Caused by: java.lang.IllegalArgumentException: No type mapped for
[122]
.....

There may be different numbers #N in "No type mapped for [#N]".
Usually a block of these errors would follow, and then everything
would return to normal. But sometimes after a block of these errors,
elastic actually crashes.

Any ideas what is going on here?


(Zmicier) #6

I have reproduced the issue with both 0.19.1 and 0.19.2

On Tuesday, April 17, 2012 3:54:42 PM UTC-4, Zmicier wrote:

I have the following multi-threaded setup.

Several threads run in parallel and may check if the same document
exists (using get). If it does not exist, a thread will try to create
it (sometimes DocumentAlreadyExistsException is thrown because another
thread has just created a document with the same id, but that's OK).

If the document exists, a thread will try to update it using a script.
Again, there is a version conflict sometimes (I
use .setRetryOnConflict(20) as part of the update), and that's also
expected.

But sometimes, and this is not deterministic, a really weird error
shows on both client and server side. Here is the error log from the
server side:

java.lang.IllegalArgumentException: No type mapped for [122]
at org.elasticsearch.index.translog.Translog$Operation
$Type.fromId(Translog.java:214)
at
org.elasticsearch.index.translog.TranslogStreams.readSource(TranslogStreams.java:

  1. at 
    

org.elasticsearch.index.engine.robin.RobinEngine.get(RobinEngine.java:
313)
at
org.elasticsearch.index.shard.service.InternalIndexShard.get(InternalIndexShard.java:

  1. at
    org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:
  2. at
    org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:
  3. at 
    

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:

  1. at
    org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:

  2. at 
    

org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction

$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:156)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On the client side, I see

org.elasticsearch.common.util.concurrent.UncategorizedExecutionException:
Failed execution
at
org.elasticsearch.action.support.AdapterActionFuture.rethrowExecutionException(AdapterActionFuture.java:

  1. at 
    

org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:

  1. ..... 
    

Caused by: java.lang.IllegalArgumentException: No type mapped for
[122]
.....

There may be different numbers #N in "No type mapped for [#N]".
Usually a block of these errors would follow, and then everything
would return to normal. But sometimes after a block of these errors,
elastic actually crashes.

Any ideas what is going on here?

On Tuesday, April 17, 2012 3:54:42 PM UTC-4, Zmicier wrote:

I have the following multi-threaded setup.

Several threads run in parallel and may check if the same document
exists (using get). If it does not exist, a thread will try to create
it (sometimes DocumentAlreadyExistsException is thrown because another
thread has just created a document with the same id, but that's OK).

If the document exists, a thread will try to update it using a script.
Again, there is a version conflict sometimes (I
use .setRetryOnConflict(20) as part of the update), and that's also
expected.

But sometimes, and this is not deterministic, a really weird error
shows on both client and server side. Here is the error log from the
server side:

java.lang.IllegalArgumentException: No type mapped for [122]
at org.elasticsearch.index.translog.Translog$Operation
$Type.fromId(Translog.java:214)
at
org.elasticsearch.index.translog.TranslogStreams.readSource(TranslogStreams.java:

  1. at 
    

org.elasticsearch.index.engine.robin.RobinEngine.get(RobinEngine.java:
313)
at
org.elasticsearch.index.shard.service.InternalIndexShard.get(InternalIndexShard.java:

  1. at
    org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:
  2. at
    org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:
  3. at 
    

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:

  1. at
    org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:

  2. at 
    

org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction

$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:156)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On the client side, I see

org.elasticsearch.common.util.concurrent.UncategorizedExecutionException:
Failed execution
at
org.elasticsearch.action.support.AdapterActionFuture.rethrowExecutionException(AdapterActionFuture.java:

  1. at 
    

org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:

  1. ..... 
    

Caused by: java.lang.IllegalArgumentException: No type mapped for
[122]
.....

There may be different numbers #N in "No type mapped for [#N]".
Usually a block of these errors would follow, and then everything
would return to normal. But sometimes after a block of these errors,
elastic actually crashes.

Any ideas what is going on here?


(Shay Banon) #7

I have run the code (thanks!) but did not manage to reproduce it yet. Whats
the spec of the machine of the elasticsearch node you are running? I have
opened an issue where we can track this:
https://github.com/elasticsearch/elasticsearch/issues/1877.

On Thu, Apr 19, 2012 at 8:07 PM, Zmicier zialonka@gmail.com wrote:

I have reproduced the issue with both 0.19.1 and 0.19.2

On Tuesday, April 17, 2012 3:54:42 PM UTC-4, Zmicier wrote:

I have the following multi-threaded setup.

Several threads run in parallel and may check if the same document
exists (using get). If it does not exist, a thread will try to create
it (sometimes DocumentAlreadyExistsException is thrown because another
thread has just created a document with the same id, but that's OK).

If the document exists, a thread will try to update it using a script.
Again, there is a version conflict sometimes (I
use .setRetryOnConflict(20) as part of the update), and that's also
expected.

But sometimes, and this is not deterministic, a really weird error
shows on both client and server side. Here is the error log from the
server side:

java.lang.**IllegalArgumentException: No type mapped for [122]
at org.elasticsearch.index.**translog.Translog$Operation
$Type.fromId(Translog.java:**214)
at
org.elasticsearch.index.translog.TranslogStreams.
readSource(TranslogStreams.**java:
59)
at
org.elasticsearch.index.**engine.robin.RobinEngine.get(**RobinEngine.java:

  1. at
    org.elasticsearch.index.shard.**service.InternalIndexShard.**get(InternalIndexShard.java:

  2. at
    org.elasticsearch.index.get.**ShardGetService.innerGet(**ShardGetService.java:

  3. at
    org.elasticsearch.index.get.**ShardGetService.get(**ShardGetService.java:

  4. at
    

org.elasticsearch.action.get.**TransportGetAction.**shardOperation(**TransportGetAction.java:

  1. at
    org.elasticsearch.action.get.**TransportGetAction.**shardOperation(**TransportGetAction.java:

  2. at
    

org.elasticsearch.action.support.single.shard.
TransportShardSingleOperationA**ction
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:156)

    at java.util.concurrent.**ThreadPoolExecutor

$Worker.runTask(**ThreadPoolExecutor.java:886)
at java.util.concurrent.**ThreadPoolExecutor
$Worker.run(**ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.**java:662)

On the client side, I see

org.elasticsearch.common.util.concurrent.
UncategorizedExecutionExceptio**n:
Failed execution
at
org.elasticsearch.action.support.AdapterActionFuture.
rethrowExecutionException(**AdapterActionFuture.java:
88)
at
org.elasticsearch.action.support.AdapterActionFuture.
actionGet(AdapterActionFuture.**java:
49)
.....
Caused by: java.lang.**IllegalArgumentException: No type mapped for
[122]
.....

There may be different numbers #N in "No type mapped for [#N]".
Usually a block of these errors would follow, and then everything
would return to normal. But sometimes after a block of these errors,
elastic actually crashes.

Any ideas what is going on here?

On Tuesday, April 17, 2012 3:54:42 PM UTC-4, Zmicier wrote:

I have the following multi-threaded setup.

Several threads run in parallel and may check if the same document
exists (using get). If it does not exist, a thread will try to create
it (sometimes DocumentAlreadyExistsException is thrown because another
thread has just created a document with the same id, but that's OK).

If the document exists, a thread will try to update it using a script.
Again, there is a version conflict sometimes (I
use .setRetryOnConflict(20) as part of the update), and that's also
expected.

But sometimes, and this is not deterministic, a really weird error
shows on both client and server side. Here is the error log from the
server side:

java.lang.**IllegalArgumentException: No type mapped for [122]
at org.elasticsearch.index.**translog.Translog$Operation
$Type.fromId(Translog.java:**214)
at
org.elasticsearch.index.translog.TranslogStreams.
readSource(TranslogStreams.**java:
59)
at
org.elasticsearch.index.**engine.robin.RobinEngine.get(**RobinEngine.java:

  1. at
    org.elasticsearch.index.shard.**service.InternalIndexShard.**get(InternalIndexShard.java:

  2. at
    org.elasticsearch.index.get.**ShardGetService.innerGet(**ShardGetService.java:

  3. at
    org.elasticsearch.index.get.**ShardGetService.get(**ShardGetService.java:

  4. at
    

org.elasticsearch.action.get.**TransportGetAction.**shardOperation(**TransportGetAction.java:

  1. at
    org.elasticsearch.action.get.**TransportGetAction.**shardOperation(**TransportGetAction.java:

  2. at
    

org.elasticsearch.action.support.single.shard.
TransportShardSingleOperationA**ction
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:156)

    at java.util.concurrent.**ThreadPoolExecutor

$Worker.runTask(**ThreadPoolExecutor.java:886)
at java.util.concurrent.**ThreadPoolExecutor
$Worker.run(**ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.**java:662)

On the client side, I see

org.elasticsearch.common.util.concurrent.
UncategorizedExecutionExceptio**n:
Failed execution
at
org.elasticsearch.action.support.AdapterActionFuture.
rethrowExecutionException(**AdapterActionFuture.java:
88)
at
org.elasticsearch.action.support.AdapterActionFuture.
actionGet(AdapterActionFuture.**java:
49)
.....
Caused by: java.lang.**IllegalArgumentException: No type mapped for
[122]
.....

There may be different numbers #N in "No type mapped for [#N]".
Usually a block of these errors would follow, and then everything
would return to normal. But sometimes after a block of these errors,
elastic actually crashes.

Any ideas what is going on here?


(system) #8