TypeMissingException when indexing with 0.15.0-snapshot

Ryan1 · February 2, 2011, 8:01pm

while testing a fix for the issue i reported here:
http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/b79c6e1f2ad763c4#

psuedocode:

delete existing index
create new index with indexing options
update index with custom mapping settings
index all test data
force index refresh
perform search

executable testcase available here:

github.com

wireframe/elastic_searchable/blob/master/test/test_elastic_searchable.rb

require File.join(File.dirname(__FILE__), 'helper')

class TestElasticSearchable < Test::Unit::TestCase
  def setup
    delete_index
  end
  ElasticSearchable.debug_output

  context 'non elastic activerecord class' do
    class Cat < ActiveRecord::Base
    end
    should 'not respond to elastic_options' do
      assert !Cat.respond_to?(:elastic_options)
    end
  end

  class Post < ActiveRecord::Base
    elastic_searchable :index_options => {'number_of_replicas' => 0, 'number_of_shards' => 1}
    after_index :indexed
    after_index_on_create :indexed_on_create

This file has been truncated. show original

exception:
org.elasticsearch.indices.TypeMissingException: [friends]
type[friends] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:
87)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:
59)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:130)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Ryan1 · February 2, 2011, 8:12pm

I've published a simple gist that exercises the issue here:

gist.github.com

https://gist.github.com/wireframe/808316

elasticsearch_testcase.rb

require 'rubygems'
require 'rubberband'

index = 'foo'
type = 'bar'
searcher = ElasticSearch.new("localhost:9200", :index => index, :type => type)
searcher.delete_index(index) rescue nil
searcher.create_index index
searcher.index({:name => 'foo'})
searcher.refresh

This file has been truncated. show original

On Feb 2, 2:01 pm, Ryan ryan.son...@gmail.com wrote:

while testing a fix for the issue i reported here:http://groups.google.com/a/elasticsearch.com/group/users/browse_threa...

psuedocode:

delete existing index

create new index with indexing options

update index with custom mapping settings

index all test data

force index refresh

perform search

executable testcase available here:https://github.com/wireframe/elastic_searchable/blob/master/test/test...

exception:
org.elasticsearch.indices.TypeMissingException: [friends]
type[friends] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
87)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
59)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperation Action
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:130)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

kimchy · February 2, 2011, 8:43pm

Can you recreate it with a curl request?
On Wednesday, February 2, 2011 at 10:12 PM, Ryan wrote:

I've published a simple gist that exercises the issue here:
teardown/recreate index for testcases · GitHub

On Feb 2, 2:01 pm, Ryan ryan.son...@gmail.com wrote:

while testing a fix for the issue i reported here:http://groups.google.com/a/elasticsearch.com/group/users/browse_threa...

psuedocode:

delete existing index

create new index with indexing options

update index with custom mapping settings

index all test data

force index refresh

perform search

executable testcase available here:https://github.com/wireframe/elastic_searchable/blob/master/test/test...

exception:
org.elasticsearch.indices.TypeMissingException: [friends]
type[friends] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
87)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
59)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperation Action
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:130)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Ryan1 · February 2, 2011, 10:01pm

excellent idea. my curl script is unable to reproduce the issue which
means that this is most likely an issue with rubberband sigh

gist.github.com

https://gist.github.com/wireframe/808546

elasticsearch_testcase.sh

curl -XDELETE 'http://localhost:9200/foo'
curl -XPUT 'http://localhost:9200/foo' -d '
{
  index: {}
}
'
curl -XPUT 'http://localhost:9200/foo/bar/123' -d '
{
  name: "baz"
}

This file has been truncated. show original

On Feb 2, 2:43 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you recreate it with a curl request?

On Wednesday, February 2, 2011 at 10:12 PM, Ryan wrote:

I've published a simple gist that exercises the issue here:
teardown/recreate index for testcases · GitHub

On Feb 2, 2:01 pm, Ryan ryan.son...@gmail.com wrote:

while testing a fix for the issue i reported here:http://groups.google.com/a/elasticsearch.com/group/users/browse_threa...

psuedocode:

delete existing index

create new index with indexing options

update index with custom mapping settings

index all test data

force index refresh

perform search

executable testcase available here:https://github.com/wireframe/elastic_searchable/blob/master/test/test...

exception:
org.elasticsearch.indices.TypeMissingException: [friends]
type[friends] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
87)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
59)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperation Action
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:130)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

kimchy · February 2, 2011, 10:51pm

Don't want to complicate things :), but it might still be an issue with ES as probably rubberband sends the requests faster then the curl script. If you add delays between operations, does it still happen?
On Thursday, February 3, 2011 at 12:01 AM, Ryan wrote:

excellent idea. my curl script is unable to reproduce the issue which
means that this is most likely an issue with rubberband sigh
elasticsearch_testcase.sh · GitHub

On Feb 2, 2:43 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you recreate it with a curl request?

On Wednesday, February 2, 2011 at 10:12 PM, Ryan wrote:

I've published a simple gist that exercises the issue here:
teardown/recreate index for testcases · GitHub

On Feb 2, 2:01 pm, Ryan ryan.son...@gmail.com wrote:

while testing a fix for the issue i reported here:http://groups.google.com/a/elasticsearch.com/group/users/browse_threa...

psuedocode:

delete existing index

create new index with indexing options

update index with custom mapping settings

index all test data

force index refresh

perform search

executable testcase available here:https://github.com/wireframe/elastic_searchable/blob/master/test/test...

exception:
org.elasticsearch.indices.TypeMissingException: [friends]
type[friends] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
87)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
59)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperation Action
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:130)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Ryan1 · February 3, 2011, 6:40pm

After spending the last day ripping out rubberband and using straight
up ruby requests, I'm still able to reproduce the issue, but i've also
learned more about how to more-correctly integrate with elasticsearch.

My workaround for now is to not delete the entire index between test
runs. removing the mapping + documents and reindexing my test
fixtures is really all i need to do. Coming from a solr background,
the elasticsearch concept of one index with multiple types is easy to
overlook. i'm so used to nuking and repopulating the index between
test runs. any other recommendations or gotcha's for using
elasticsearch in a test environment would be appreciated!

Thanks again for your time.

On Feb 2, 4:51 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Don't want to complicate things :), but it might still be an issue with ES as probably rubberband sends the requests faster then the curl script. If you add delays between operations, does it still happen?

On Thursday, February 3, 2011 at 12:01 AM, Ryan wrote:

excellent idea. my curl script is unable to reproduce the issue which
means that this is most likely an issue with rubberband sigh
elasticsearch_testcase.sh · GitHub

On Feb 2, 2:43 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you recreate it with a curl request?

On Wednesday, February 2, 2011 at 10:12 PM, Ryan wrote:

I've published a simple gist that exercises the issue here:
teardown/recreate index for testcases · GitHub

On Feb 2, 2:01 pm, Ryan ryan.son...@gmail.com wrote:

while testing a fix for the issue i reported here:http://groups.google.com/a/elasticsearch.com/group/users/browse_threa...

psuedocode:

delete existing index

create new index with indexing options

update index with custom mapping settings

index all test data

force index refresh

perform search

executable testcase available here:https://github.com/wireframe/elastic_searchable/blob/master/test/test...

exception:
org.elasticsearch.indices.TypeMissingException: [friends]
type[friends] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
87)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
59)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperation Action
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:130)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

kimchy · February 3, 2011, 6:47pm

Heya,

In elasticsearch tests I delete an index and create it each time. I think that we should solve this problem, lemme ping you back when I have a solution. I think, for a testing perspective, deleting an index and creating it is the correct approach, so its a shame you won't be able to do it.

-shay.banon
On Thursday, February 3, 2011 at 8:40 PM, Ryan wrote:

After spending the last day ripping out rubberband and using straight
up ruby requests, I'm still able to reproduce the issue, but i've also
learned more about how to more-correctly integrate with elasticsearch.

My workaround for now is to not delete the entire index between test
runs. removing the mapping + documents and reindexing my test
fixtures is really all i need to do. Coming from a solr background,
the elasticsearch concept of one index with multiple types is easy to
overlook. i'm so used to nuking and repopulating the index between
test runs. any other recommendations or gotcha's for using
elasticsearch in a test environment would be appreciated!

Thanks again for your time.

On Feb 2, 4:51 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Don't want to complicate things :), but it might still be an issue with ES as probably rubberband sends the requests faster then the curl script. If you add delays between operations, does it still happen?

On Thursday, February 3, 2011 at 12:01 AM, Ryan wrote:

excellent idea. my curl script is unable to reproduce the issue which
means that this is most likely an issue with rubberband sigh
elasticsearch_testcase.sh · GitHub

On Feb 2, 2:43 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you recreate it with a curl request?

On Wednesday, February 2, 2011 at 10:12 PM, Ryan wrote:

I've published a simple gist that exercises the issue here:
teardown/recreate index for testcases · GitHub

On Feb 2, 2:01 pm, Ryan ryan.son...@gmail.com wrote:

while testing a fix for the issue i reported here:http://groups.google.com/a/elasticsearch.com/group/users/browse_threa...

psuedocode:

delete existing index

create new index with indexing options

update index with custom mapping settings

index all test data

force index refresh

perform search

executable testcase available here:https://github.com/wireframe/elastic_searchable/blob/master/test/test...

exception:
org.elasticsearch.indices.TypeMissingException: [friends]
type[friends] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
87)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
59)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperation Action
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:130)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

kimchy · February 3, 2011, 8:56pm

I just pushed an improvement to the introduction of a type in elasticsearch through an index operation. Can you give it a go and see if it solves your problem? (I can't recreate your problem locally...).
On Thursday, February 3, 2011 at 8:47 PM, Shay Banon wrote:
Heya,

In elasticsearch tests I delete an index and create it each time. I think that we should solve this problem, lemme ping you back when I have a solution. I think, for a testing perspective, deleting an index and creating it is the correct approach, so its a shame you won't be able to do it.

-shay.banon
On Thursday, February 3, 2011 at 8:40 PM, Ryan wrote:

After spending the last day ripping out rubberband and using straight
up ruby requests, I'm still able to reproduce the issue, but i've also
learned more about how to more-correctly integrate with elasticsearch.

My workaround for now is to not delete the entire index between test
runs. removing the mapping + documents and reindexing my test
fixtures is really all i need to do. Coming from a solr background,
the elasticsearch concept of one index with multiple types is easy to
overlook. i'm so used to nuking and repopulating the index between
test runs. any other recommendations or gotcha's for using
elasticsearch in a test environment would be appreciated!

Thanks again for your time.

On Feb 2, 4:51 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Don't want to complicate things :), but it might still be an issue with ES as probably rubberband sends the requests faster then the curl script. If you add delays between operations, does it still happen?

On Thursday, February 3, 2011 at 12:01 AM, Ryan wrote:

excellent idea. my curl script is unable to reproduce the issue which
means that this is most likely an issue with rubberband sigh
elasticsearch_testcase.sh · GitHub

On Feb 2, 2:43 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Can you recreate it with a curl request?

On Wednesday, February 2, 2011 at 10:12 PM, Ryan wrote:

I've published a simple gist that exercises the issue here:
teardown/recreate index for testcases · GitHub

On Feb 2, 2:01 pm, Ryan ryan.son...@gmail.com wrote:

while testing a fix for the issue i reported here:http://groups.google.com/a/elasticsearch.com/group/users/browse_threa...

psuedocode:

delete existing index

create new index with indexing options

update index with custom mapping settings

index all test data

force index refresh

perform search

executable testcase available here:https://github.com/wireframe/elastic_searchable/blob/master/test/test...

exception:
org.elasticsearch.indices.TypeMissingException: [friends]
type[friends] missing
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
87)
at
org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGet Action.java:
59)
at
org.elasticsearch.action.support.single.shard.TransportShardSingleOperation Action
$AsyncSingleAction$1.run(TransportShardSingleOperationAction.java:130)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)