2.2.0 ESIntegTestCase: ensureYellow() results in "timed out waiting for yellow" - "unavailable_shards_exception"


(Ólafur Gauti Guðmundsson) #1

I am in the process of rewriting our integration tests to work with Elasticsearch 2.2.0.
I have an abstract test case class which extends org.elasticsearch.test.ESIntegTestCase:

@ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.SUITE, numDataNodes = 1)
@ThreadLeakScope(ThreadLeakScope.Scope.NONE)
public abstract class AbstractApplicationTest extends ESIntegTestCase {
   ...

    @Override
    protected Settings nodeSettings(int nodeOrdinal) {
        return Settings.settingsBuilder()
                .put(super.nodeSettings(nodeOrdinal))
                .put(IndexMetaData.SETTING_NUMBER_OF_SHARDS, 1)
                .put(IndexMetaData.SETTING_NUMBER_OF_REPLICAS, 1)
                .put(Node.HTTP_ENABLED, true)
                .put("script.groovy.sandbox.enabled", true)
                .put("script.engine.groovy.inline.search", true)
                .put("script.engine.groovy.inline.update", "true")
                .put("script.inline", true)
                .put("script.update", true)
                .put("script.indexed", true)
                .put("script.default_lang", "groovy")
                .build();
    }

    @Override
    protected Collection<Class<? extends Plugin>> nodePlugins() {
        return pluginList(GroovyPlugin.class, DeleteByQueryPlugin.class, AnalysisICUPlugin.class);
    }

    @Before
    public void setUpElastic() throws Exception {
        // add the index and the mappings
        prepareCreate("bokun")
                .setSettings(Settings.builder().loadFromSource(loadFromClasspath("indices/settings.json")))
                .addMapping(AccommodationSearchIndex.INDEX_ACCOMMODATION_TYPE, loadFromClasspath("indices/accommodation.json"))
                .addMapping(AccommodationSearchIndex.INDEX_ROOM_TYPE, loadFromClasspath("indices/room.json"))
                ...
                .execute().actionGet();
        ensureYellow();
    }
    ...
}

My test case class extends this abstract class. It has 6 tests, and constantly 3-4 of them fail with the "timed out waiting for yellow" message during the call ensureYellow() above.

The test case scope is ESIntegTestCase.Scope.SUITE, which means that a cluster will be created for each SUITE rather than test, and then I am creating the index and adding mappings before each test. I assume that after each test the index is deleted, as this page states: https://www.elastic.co/guide/en/elasticsearch/reference/current/integration-tests.html

By default the tests are run with unique cluster per test suite. Of course all indices and templates are deleted between each test.

Does anyone know what I am doing wrong here?

P.S. I am using SBT to run my tests.


(Ólafur Gauti Guðmundsson) #2

If I comment out the ensureYellow() call, then I am getting "unavailable_shards_exception" from Elasticsearch when executing index requests:

{
    "took": 60001,
    "errors": true,
    "items": [{
        "index": {
            "_index": "bokun",
            "_type": "activity_availability",
            "_id": "20_20160309",
            "status": 503,
            "error": {
                "type": "unavailable_shards_exception",
                "reason": "[bokun][2] primary shard is not active Timeout: [1m], request: [shard bulk {[bokun][2]}]"
            }
        }
    }, {
        "index": {
            "_index": "bokun",
            "_type": "activity_availability",
            "_id": "20_20160310",
            "status": 503,
            "error": {
                "type": "unavailable_shards_exception",
                "reason": "[bokun][1] primary shard is not active Timeout: [1m], request: [shard bulk {[bokun][1]}]"
            }
        }
    }, 
    ...
    {
        "index": {
            "_index": "bokun",
            "_type": "activity_availability",
            "_id": "20_20160322",
            "status": 503,
            "error": {
                "type": "unavailable_shards_exception",
                "reason": "[bokun][0] primary shard is not active Timeout: [1m], request: [shard bulk {[bokun][0]}]"
            }
        }
    }, {
        "index": {
            "_index": "bokun",
            "_type": "activity",
            "_id": "12",
            "status": 503,
            "error": {
                "type": "unavailable_shards_exception",
                "reason": "[bokun][2] primary shard is not active Timeout: [1m], request: [shard bulk {[bokun][2]}]"
            }
        }
    }]
}

(Ólafur Gauti Guðmundsson) #3

I've solved this. Elasticsearch logging was not showing up in my console. When I enabled that, I could see the actual cause of the problem:

ensureYellow timed out, cluster state:
version: 8
state uuid: twZ91SH7QIytVTrQuixDbg
from_diff: false
meta data version: 3
nodes: 
   {node_s0}{H_1t-4MwSKueM5vXRw_P4A}{local}{local[1]}{mode=local}, local, master
routing_table (version 6):
-- index [bokun]
----shard_id [bokun][0]
--------[bokun][0], node[null], [P], v[3], s[UNASSIGNED], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-03-09T13:07:48.812Z], details[failed to update mappings, failure IllegalArgumentException[Mapper for [translations.excerpt] conflicts with existing mapping in other types:
[mapper [translations.excerpt] has different [store] values]]]]

(system) #4