Couch-ES integration

I'm trying to write some integration tests to try out the integration
between couchdb and ES.

The idea is that I should:

  • create a database
  • add some entries to the eb
  • create a river

And it should be automatically indexed and be able to search from it.

I can't however get this working, no matter how many sleep and refresh() I
add there..

The setup and teardown functions look like this:

def setup_es():
global ES_CON
ES_CON = ES(ES_URI)
test_river = CouchDBRiver(host=COUCH_HOST, port=COUCH_PORT,
db=TEST_DATABASE)
ES_CON.create_river(test_river, river_name=TEST_INDEX)
ES_CON.refresh()

def setup_db():
global SERVER
SERVER = Server(uri=COUCH_URI)
if TEST_DATABASE in SERVER.all_dbs():
SERVER.delete_db(TEST_DATABASE)

db = SERVER.create_db(TEST_DATABASE)
db.save_doc(USER_DOC)

def clear():
SERVER.delete_db(TEST_DATABASE)
ES_CON.delete_river(TEST_INDEX)
ES_CON.indices.delete_index(TEST_INDEX)

And this test should for example pass:
def test_user_doc_indexed(self):
self.assertEqual(ES_CON.search(MatchAllQuery(),
indices=[TEST_INDEX]).count(), 1)

Is it a timing problem?
I'll just leave a test db running maybe if I have no choice, but would
prefer to do it all in the test module..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Even trying to don't delete the database and the river doesn't seem to work
unfortunately.

Now the problem is also that if I just create the river and the index I get
this:

SearchPhaseExecutionException: Failed to execute phase [query], total
failure; shardFailures {[na][test_index][0]: No active
shards}{[na][test_index][1]: No active shards}{[na][test_index][2]: No
active shards}{[na][test_index][3]: No active
shards}{[na][test_index][4]: No active shards}

BUT if I actually force indexing some data with something like this:
ES_CON.index(ud, TEST_INDEX, doc_type='User')

it works fine and the data is found, but the river is still not doing its
job..

On Tuesday, February 26, 2013 12:12:25 PM UTC, andrea crotti wrote:

I'm trying to write some integration tests to try out the integration
between couchdb and ES.

The idea is that I should:

  • create a database
  • add some entries to the eb
  • create a river

And it should be automatically indexed and be able to search from it.

I can't however get this working, no matter how many sleep and refresh() I
add there..

The setup and teardown functions look like this:

def setup_es():
global ES_CON
ES_CON = ES(ES_URI)
test_river = CouchDBRiver(host=COUCH_HOST, port=COUCH_PORT,
db=TEST_DATABASE)
ES_CON.create_river(test_river, river_name=TEST_INDEX)
ES_CON.refresh()

def setup_db():
global SERVER
SERVER = Server(uri=COUCH_URI)
if TEST_DATABASE in SERVER.all_dbs():
SERVER.delete_db(TEST_DATABASE)

db = SERVER.create_db(TEST_DATABASE)
db.save_doc(USER_DOC)

def clear():
SERVER.delete_db(TEST_DATABASE)
ES_CON.delete_river(TEST_INDEX)
ES_CON.indices.delete_index(TEST_INDEX)

And this test should for example pass:
def test_user_doc_indexed(self):
self.assertEqual(ES_CON.search(MatchAllQuery(),
indices=[TEST_INDEX]).count(), 1)

Is it a timing problem?
I'll just leave a test db running maybe if I have no choice, but would
prefer to do it all in the test module..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Andrea,

I think you should understand something.
When you create the river, the river takes some time to really start, establish the contact with CouchDB and index documents.
So, if 1ms after the river creation (even if you refresh the river), you are not sure that you will have the first document in it.
Or, the index is on the way of being created but all shards are not yet available.

Here are some suggestions:
1/ create the index first. waitForAYellowClusterStatus. Then add the river. Your search won't fail after that
2/ wait for a reasonable time, let's say 5 seconds.

I prefer the first option.

Does it help?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 26 févr. 2013 à 13:36, andrea crotti andrea.crotti.0@gmail.com a écrit :

Even trying to don't delete the database and the river doesn't seem to work unfortunately.

Now the problem is also that if I just create the river and the index I get this:

SearchPhaseExecutionException: Failed to execute phase [query], total failure; shardFailures {[na][test_index][0]: No active shards}{[na][test_index][1]: No active shards}{[na][test_index][2]: No active shards}{[na][test_index][3]: No active shards}{[na][test_index][4]: No active shards}

BUT if I actually force indexing some data with something like this:
ES_CON.index(ud, TEST_INDEX, doc_type='User')

it works fine and the data is found, but the river is still not doing its job..

On Tuesday, February 26, 2013 12:12:25 PM UTC, andrea crotti wrote:
I'm trying to write some integration tests to try out the integration between couchdb and ES.

The idea is that I should:

  • create a database
  • add some entries to the eb
  • create a river

And it should be automatically indexed and be able to search from it.

I can't however get this working, no matter how many sleep and refresh() I add there..

The setup and teardown functions look like this:

def setup_es():
global ES_CON
ES_CON = ES(ES_URI)
test_river = CouchDBRiver(host=COUCH_HOST, port=COUCH_PORT, db=TEST_DATABASE)
ES_CON.create_river(test_river, river_name=TEST_INDEX)
ES_CON.refresh()

def setup_db():
global SERVER
SERVER = Server(uri=COUCH_URI)
if TEST_DATABASE in SERVER.all_dbs():
SERVER.delete_db(TEST_DATABASE)

db = SERVER.create_db(TEST_DATABASE)
db.save_doc(USER_DOC)

def clear():
SERVER.delete_db(TEST_DATABASE)
ES_CON.delete_river(TEST_INDEX)
ES_CON.indices.delete_index(TEST_INDEX)

And this test should for example pass:
def test_user_doc_indexed(self):
self.assertEqual(ES_CON.search(MatchAllQuery(), indices=[TEST_INDEX]).count(), 1)

Is it a timing problem?
I'll just leave a test db running maybe if I have no choice, but would prefer to do it all in the test module..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.