Problem getting all records correctly indexed in elastic search


(Marc Witteveen) #1

I am having a problem with elastic search and mongodb which I can not seem
to solve. I've just setup a complete clean installation and the issue is
there also.

I am having the following installation

MongoDb: 2.4.9
Elastic search: 1.0.0
Attachment and MongoDB river and head plugin installed

The data that needs be searchable is in one collection only.

The tutorial I have followed is https://coderwall.com/p/sy1qcw
https://www.facebook.com/l.php?u=https%3A%2F%2Fcoderwall.com%2Fp%2Fsy1qcw&h=KAQFOviju&enc=AZORWnNjzlu8Kjv-5EsSmBPklHQanj8HFaeX-ipLwU9lz-QcyR7QGHYM2PfBnn5zu9mSvDq-JsjO8snF9oi2PFLXwlQA4GeMn8rsjEBoNTQ6uHPOTKjlq2Cp4lZXKvEESlpv_WEsXHGGanSnqzdn5894zN_ISz3kcTqcvOBRe-hWvA&s=1,
obvious I've used more up to date versions of the software.

The problems I am having:

  1. The data is not imported 100%, records are missing;
  2. The records have incomplete data, always the same field (an array) is
    not competently added, just the 1st element.
  3. If changes are made in the database they are not applies to the elastic
    search index.

How I get data into the database is as followed:
a) I download a zipfile from a server, this one is extracted and parsed
b) The parsed data is inserted into the mongodb collection
c) Then multiple zipfiles (containing meta data) are downloaded, extracted
and parsed and inserted into the previous created database records (update)

On step 3, once the meta data is parsed the record e.g. ABC1 is updated
with meta data A... then META data B is parsed, etc. and record ABC1 is
updated with meta data B.

The problem 2 and 3 might be related.

The index I have created is:

curl -XPUT "localhost:9200/_river/mongodbriver/_meta" -d'{
"type": "mongodb",
"mongodb": {
"db": "tenders_db",
"collection": "tenders"
},
"index": {
"name": "tenders_index",
"type": "tender_documents"
}
}'

Let me know if you need more information.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2a526e23-875e-4703-b2aa-569277c848f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #2