Need help to index data from MongoDB (3.2.7) into Elastic Search (5.6.3)

[rahmathullatssr (Rahumathulla)]

Hi All, I need help to index MongoDB (3.2.7) data into Elastic Search (5.6.3) In a real-time manner. After some research, I found that river plugin is deprecated. But I could read that MongoConnector can be used.

Is MongoConnector, a better option for this use-case? Or is there any other options available?

[rahmathullatssr (Rahumathulla)]

I tried with MongoConnector with the following stack in my local environment.

1. Python 2.7.1
2. Elasticsearch 5.6.3
3. MongoDB 3.2.7
4. elastic2-doc-manager[elastic5]

Steps:

• Installed the required plugins mongo-connector and elastic2_doc_manager.
• Started MongoDB in 'singleNodeRepl' Mode.
• Inserted record in Mongo DB to check whether the data is getting indexed into ES. But indexing is not taking place properly.

I could see the below error from mongo-connector.log

2017-10-31 09:30:41,467 [ALWAYS] mongo_connector.connector:51 - Starting mongo-connector version: 2.5.1
2017-10-31 09:30:41,470 [ALWAYS] mongo_connector.connector:51 - Python version: 2.7.1 (r271:86832, Nov 27 2010, 17:19:03) [MSC v.1500 64 bit (AMD64)]
2017-10-31 09:30:41,474 [ALWAYS] mongo_connector.connector:51 - Platform: Windows-7-6.1.7601-SP1
2017-10-31 09:30:41,474 [ALWAYS] mongo_connector.connector:51 - pymongo version: 3.5.1
2017-10-31 09:31:11,486 [ERROR] mongo_connector.util:106 - Fatal Exception
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\mongo_connector-2.5.1-py2.7.egg\mongo_connector\util.py", line 104, in wrapped
func(*args, **kwargs)
File "C:\Python27\lib\site-packages\mongo_connector-2.5.1-py2.7.egg\mongo_connector\connector.py", line 347, in run
self.main_conn.admin.command('buildInfo')['version'])
File "C:\Python27\lib\site-packages\pymongo-3.5.1-py2.7-win-amd64.egg\pymongo\database.py", line 513, in command
with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
File "C:\Python27\lib\contextlib.py", line 17, in enter
return self.gen.next()
File "C:\Python27\lib\site-packages\pymongo-3.5.1-py2.7-win-amd64.egg\pymongo\mongo_client.py", line 904, in _socket_for_reads
with self._get_socket(read_preference) as sock_info:
File "C:\Python27\lib\contextlib.py", line 17, in enter
return self.gen.next()
File "C:\Python27\lib\site-packages\pymongo-3.5.1-py2.7-win-amd64.egg\pymongo\mongo_client.py", line 868, in _get_socket
server = self._get_topology().select_server(selector)
File "C:\Python27\lib\site-packages\pymongo-3.5.1-py2.7-win-amd64.egg\pymongo\topology.py", line 214, in select_server
address))
File "C:\Python27\lib\site-packages\pymongo-3.5.1-py2.7-win-amd64.egg\pymongo\topology.py", line 189, in select_servers
self._error_message(selector))
ServerSelectionTimeoutError: localhost:27017: timed out

Can anybody help me on this?

[rahmathullatssr (Rahumathulla)]

I have made little more progress on this:

Through MongoConnector, index is getting created but the data has not been reflected.

Below are the details of my collections:
DB: demo
Collection Name: states
Documents count : 4

Following are the documents:
singleNodeRepl:PRIMARY> db.states.find({}).pretty();
{ "_id" : 3, "state_name" : "ArunachalPradesh" }
{ "_id" : 2, "state_name" : "Bihar" }
{ "_id" : 4, "state_name" : "Kerala" }
{ "_id" : 1, "state_name" : "Andhrapradesh" }

But when I tried to access the data after index creation, I am getting the flag found : false.

Could you please tell me why I am unable to access the data?