Extract data from elasticsearch and store into mongodb

[chinmoy_padhi]

Hi,
I'm looking for an option to extract data from Elasticsearch and store into the mongodb using some python script. Please help me with that.

[carly.richmond]

Hi Chinmoy,

Does it have to be done using Python? There is a similar question from February that suggests trying Logstash with an http output which might help:

Can i export data from elasticsearch into mongodb database?

Hi, I would like to know can I export data from Elasticsearch to mongodb? or from Elasticsearch to another database? I prefer Mongodb but if it is not possible I would like to know which database can I use and how if there is an example because I did not find. I have amount of data in Elasticsearch that's why i would to export it in another database. Any help would be sincerely appreciate!

[chinmoy_padhi]

Thanks @carly.richmond , by the way here what I have tried after googling,
can you suggest the script how it can be improved or
how can I query from an index pattern

from elasticsearch import Elasticsearch
from elasticsearch.helpers import scan
import pandas as pd
import pymongo
from pymongo import MongoClient


esClient = Elasticsearch('http://localhost:9200/')

def get_data_from_elastic():
    query = {
        "query": {
            "match": {
                 "state": "failed"
            }
        }
    }
    res=scan(client = esClient,
        query = query,
        scroll = '1m',
        index = 'test-2022-11-09',
        raise_on_error=True,
        preserve_order=False,
        clear_scroll=True)
    
    result = list(res)
    temp = []
    
    for hit in result:
        temp.append(hit['_source'])
    
    df = pd.DataFrame(temp)
    return df
df = get_data_from_elastic()
print(df.head())

client = MongoClient('mongodb://admin:vbox@10.20.30.40:27001/admin')
db = client['elasticDB']
elastic_data = db['elastic_data']
    
df.reset_index(inplace=True)
data_dict = df.to_dict("records")
elastic_data.insert_many(data_dict)

[carly.richmond]

Hi Chinmoy,

I've not used the Python client personally, but looking at the example in the Python docs for the scan method you've used it looks like it might be valid to provide the index pattern to the index property of the scan method:

https://elasticsearch-py.readthedocs.io/en/stable/helpers.html?highlight=scan#scan

Have you given it a try?

[chinmoy_padhi]

Awesome!!! That's what I was looking for. Thanks a lot @carly.richmond