Extract data from elasticsearch and store into mongodb

Elastic Stack Elasticsearch
1

Hi,
I'm looking for an option to extract data from Elasticsearch and store into the mongodb using some python script. Please help me with that.

2

Hi Chinmoy,

Does it have to be done using Python? There is a similar question from February that suggests trying Logstash with an http output which might help:

3

Thanks @carly.richmond , by the way here what I have tried after googling,
can you suggest the script how it can be improved or
how can I query from an index pattern

from elasticsearch import Elasticsearch
from elasticsearch.helpers import scan
import pandas as pd
import pymongo
from pymongo import MongoClient


esClient = Elasticsearch('http://localhost:9200/')

def get_data_from_elastic():
    query = {
        "query": {
            "match": {
                 "state": "failed"
            }
        }
    }
    res=scan(client = esClient,
        query = query,
        scroll = '1m',
        index = 'test-2022-11-09',
        raise_on_error=True,
        preserve_order=False,
        clear_scroll=True)
    
    result = list(res)
    temp = []
    
    for hit in result:
        temp.append(hit['_source'])
    
    df = pd.DataFrame(temp)
    return df
df = get_data_from_elastic()
print(df.head())

client = MongoClient('mongodb://admin:vbox@10.20.30.40:27001/admin')
db = client['elasticDB']
elastic_data = db['elastic_data']
    
df.reset_index(inplace=True)
data_dict = df.to_dict("records")
elastic_data.insert_many(data_dict)
4

Hi Chinmoy,

I've not used the Python client personally, but looking at the example in the Python docs for the scan method you've used it looks like it might be valid to provide the index pattern to the index property of the scan method:

https://elasticsearch-py.readthedocs.io/en/stable/helpers.html?highlight=scan#scan

Have you given it a try?

5

Awesome!!! That's what I was looking for. Thanks a lot @carly.richmond

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.