# Extract data from elasticsearch and store into mongodb

**URL:** <https://discuss.elastic.co/t/extract-data-from-elasticsearch-and-store-into-mongodb/318337>\
**Category:** Elasticsearch\
**Created:** [November 7, 2022, 2:22pm UTC](https://discuss.elastic.co/t/extract-data-from-elasticsearch-and-store-into-mongodb/318337 "2022-11-07T14:22:05Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![chinmoy\_padhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chinmoy_padhi/32/68568_2.png) [@chinmoy\_padhi](https://discuss.elastic.co/u/chinmoy_padhi)\
**Post date:** [November 7, 2022, 2:22pm UTC](https://discuss.elastic.co/t/extract-data-from-elasticsearch-and-store-into-mongodb/318337/1 "2022-11-07T14:22:05Z")

</div>

Hi,  
I'm looking for an option to extract data from Elasticsearch and store into the mongodb using some python script. Please help me with that.

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [November 7, 2022, 5:30pm UTC](https://discuss.elastic.co/t/extract-data-from-elasticsearch-and-store-into-mongodb/318337/2 "2022-11-07T17:30:23Z")

</div>

Hi Chinmoy,

Does it have to be done using Python? There is a similar question from February that suggests trying Logstash with an http output which might help:

> [@Can i export data from elasticsearch into mongodb database?](https://discuss.elastic.co/t/can-i-export-data-from-elasticsearch-into-mongodb-database/296118):
>
> Hi, I would like to know can I export data from Elasticsearch to mongodb? or from Elasticsearch to another database? I prefer Mongodb but if it is not possible I would like to know which database can I use and how if there is an example because I did not find. I have amount of data in Elasticsearch that's why i would to export it in another database. Any help would be sincerely appreciate!

---

<div class="post-metadata">

**Author:** ![chinmoy\_padhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chinmoy_padhi/32/68568_2.png) [@chinmoy\_padhi](https://discuss.elastic.co/u/chinmoy_padhi)\
**Post date:** [November 10, 2022, 4:44pm UTC](https://discuss.elastic.co/t/extract-data-from-elasticsearch-and-store-into-mongodb/318337/3 "2022-11-10T16:44:01Z")

</div>

Thanks @carly.richmond , by the way here what I have tried after googling,  
can you suggest the script how it can be improved or  
how can I query from an index pattern

```auto
from elasticsearch import Elasticsearch
from elasticsearch.helpers import scan
import pandas as pd
import pymongo
from pymongo import MongoClient

esClient = Elasticsearch('http://localhost:9200/')

def get_data_from_elastic():
    query = {
        "query": {
            "match": {
                 "state": "failed"
            }
        }
    }
    res=scan(client = esClient,
        query = query,
        scroll = '1m',
        index = 'test-2022-11-09',
        raise_on_error=True,
        preserve_order=False,
        clear_scroll=True)
    
    result = list(res)
    temp = []
    
    for hit in result:
        temp.append(hit['_source'])
    
    df = pd.DataFrame(temp)
    return df
df = get_data_from_elastic()
print(df.head())

client = MongoClient('mongodb://admin:vbox@10.20.30.40:27001/admin')
db = client['elasticDB']
elastic_data = db['elastic_data']
    
df.reset_index(inplace=True)
data_dict = df.to_dict("records")
elastic_data.insert_many(data_dict)

```

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [November 10, 2022, 4:56pm UTC](https://discuss.elastic.co/t/extract-data-from-elasticsearch-and-store-into-mongodb/318337/4 "2022-11-10T16:56:35Z")

</div>

Hi Chinmoy,

I've not used the Python client personally, but looking at the example in the Python docs for the scan method you've used it looks like it might be valid to provide the index pattern to the index property of the scan method:

[https://elasticsearch-py.readthedocs.io/en/stable/helpers.html?highlight=scan#scan](https://elasticsearch-py.readthedocs.io/en/stable/helpers.html?highlight=scan#scan)

Have you given it a try?

---

<div class="post-metadata">

**Author:** ![chinmoy\_padhi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chinmoy_padhi/32/68568_2.png) [@chinmoy\_padhi](https://discuss.elastic.co/u/chinmoy_padhi)\
**Post date:** [November 11, 2022, 7:44am UTC](https://discuss.elastic.co/t/extract-data-from-elasticsearch-and-store-into-mongodb/318337/5 "2022-11-11T07:44:32Z")

</div>

Awesome!!! That's what I was looking for. Thanks a lot @carly.richmond

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 9, 2022, 7:44am UTC](https://discuss.elastic.co/t/extract-data-from-elasticsearch-and-store-into-mongodb/318337/6 "2022-12-09T07:44:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.