# ElasticSearch - nested query in batches?

**URL:** <https://discuss.elastic.co/t/elasticsearch-nested-query-in-batches/136256>\
**Category:** Elasticsearch\
**Created:** [June 18, 2018, 7:50am UTC](https://discuss.elastic.co/t/elasticsearch-nested-query-in-batches/136256 "2018-06-18T07:50:38Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![magorbe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magorbe/32/36871_2.png) [@magorbe](https://discuss.elastic.co/u/magorbe)\
**Post date:** [June 18, 2018, 7:50am UTC](https://discuss.elastic.co/t/elasticsearch-nested-query-in-batches/136256/1 "2018-06-18T07:50:38Z")

</div>

Hello! I have a problem when I do a query. The mapping is very simple :

```
{
    "index": {
        "aliases": {},
        "mappings": {
            "level1": {
                "properties": {
                    "id": {
                        "type": "string"
                    },
                    "level2": {
                        "type": "nested",
                        "properties": {
                            "level3": {
                                "type": "nested",
                                "properties": {
                                    "value1": {
                                        "type": "string"
                                    },
                                    "value2": {
                                        "type": "long"
                                    },
                                    "id": {
                                        "type": "string"
                                    },
                                    "value3": {
                                        "type": "long"
                                    }
                                }
                            },
                            "id": {
                                "type": "string"
                            }
                        }
                    }
                }
            }
        },
        "settings": {
            "index": {
                "creation_date": "1505476515647",
                "number_of_shards": "5",
                "number_of_replicas": "1",
                "uuid": "_0IiQCPrQ1i-kDP1481y8w",
                "version": {
                    "created": "2030099"
                }
            }
        },
        "warmers": {}
    }
}

```

And the query is :

`{"query": {"terms": {"_id": ["value51"] }}}`

When I do the query in Python I receive data with this structure:

```
_source (dict)
  level1 (list)
     level2 (list)
        data1 (dict)
              id
              value1
              value2
              value3
        data2 (dict)
        data3 (dict)
        ...
        data65000 (dict)

```

The problem is that 65,000 data are too many, and I run out of memory, I would like to know if \_search or ElasticSearch in general has some way of bringing that information (data1,data2,data3...) in batches, or if there is some way to make that query so that I do not run out of memory on the computer. Any idea?

Thank you. 🙂

---

<div class="post-metadata">

**Author:** ![Johnnycc1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johnnycc1/32/26069_2.png) [@Johnnycc1](https://discuss.elastic.co/u/Johnnycc1)\
**Post date:** [June 18, 2018, 11:13am UTC](https://discuss.elastic.co/t/elasticsearch-nested-query-in-batches/136256/2 "2018-06-18T11:13:11Z")

</div>

So you have one document in elastic search with 65000 nested sub documents - and when this one document is returned to python you run out of memory.

If elastic search could be configured to return part of the document in chunks - python would still run out of memory loading this one document.

Solution one - model your data differently - flatten out this nested document.

Solution two - get more memory 😀.

---

<div class="post-metadata">

**Author:** ![magorbe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magorbe/32/36871_2.png) [@magorbe](https://discuss.elastic.co/u/magorbe)\
**Post date:** [June 18, 2018, 11:16am UTC](https://discuss.elastic.co/t/elasticsearch-nested-query-in-batches/136256/3 "2018-06-18T11:16:08Z")

</div>

Thanks! Solution two for now is impossible. Would solution one be easy to do?

---

<div class="post-metadata">

**Author:** ![Johnnycc1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johnnycc1/32/26069_2.png) [@Johnnycc1](https://discuss.elastic.co/u/Johnnycc1)\
**Post date:** [June 18, 2018, 11:20am UTC](https://discuss.elastic.co/t/elasticsearch-nested-query-in-batches/136256/4 "2018-06-18T11:20:17Z")

</div>

Are you loading the data using logstash? If so you could just use the split filter and you will get one document for each document in the level 2 list - with all the upper level fields repeated.

---

<div class="post-metadata">

**Author:** ![magorbe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magorbe/32/36871_2.png) [@magorbe](https://discuss.elastic.co/u/magorbe)\
**Post date:** [June 18, 2018, 11:37am UTC](https://discuss.elastic.co/t/elasticsearch-nested-query-in-batches/136256/5 "2018-06-18T11:37:03Z")

</div>

No,I don't use logstash, I am doing :

`get_query = self.get(id=id, index=self.default_index, doc_type=self.default_doc_type)`

With the get of ES library. And i receive all the data.

---

<div class="post-metadata">

**Author:** ![magorbe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magorbe/32/36871_2.png) [@magorbe](https://discuss.elastic.co/u/magorbe)\
**Post date:** [June 18, 2018, 12:41pm UTC](https://discuss.elastic.co/t/elasticsearch-nested-query-in-batches/136256/6 "2018-06-18T12:41:11Z")

</div>

I don't understand the way to do that, can you give me any example ? Thanks!!!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 16, 2018, 12:41pm UTC](https://discuss.elastic.co/t/elasticsearch-nested-query-in-batches/136256/7 "2018-07-16T12:41:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.