# Bulk Upload but search cant find all docs

**URL:** <https://discuss.elastic.co/t/bulk-upload-but-search-cant-find-all-docs/277634>\
**Category:** Kibana\
**Created:** [July 2, 2021, 9:36am UTC](https://discuss.elastic.co/t/bulk-upload-but-search-cant-find-all-docs/277634 "2021-07-02T09:36:45Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![GaryD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/garyd/32/78407_2.png) [@GaryD](https://discuss.elastic.co/u/GaryD)\
**Post date:** [July 2, 2021, 9:36am UTC](https://discuss.elastic.co/t/bulk-upload-but-search-cant-find-all-docs/277634/1 "2021-07-02T09:36:45Z")

</div>

I have done a bulk upload via the API in python as follows:

```auto
def to_bulk_doc(_doc):
        return {
            # default value: 'index'
            '_op_type': 'index',
            '_index': es_index,
            '_id': uuid.uuid4(),
            '_source': _doc,
            '_type': 'document'
        }
  for doc in json_docs_list:
            json_doc = json_docs_list[0]
            doc_resources: list = json.loads(doc)['resources']
            # split doc_resources into a list-of-lists, where each list has max=max_batch_size elements
            chunks = chunked(doc_resources, max_batch_size)
            for batch in chunks:
                # convert batch of json docs to a format compatible with bulk API using the 'to_bulk_doc' 
                function defined above
                actions = map(to_bulk_doc, batch)
                res = helpers.bulk(client=es, actions=actions)
                print(res)

```

This successfully uploads the documents to the index:  
In Dev Tools: GET /\_cat/indices/es\_index?v=true

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size  
green open es\_index Xngti... 5 1 304444 0 445.3mb 222.5mb

But searching in Kibana Dev Tools with:  
GET es\_index/\_search  
only returns 346 lines and nothing else, yet there is supposed to be 304K documents?

Any ideas what might be wrong?  
Thanks!

---

<div class="post-metadata">

**Author:** ![Marco\_Liberati](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marco_liberati/32/82953_2.png) [@Marco\_Liberati](https://discuss.elastic.co/u/Marco_Liberati)\
**Post date:** [July 2, 2021, 12:31pm UTC](https://discuss.elastic.co/t/bulk-upload-but-search-cant-find-all-docs/277634/2 "2021-07-02T12:31:25Z")

</div>

Hi @GaryD ,

welcome to the Kibana community.

How many hits are shown in the `hits.total.value`? Still 346 or 304k?

---

<div class="post-metadata">

**Author:** ![GaryD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/garyd/32/78407_2.png) [@GaryD](https://discuss.elastic.co/u/GaryD)\
**Post date:** [July 2, 2021, 1:20pm UTC](https://discuss.elastic.co/t/bulk-upload-but-search-cant-find-all-docs/277634/3 "2021-07-02T13:20:21Z")

</div>

> [@Marco\_Liberati](#):
>
> hits.total.value

Thanks Marco, have you got the Kibana Dev Tools Get for that?  
GET hits.total.value returns 404

---

<div class="post-metadata">

**Author:** ![Marco\_Liberati](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marco_liberati/32/82953_2.png) [@Marco\_Liberati](https://discuss.elastic.co/u/Marco_Liberati)\
**Post date:** [July 2, 2021, 1:22pm UTC](https://discuss.elastic.co/t/bulk-upload-but-search-cant-find-all-docs/277634/4 "2021-07-02T13:22:25Z")

</div>

Hi @GaryD

I think that there's a misunderstanding.

I meant this field in the response:

```auto
{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
...
  },
  "hits" : {
    "total" : {
      "value" : 4675, <== this number here
      "relation" : "eq"
    },
...

```

---

<div class="post-metadata">

**Author:** ![GaryD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/garyd/32/78407_2.png) [@GaryD](https://discuss.elastic.co/u/GaryD)\
**Post date:** [July 2, 2021, 1:25pm UTC](https://discuss.elastic.co/t/bulk-upload-but-search-cant-find-all-docs/277634/5 "2021-07-02T13:25:59Z")

</div>

Thanks Marco, I dont seem to have a value attribute:

```auto
{
  "took": 19,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 304444,
    "max_score": 1,
    "hits": [
      {
        "_index": "es_index",
        "_type": "document",
        "_id": "0cca6ec0-4772-4526-93bb-a437ced2698d",
        "_score": 1,
        "_source": {
          "metadata": {
            "guid": "8532129b-65c9-4701-add4-23601af9c3e5",
            "url": "/v2/events/8532129b-65c9-4701-add4-23601af9c3e5",
            "created_at": "2016-11-20T00:02:10Z",
            "updated_at": null
          },

```

---

<div class="post-metadata">

**Author:** ![Marco\_Liberati](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marco_liberati/32/82953_2.png) [@Marco\_Liberati](https://discuss.elastic.co/u/Marco_Liberati)\
**Post date:** [July 2, 2021, 1:28pm UTC](https://discuss.elastic.co/t/bulk-upload-but-search-cant-find-all-docs/277634/6 "2021-07-02T13:28:53Z")

</div>

You have `hits.total` which is showing that all the 304k documents have been hit.  
Note that ES is returning by design a limited set of hits in the `hits` list: you could retrieve more results using the `size` parameter in the requests, but I'd advice to not fetch too many documents at once, rather paginate the requests.

You can read more about it here: [Paginate search results | Elasticsearch Guide [7.13] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html)

---

<div class="post-metadata">

**Author:** ![GaryD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/garyd/32/78407_2.png) [@GaryD](https://discuss.elastic.co/u/GaryD)\
**Post date:** [July 2, 2021, 1:36pm UTC](https://discuss.elastic.co/t/bulk-upload-but-search-cant-find-all-docs/277634/7 "2021-07-02T13:36:03Z")

</div>

Thanks Marco, much appreciated.  
So if I have to paginate can I still build visual dashboard on the full index and search for hits like a normal index that has been populated by filebeat etc?

---

<div class="post-metadata">

**Author:** ![Marco\_Liberati](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marco_liberati/32/82953_2.png) [@Marco\_Liberati](https://discuss.elastic.co/u/Marco_Liberati)\
**Post date:** [July 2, 2021, 1:39pm UTC](https://discuss.elastic.co/t/bulk-upload-but-search-cant-find-all-docs/277634/8 "2021-07-02T13:39:52Z")

</div>

When building visualization you most probably will pass through some aggregation, so all your documents will be hit, no pagination will occur on that side.

---

<div class="post-metadata">

**Author:** ![GaryD](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/garyd/32/78407_2.png) [@GaryD](https://discuss.elastic.co/u/GaryD)\
**Post date:** [July 2, 2021, 1:44pm UTC](https://discuss.elastic.co/t/bulk-upload-but-search-cant-find-all-docs/277634/9 "2021-07-02T13:44:44Z")

</div>

Thank you, let me read some more on pagination and see if we can get the data out we need. Appreciate your fast responses here. Have a good weekend.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 30, 2021, 1:45pm UTC](https://discuss.elastic.co/t/bulk-upload-but-search-cant-find-all-docs/277634/10 "2021-07-30T13:45:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.