# Indexing (storing) large binary object

**URL:** <https://discuss.elastic.co/t/indexing-storing-large-binary-object/230454>\
**Category:** Elasticsearch\
**Created:** [April 30, 2020, 12:46am UTC](https://discuss.elastic.co/t/indexing-storing-large-binary-object/230454 "2020-04-30T00:46:36Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![McKittrick\_Kaminski](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mckittrick_kaminski/32/59595_2.png) [@McKittrick\_Kaminski](https://discuss.elastic.co/u/McKittrick_Kaminski)\
**Post date:** [April 30, 2020, 12:46am UTC](https://discuss.elastic.co/t/indexing-storing-large-binary-object/230454/1 "2020-04-30T00:46:36Z")

</div>

Hello guys.

I'm making a plugin for Kibana, in which I want to store executables in Elasticsearch. (I know it sounds weird, trust me, there's a good reason for it.).

I've developed the client side code that will read the file (up to about 100MB), break it into smaller chunks (128KB atm, configurable, obviously), and upload them to Kibana's server side. Kibana's server side them prepares each chunk in base64, then indexes them into Elasticsearch in a particular field in a particular index.

I'm facing really slow upload (indexing) times for files larger than about 25MB, and I'm having trouble finding out where the bottleneck is.

The mapping for the field with data looks as follows

```auto
"filename": { "type": "keyword"},
"data": {
  "enabled": false,
  "properties": {
    "data": {"type": "binary"}, //<-- 160KB after base64
    "chunkNo": {"type": "integer"}
  }
}

```

so that it ends up this in ES

```auto
file._source
{
    "name": "metricbeat.exe",
    "data": [
        {
            "data": "BASE64...BASE64......BASE64......BASE64......BASE64...",
            "chunkNo": 0
        },
        {
            "data": "BASE64...BASE64......BASE64......BASE64......BASE64...",
            "chunkNo": 1
        },
        {
            "data": "BASE64...BASE64......BASE64......BASE64......BASE64...",
            "chunkNo": 2
        }
    ]
}

```

Couple of facts:

- Indexing is fast \<10MB (\<1s per 128k chunk)
- starts to slow down 10MB -\> 50MB (1-3s per 128k chunk)
- slowest at 50MB+ (3-6s per 128k chunk)
- My index mapping has one field that has a `"data": { "type": "binary" }"` mapping.
- I've tried having this field both `object` and `nested` types.
- I've tried setting `refresh_interval = -1`
- ES very quickly reaches is Java Heap max, and garbage collects _very_ often.
- garbage collection times (that it shows in the stdout) is up to about 1.5 seconds
- I have `enabled = false` set for the binary field the laaarge base64 string is saved.

I don't need any sort of indexing or searching capability whatsoever. In all of my other calls to this index I use `_source_excludes=['datafield']` to ignore this data, until I need it. Reading data doesn't seem to take too long, either.

What settings can I set on the index, indexing API call to optimize this? Should I be preparing the data in Kibana's server side differently?

Thanks!

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 30, 2020, 3:03am UTC](https://discuss.elastic.co/t/indexing-storing-large-binary-object/230454/2 "2020-04-30T03:03:43Z")

</div>

Can't you send each chunk as a separate document?

---

<div class="post-metadata">

**Author:** ![McKittrick\_Kaminski](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mckittrick_kaminski/32/59595_2.png) [@McKittrick\_Kaminski](https://discuss.elastic.co/u/McKittrick_Kaminski)\
**Post date:** [April 30, 2020, 3:27am UTC](https://discuss.elastic.co/t/indexing-storing-large-binary-object/230454/3 "2020-04-30T03:27:02Z")

</div>

Isn't that the purpose of the `nested` type, but with a little syntactic sugar? I've tried nested and gotten the same results.

Do you think manually saving documents and linking the IDs (effectively implementing nested documents in the kibana server plugin code) of the corresponding chunks into the "main" document would have a different effect?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 30, 2020, 5:17am UTC](https://discuss.elastic.co/t/indexing-storing-large-binary-object/230454/4 "2020-04-30T05:17:18Z")

</div>

Separate documents would be better as all nested documents are reindexed when you make any modification as the get larger and larger. Indexing chunks as separate documents will scale And perform a lot better than nested documents. I would however still recommend storing this type of content outside Elasticsearch.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 30, 2020, 7:31am UTC](https://discuss.elastic.co/t/indexing-storing-large-binary-object/230454/5 "2020-04-30T07:31:19Z")

</div>

Yeah. I'm also concerned by the HTTP layer. Sending 100mb of binary (which will end up being more than 100mb as BASE64 text) won't work by default. Also the HEAP size which is needed to hold the full document in memory is much bigger.

Than search, replication, ... will require a lot of disk reads and network usage.

My 2 cents.

That being said, why using elasticsearch in that case?

---

<div class="post-metadata">

**Author:** ![McKittrick\_Kaminski](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mckittrick_kaminski/32/59595_2.png) [@McKittrick\_Kaminski](https://discuss.elastic.co/u/McKittrick_Kaminski)\
**Post date:** [April 30, 2020, 11:21am UTC](https://discuss.elastic.co/t/indexing-storing-large-binary-object/230454/6 "2020-04-30T11:21:22Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> Separate documents would be better as all nested documents are reindexed when you make any modification as the get larger and larger. Indexing chinks as separate documents will scale And perform a lot better than nested documents.

I'll give fully separate documents a shot after this branch, thanks for the input.

> [@Christian\_Dahlqvist](#):
>
> I would however still recommend storing this type of content outside Elasticsearch.

> [@dadoonet](#):
>
> That being said, why using elasticsearch in that case?

I know, I know. I'm buidling an application (plugin) that pairs with a golang agent to do some service enumeration on an endpoint, bring results into Kibana, and Kibana issue relevant beats and configs. There will only be a dozen or so binaries stored in ES, and pulls on those binaries wont be constant. Its not a CDN feature, just a bit of supporting architecture. 98% of the value of this plugin is to bring log content from endpoints into ES easier for a hands-off party. 2% of it requires the capability of delivering binaries.

If I can get this functionality built with any sort of reasonable SLA's without breaking the whole stack, I'll probably use it. More likely, I'll just code in pulling from external sources like S3 buckets. Just wanted to keep it all in house single-deployment.

Thanks guys!

wait wait wait....

> [@Christian\_Dahlqvist](#):
>
> Indexing chinks

![image](https://us1.discourse-cdn.com/elastic/original/3X/6/e/6e5faf7f467d36b91dd52399d69c4d382d120fe6.jpeg)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 30, 2020, 11:38am UTC](https://discuss.elastic.co/t/indexing-storing-large-binary-object/230454/7 "2020-04-30T11:38:36Z")

</div>

Spell checker.... sigh

---

<div class="post-metadata">

**Author:** ![McKittrick\_Kaminski](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mckittrick_kaminski/32/59595_2.png) [@McKittrick\_Kaminski](https://discuss.elastic.co/u/McKittrick_Kaminski)\
**Post date:** [April 30, 2020, 3:01pm UTC](https://discuss.elastic.co/t/indexing-storing-large-binary-object/230454/8 "2020-04-30T15:01:54Z")

</div>

I've decided to go with bucket storage with my cloud provider. APIs for node are pretty clean, and waaaay cheaper and simple than I though.

Not worth

Thanks anyway tho.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 28, 2020, 3:02pm UTC](https://discuss.elastic.co/t/indexing-storing-large-binary-object/230454/9 "2020-05-28T15:02:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.