Ingest-attachment using CBOR sample

[carantunes (Carina de Oliveira Antunes)]

Looks like this is still an issue today, more than a year later.

Ingest-attachment using CBOR examples

Need some clarification regarding the following thread: Alex says: The trick is to just use the cbor builder to create your content, and add a bytearray to the field, you want to process instead of a base64 encoded string. What exactly is meant by "add a bytearray to the field?" I have been mapping an 'attachment-data' field with type 'keyword' and writing to that a base64 encoded string through the pipeline. I'm using ruby client library to encode the data. Now, I'm trying to cbor the at…

Can you provide a working sample of how to use ingest-attachment with cbor?
I'm struggling with processing large files.

[spinscale (Alexander Reelsen)]

hey,

you need to find a library for your progamming language that creates CBOR, so that the whole document including all fields is sent as CBOR, not only the attachment itself.

Does that make sense?

--Alex

[carantunes (Carina de Oliveira Antunes)]

Thank you.

For anyone struggling out there, here's a very basic working python sample, based on this configuration:

PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data"
      }
    }
  ]
}

Sample:

import cbor2
import requests

filename = 'some-file'
headers = {'content-type': 'application/cbor'}

with open(filename, 'rb') as f:
    doc = {
        'data': f.read()
    }
    requests.put(
        'http://localhost:9200/my_index/my_type/my_id?pipeline=attachment&pretty', 
        data=cbor2.dumps(doc), 
        headers=headers
    )

Be aware however that not all clients have support. The python client for example doesn't allow custom headers on the index method (https://github.com/elastic/elasticsearch-py/blob/master/elasticsearch/client/init.py#L330 )