Ingest-attachment using CBOR examples

Elastic Stack Elasticsearch
1

Need some clarification regarding the following thread:

Alex says:
The trick is to just use the cbor builder to create your content, and add a bytearray to the field, you want to process instead of a base64 encoded string.

What exactly is meant by "add a bytearray to the field?"

I have been mapping an 'attachment-data' field with type 'keyword' and writing to that a base64 encoded string through the pipeline. I'm using ruby client library to encode the data.

Now, I'm trying to cbor the attachment data, but it's not clear what type the 'attachment-data' field should be. When I use 'keyword' or 'binary', it seems to be expecting base64 encoded data.

2

Any ideas?

With the attachment pipeline field set to 'attachment_data' with type 'keyword', I'm trying to set the 'Content-Type' to 'application/cbor' and the body to a CBORed( { attachment_data: } ) but I get error:

[2017-07-19T19:44:53,404][DEBUG][o.e.a.b.TransportBulkAction] [J_rrNYb] failed to execute pipeline [attachment] for document [my_index/my_type/my_id]
org.elasticsearch.ElasticsearchParseException: Failed to parse content to map
	at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:143) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:114) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.action.index.IndexRequest.sourceAsMap(IndexRequest.java:355) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.ingest.PipelineExecutionService.innerExecute(PipelineExecutionService.java:164) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.ingest.PipelineExecutionService.access$000(PipelineExecutionService.java:41) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:88) [elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.0.jar:5.5.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_31]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_31]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]
Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 middle byte 0xec
 at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@474770f0; line: -1, column: 32]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.6.jar:2.8.6]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.6.jar:2.8.6]
	at com.fasterxml.jackson.dataformat.cbor.CBORParser._reportInvalidOther(CBORParser.java:3152) ~[jackson-dataformat-cbor-2.8.6.jar:2.8.6]
	at com.fasterxml.jackson.dataformat.cbor.CBORParser._reportInvalidOther(CBORParser.java:3157) ~[jackson-dataformat-cbor-2.8.6.jar:2.8.6]
	at com.fasterxml.jackson.dataformat.cbor.CBORParser._finishLongText(CBORParser.java:2062) ~[jackson-dataformat-cbor-2.8.6.jar:2.8.6]
	at com.fasterxml.jackson.dataformat.cbor.CBORParser._finishTextToken(CBORParser.java:1973) ~[jackson-dataformat-cbor-2.8.6.jar:2.8.6]
	at com.fasterxml.jackson.dataformat.cbor.CBORParser.getText(CBORParser.java:1353) ~[jackson-dataformat-cbor-2.8.6.jar:2.8.6]
	at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:86) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:369) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:317) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:280) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.map(AbstractXContentParser.java:235) ~[elasticsearch-5.5.0.jar:5.5.0]
	at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:141) ~[elasticsearch-5.5.0.jar:5.5.0]
	... 10 more

What should the attachment processor 'field' type be?
What should the index request body contain?
What should the content-type be?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.