# Can the Ingest Attachment Processor Plugin extract array data? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 25, 2016, 2:39pm UTC](https://discuss.elastic.co/t/can-the-ingest-attachment-processor-plugin-extract-array-data/67199 "2016-11-25T14:39:47Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![M4urice](https://avatars.discourse-cdn.com/v4/letter/m/9fc348/32.png) [@M4urice](https://discuss.elastic.co/u/M4urice)\ **Post date:** [November 25, 2016, 2:39pm UTC](https://discuss.elastic.co/t/can-the-ingest-attachment-processor-plugin-extract-array-data/67199/1 "2016-11-25T14:39:47Z")
Hello! 🙂 So I am diving into v5 at the moment and want to use the Ingest Attachment Processor Plugin ([https://www.elastic.co/guide/en/elasticsearch/plugins/current/ingest-attachment.htm](https://www.elastic.co/guide/en/elasticsearch/plugins/current/ingest-attachment.htm)l) in combination with the Array "datatype" ([https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html)) A **document** would look somewhat like this: ``` { "_index": "visual-draft-3", "_type": "document", "_id": "441FDE6CFFF3D11EC12570F10053DE49", "_score": 1, "_source": { "EingangMuster": null, "Produkt_13": null, "Produkt_11": null, "attachments": [ { "filename": "somedoc.pdf", "data": "base64DataString" }, { "filename": "somedoc.docx", "data": "base64DataString" } ] } } ``` **Ingest Pipeline:** ``` esClient.ingest.putPipeline({ id: "attachment_pipe", body: { "description": "Process document attachments", "processors": [{ "attachment": { "field": "data", "indexed_chars": -1 } }] } }, function(error, response) { console.log(error, response); }); ``` This does not work out of the box for me, since I now have an array of attachments in one document which itself has the data field for the pipeline. The **error output** : ``` org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: ElasticsearchParseException[Error parsing document in field [data]]; nested: IllegalArgumentException[field [data] not present as part of path [data]]; at org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:156) ~[elasticsearch-5.0.1.jar:5.0.1] at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:107) ~[elasticsearch-5.0.1.jar:5.0.1] at org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:58) ~[elasticsearch-5.0.1.jar:5.0.1] at org.elasticsearch.ingest.PipelineExecutionService.innerExecute(PipelineExecutionService.java:166) ~[elasticsearch-5.0.1.jar:5.0.1] at org.elasticsearch.ingest.PipelineExecutionService.access$000(PipelineExecutionService.java:41) ~[elasticsearch-5.0.1.jar:5.0.1] at org.elasticsearch.ingest.PipelineExecutionService$1.doRun(PipelineExecutionService.java:65) [elasticsearch-5.0.1.jar:5.0.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:520) [elasticsearch-5.0.1.jar:5.0.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.0.1.jar:5.0.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_92] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_92] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92] Caused by: java.lang.IllegalArgumentException: ElasticsearchParseException[Error parsing document in field [data]]; nested: IllegalArgumentException[field [data] not present as part of path [data]]; ... 11 more Caused by: org.elasticsearch.ElasticsearchParseException: Error parsing document in field [data] at org.elasticsearch.ingest.attachment.AttachmentProcessor.execute(AttachmentProcessor.java:131) ~[?:?] at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:100) ~[elasticsearch-5.0.1.jar:5.0.1] ... 9 more Caused by: java.lang.IllegalArgumentException: field [data] not present as part of path [data] at org.elasticsearch.ingest.IngestDocument.resolve(IngestDocument.java:308) ~[elasticsearch-5.0.1.jar:5.0.1] at org.elasticsearch.ingest.IngestDocument.getFieldValue(IngestDocument.java:114) ~[elasticsearch-5.0.1.jar:5.0.1] at org.elasticsearch.ingest.IngestDocument.getFieldValueAsBytes(IngestDocument.java:141) ~[elasticsearch-5.0.1.jar:5.0.1] at org.elasticsearch.ingest.attachment.AttachmentProcessor.execute(AttachmentProcessor.java:71) ~[?:?] at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:100) ~[elasticsearch-5.0.1.jar:5.0.1] ... 9 more ``` How can the ingest plugin access the nested Array data? **Possible alternatives :** - Create a type for attachments and use the plugin there (then I would have to combine query results) - Add _n_ data fields to each document and add the attachment data there (don't want to do this, many null fields, and cluttered structure, sometimes **n\>50**!) I welcome any ideas, solutions and creative input! 😉 ---
**Author:** ![M4urice](https://avatars.discourse-cdn.com/v4/letter/m/9fc348/32.png) [@M4urice](https://discuss.elastic.co/u/M4urice)\ **Post date:** [November 29, 2016, 3:21pm UTC](https://discuss.elastic.co/t/can-the-ingest-attachment-processor-plugin-extract-array-data/67199/2 "2016-11-29T15:21:20Z")
_push_ What can I do to get an answer? Is some information missing? Is the question not clear? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 29, 2016, 5:23pm UTC](https://discuss.elastic.co/t/can-the-ingest-attachment-processor-plugin-extract-array-data/67199/3 "2016-11-29T17:23:33Z")
The question is really clear. It's on my TODO (TO ANSWER) list 🙂 I wonder if you can try to use a [foreach processor](https://www.elastic.co/guide/en/elasticsearch/reference/5.0/foreach-processor.html) for that? Can you give it a try and come back here? ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [November 30, 2016, 10:11am UTC](https://discuss.elastic.co/t/can-the-ingest-attachment-processor-plugin-extract-array-data/67199/4 "2016-11-30T10:11:57Z")
The foreach should be able to help you here like David said. If it doesn't then I think that should be fixed. ---
**Author:** ![gameldar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gameldar/32/14034_2.png) [@gameldar](https://discuss.elastic.co/u/gameldar)\ **Post date:** [December 21, 2016, 1:56am UTC](https://discuss.elastic.co/t/can-the-ingest-attachment-processor-plugin-extract-array-data/67199/5 "2016-12-21T01:56:11Z")
It doesn't seem to work exactly as you'd like, the attachment details are only available for the first entry because it looks like they are inserted at the top level, rather than in the array element being accessed. But I'll freely admit I'm not 100% sure I'm using the processors correctly. However, using the simulate API with the following document: ``` POST _ingest/pipeline/_simulate?verbose&pretty { "pipeline" : { "description": "_description", "processors": [ { "foreach": { "field": "attachments", "processor": { "attachment": { "field": "_ingest._value.data" } } } } ] }, "docs": [ { "_index": "index", "_type": "type", "_id": "id", "_source": { "attachments": [ { "filename": "ipsum.txt", "data": "dGhpcyBpcwpqdXN0IHNvbWUgdGV4dAo=" }, { "filename": "test.txt", "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=" } ] } } ] } ``` This results in: > ``` > { > "docs" : [ > { > "processor_results" : [ > { > "doc" : { > "_id" : "id", > "_index" : "index", > "_type" : "type", > "_source" : { > "attachments" : [ > { > "data" : "dGhpcyBpcwpqdXN0IHNvbWUgdGV4dAo=", > "filename" : "ipsum.txt" > }, > { > "data" : "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=", > "filename" : "test.txt" > } > ], > "attachment" : { > "language" : "ro", > "content_type" : "application/rtf", > "content" : "Lorem ipsum dolor sit amet", > "content_length" : 28 > } > }, > "_ingest" : { > "_value" : null, > "timestamp" : "2016-12-21T01:42:50.269+0000" > } > } > } > ] > } > ] > } > > ``` (apologies for the format seems I can't have two code blocks without the quoting) ---
**Author:** ![gameldar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gameldar/32/14034_2.png) [@gameldar](https://discuss.elastic.co/u/gameldar)\ **Post date:** [December 21, 2016, 2:29am UTC](https://discuss.elastic.co/t/can-the-ingest-attachment-processor-plugin-extract-array-data/67199/6 "2016-12-21T02:29:08Z")
Ok - I found out how to do it correctly (yay for opensource). The key is to set the target\_field on the attachment processor, and set this to a field on the `_ingest._value`: e.g. ``` POST _ingest/pipeline/_simulate?verbose&pretty { "pipeline" : { "description": "_description", "processors": [ { "foreach": { "field": "attachments", "processor": { "attachment": { "target_field": "_ingest._value.attachment", "field": "_ingest._value.data" } } } } ] }, "docs": [ { "_index": "index", "_type": "type", "_id": "id", "_source": { "attachments": [ { "filename": "ipsum.txt", "data": "dGhpcyBpcwpqdXN0IHNvbWUgdGV4dAo=" }, { "filename": "test.txt", "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=" } ] } } ] } ``` This will then correctly attach it to the array value, rather than the top level field "attachment" which is the default ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 21, 2016, 6:06am UTC](https://discuss.elastic.co/t/can-the-ingest-attachment-processor-plugin-extract-array-data/67199/7 "2016-12-21T06:06:37Z")
Great! Wondering if it is worth adding this as an example in the documentation? Would you like to open a PR to add this? ---
**Author:** ![gameldar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gameldar/32/14034_2.png) [@gameldar](https://discuss.elastic.co/u/gameldar)\ **Post date:** [December 21, 2016, 7:40am UTC](https://discuss.elastic.co/t/can-the-ingest-attachment-processor-plugin-extract-array-data/67199/8 "2016-12-21T07:40:17Z")
I've done so with Issue #22294 - hopefully I've done it correctly. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 18, 2017, 7:40am UTC](https://discuss.elastic.co/t/can-the-ingest-attachment-processor-plugin-extract-array-data/67199/9 "2017-01-18T07:40:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.