# Tuning Attachment Ingest with arrays (get rid of the raw data!)
**URL:** https://discuss.elastic.co/t/tuning-attachment-ingest-with-arrays-get-rid-of-the-raw-data/72220
**Category:** Elasticsearch
**Created:** [January 19, 2017, 8:58pm UTC](https://discuss.elastic.co/t/tuning-attachment-ingest-with-arrays-get-rid-of-the-raw-data/72220 "2017-01-19T20:58:47Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@hogbinj](https://discuss.elastic.co/u/hogbinj)
#### Post date: [January 19, 2017, 8:58pm UTC](https://discuss.elastic.co/t/tuning-attachment-ingest-with-arrays-get-rid-of-the-raw-data/72220/1 "2017-01-19T20:58:47Z")
I'm using the pipeline in ingest an array of documents
```
PUT _ingest/pipeline/attachment
{
"description": "Extract attachment information",
"processors": [
{
"foreach": {
"field": "attachments",
"processor": {
"attachment": {
"field": "_ingest._value.data",
"target_field": "_ingest._value.attachment",
"properties": ["content"]
}
}
}
}
]
}
```
As a sample
```
PUT jh_index/my_type/my_id?pipeline=attachment
{
"attachments":
[
{
"data": "_encoded document - large amount of base64 guff_"},
{
"data": "_another encoded document - even more base64 guff_"}
]
}
```
Works like a champ however what i end up with is...
```
{
"_index": "jh_index",
"_type": "my_type",
"_id": "my_id",
"_version": 15,
"found": true,
"_source": {
"attachments": [
{
"data": " **_Large amount of base 64 guff I don't want_**",
"attachment": {
"content": "NEW HEADING1\nLorem ipsum dolor ......."
}
},
{
"data": " **_Another large amount of base 64 guff I don't want_**",
"attachment": {
"content": "HEADING1\nClick Insert and then choose the ........."
}
}
]
}
}
```
I've tried using "processor **s**" rather than "processor" so that I can add a
```
{
"remove": {"field": "_ingest._value.data"}
}
```
but that seems to have been developed out on purpose [Modify foreach processor to accept a single processor instead of collection #19345](https://github.com/elastic/elasticsearch/issues/19345)
I don't seem to be able to have 2 "foreach" one after the other in a pipeline
How do I remove the attachments.data field?
Thx
J/.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 19, 2017, 9:18pm UTC](https://discuss.elastic.co/t/tuning-attachment-ingest-with-arrays-get-rid-of-the-raw-data/72220/2 "2017-01-19T21:18:08Z")
I did not try but I don't see why you would not be able to add a second foreach.
Did you try it?
If so, share what you tried.
---
### Author:  [@talevy](https://discuss.elastic.co/u/talevy)
#### Post date: [January 19, 2017, 9:46pm UTC](https://discuss.elastic.co/t/tuning-attachment-ingest-with-arrays-get-rid-of-the-raw-data/72220/3 "2017-01-19T21:46:37Z")
here is an example of a pipeline that defines two `foreach` processors
```
POST _ingest/pipeline/_simulate
{
"pipeline" : {
"processors" : [
{
"foreach" : {
"field": "field",
"processor" : {
"uppercase" : { "field" : "_ingest._value.data" }
}
}
},
{
"foreach" : {
"field": "field",
"processor" : {
"remove" : { "field" : "_ingest._value.data" }
}
}
}
]
},
"docs" : [
{
"_source" : {
"field": [{"data": "a"}, {"data": "b"}, {"data": "c"}]
}
}
]
}
```
seems to work in removing the field `data`
---
### Author:  [@hogbinj](https://discuss.elastic.co/u/hogbinj)
#### Post date: [January 19, 2017, 10:21pm UTC](https://discuss.elastic.co/t/tuning-attachment-ingest-with-arrays-get-rid-of-the-raw-data/72220/4 "2017-01-19T22:21:50Z")
Kibana won't let you add more than one "foreach". I'm on cloud so it's the latest version.
I'm trying this
```
PUT _ingest/pipeline/attachment
{
"description": "Extract attachment information",
"processors": [
{
"foreach": {
"field": "attachments",
"processor": {
"attachment": {
"field": "_ingest._value.data",
"target_field": "_ingest._value.attachment",
"properties": ["content"]
}
}
},
"foreach": {
"field": "attachments",
"processor": {
"field": "field",
"processor" : {
"remove" : { "field" : "_ingest._value.data" }
}
}
}
}
]
}
```
---
### Author:  [@hogbinj](https://discuss.elastic.co/u/hogbinj)
#### Post date: [January 19, 2017, 10:42pm UTC](https://discuss.elastic.co/t/tuning-attachment-ingest-with-arrays-get-rid-of-the-raw-data/72220/5 "2017-01-19T22:42:26Z")
I was missing one closing curly bracket!. This code works.
Thank you so very much to [Tal Levy](https://github.com/talevy)
```
PUT _ingest/pipeline/attachment
{
"description": "Extract attachment information",
"processors": [
{
"foreach": {
"field": "attachments",
"processor": {
"attachment": {
"field": "_ingest._value.data",
"target_field": "_ingest._value.attachment",
"properties": ["content"]
}
}
}
},
{
"foreach": {
"field": "attachments",
"processor" : {
"remove" : { "field" : "_ingest._value.data" }
}
}
}
]
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 16, 2017, 10:43pm UTC](https://discuss.elastic.co/t/tuning-attachment-ingest-with-arrays-get-rid-of-the-raw-data/72220/6 "2017-02-16T22:43:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.