Problem with Ingest Attachment Processor Plugin

[tomky (Tom)]

Hi!
I am new to this plugin and need your help.

This is what I do:

1. Create index:

PUT localhost:9200/myindex

2. And mappings:

PUT localhost:9200/myindex/mytype/_mapping
{
	"messages": {
	  "properties": {
		"subject": {"type": "string"},
		"sender": {"type": "string"},
		"receiver": {"type": "string"},
		"body": {"type": "text"}
	  }
	}
}

3. Now I create attachment by:

PUT http://localhost:9200/_ingest/pipeline/attachment
{
   "description" : "Extract attachment information",
   "processors" : [
   {
	   "attachment" : {
			"field" : "data",
			"indexed_chars" : -1
		}
   }]
}

4. I add some data and it works fine with searching etc.

5. But when I add an attachment to my previously added data like this:

PUT http://localhost:9200/myindex/mytype/1?pipeline=attachment
{
	"data": "some base64 data"
}

Then it overrides my whole previuosly added data and I am unable to search in my index.

What am I doing wrong? Please help.

[spinscale (Alexander Reelsen)]

Hey,

you need to supply the full document here and not just the field you want to add, when you use the Index API

--Alex

[dadoonet (David Pilato)]

Ingest plugin does not support partial updates. So you need to provide the full JSON document:

{
  "data": "BASE64 HERE",
  "subject": "foo",
  "sender": "foo",
  "body": "foo"
}

PS: note that you should use text in your mapping and not string.

[dadoonet (David Pilato)]

Sounds like @spinscale beat me on that one!

[tomky (Tom)]

OMG! So simple

[tomky (Tom)]

Why should I use text?

[tomky (Tom)]

When I want to use pipeline in PHP what should I do?
Code below doesn't work:

$params = [
    'index' => 'myindex',
    'type' => 'mytype',
    'id' => '1',
    'body' => [
		'data' => 'BASE64_DATA',
		'subject' => 'test',
		'sender' => 'test',
		'body' => 'test',
	],
];
$client->ingest()->putPipeline($params);

[tomky (Tom)]

I found a solution.
I have to use 'pipeline' => 'attachment' in my $params and then index it normally.