# Extracting full-text from an attachment field **URL:** \ **Category:** Elasticsearch\ **Created:** [May 30, 2018, 7:18pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933 "2018-05-30T19:18:38Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![silversun](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@silversun](https://discuss.elastic.co/u/silversun)\ **Post date:** [May 30, 2018, 7:18pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/1 "2018-05-30T19:18:38Z")
I am a newbie for Elasticsearch. I'm working on an application which integrates Drupal with **Elasticsearch 2.4**. Our current ES index has data with a field called "file\_resource\_url" with a full URL path to a PDF/DOC file. However, the text within those PDF/DOC file is currently not indexed and those attachments are not searchable. I would like to users to be able to search against the full-text of the documents specified in the "file\_resource\_url" field. Please provide guidance on how to go about this. Also, note that the app with ES is currently deployed on a cloud foundry instance where my access is quite limited in terms of installing something new. ---
**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\ **Post date:** [June 1, 2018, 9:44am UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/2 "2018-06-01T09:44:21Z")
> [@silversun](#): > > Also, note that the app with ES is currently deployed on a cloud foundry instance where my access is quite limited in terms of installing something new. That's a shame, because there is the [Ingest Attachment Processor Plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-attachment.html) (or the [Mapper Attachment Plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/2.4/mapper-attachments.html) for pre-6.0 indices) that should be well suited for the job. I would check out if you can install it anyway. If not, maybe its time to look into our [Elastic Cloud](https://www.elastic.co/cloud) offering, I'm pretty sure the plugin is available there. ---
**Author:** ![silversun](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@silversun](https://discuss.elastic.co/u/silversun)\ **Post date:** [June 1, 2018, 2:47pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/3 "2018-06-01T14:47:58Z")
Thanks for your response! I can reach out to them about Mapper Attachment plugin. I have installed that plugin in my local. But, I am unable to update the field type to attachment. Here is how my index currently looks - ``` { "took": 13, "timed_out": false, "_shards": { "total": 14, "successful": 14, "failed": 0 }, "hits": { "total": 172, "max_score": 1, "hits": [ { "_index": "appindex", "_type": "appindex", "_id": "8", "_score": 1, "_source": { "id": 8, "body:format": "filtered_html", "body:value": "text text text", "changed": "1506460443", ... "field_attachment_resource:file:fid": "12", "field_attachment_resource:file:mime": "application/pdf", "field_attachment_resource:file:name": "SampleDocument.pdf", ``` This is the current mapping of "field\_attachement\_resource": ``` { "appindex" : { "mappings" : { "appindex" : { "properties" : { "field_attachment_resource:file:fid" : { "type" : "string" }, "field_attachment_resource:file:mime" : { "type" : "string" }, "field_attachment_resource:file:name" : { "type" : "string" }, ``` I tried to update the type of "field\_attachment\_resource:file:name" from string to "attachment" - ``` curl -X PUT 'http://localhost:9200/appindex/appindex/_mapping?ignore_conflicts=true' -d \ '{ "field_attachment_resource": { "properties": { "name": { "type": "attachment" } } } }' ``` It failed with this error - **{"error":{"root\_cause":[{"type":"mapper\_parsing\_exception","reason":"Root mapping definition has unsupported parameters: [field\_attachment\_resource : {properties={name={type=attachment}}}]"}],"type":"mapper\_parsing\_exception","reason":"Root mapping definition has unsupported parameters: [field\_attachment\_resource : {properties={name={type=attachment}}}]"},"status":400}** Please advise on what I might be wrong. Thanks for your help!!!! ---
**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\ **Post date:** [June 1, 2018, 3:07pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/4 "2018-06-01T15:07:44Z")
At first glance this looks like the plugin didn't get installed properly. Have you restarted all nodes? Can you check if the hello-world example from [https://www.elastic.co/guide/en/elasticsearch/plugins/5.6/mapper-attachments-helloworld.html](https://www.elastic.co/guide/en/elasticsearch/plugins/5.6/mapper-attachments-helloworld.html) (or your ES version) works? ---
**Author:** ![silversun](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@silversun](https://discuss.elastic.co/u/silversun)\ **Post date:** [June 1, 2018, 3:46pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/5 "2018-06-01T15:46:23Z")
Sure, let me try that real quick. ---
**Author:** ![silversun](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@silversun](https://discuss.elastic.co/u/silversun)\ **Post date:** [June 1, 2018, 3:49pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/6 "2018-06-01T15:49:37Z")
I just tried the Hello World for Elastic search 2.4. Looks like it ran successfully. Please see below. ``` curl -X POST "localhost:9200/trying-out-mapper-attachments/person/_search" -H 'Content-Type: application/json' -d' { "query": { "query_string": { "query": "ipsum" > }}} > ' ``` > {"took":65,"timed\_out":false,"\_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max\_score":0.095891505,"hits":[{"\_index":"trying-out-mapper-attachments","\_type":"person","\_id":"1","\_score":0.095891505,"\_source": > { > "cv": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=" > } > }]}} ---
**Author:** ![silversun](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@silversun](https://discuss.elastic.co/u/silversun)\ **Post date:** [June 1, 2018, 3:57pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/7 "2018-06-01T15:57:36Z")
@cbuescher As Hello World looks like it is working well (response copied above). I wonder if my CURL command to update mapping of an existing field is not right. Could you please advise on that? ---
**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\ **Post date:** [June 1, 2018, 4:08pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/8 "2018-06-01T16:08:35Z")
> [@silversun](#): > > appindex/appindex/\_mapping From the docs it looks like the right incantation is ```auto PUT index/_mapping/type ``` So I guess in your case it might be `appindex/_mapping/appindex` (assuming `appindex` is also a type, which is a bit confusing) ---
**Author:** ![silversun](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@silversun](https://discuss.elastic.co/u/silversun)\ **Post date:** [June 1, 2018, 4:09pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/9 "2018-06-01T16:09:53Z")
This is what I have used to update mapping - ``` curl -X PUT 'http://localhost:9200/appindex/appindex/_mapping?ignore_conflicts=true' -d \ '{ "appindex": { "properties": { "field_attachment_resource:file:name": { "type": "attachment" } } } }' ``` It give me this error - `{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"mapper [field_attachment_resource:file:name] of different type, current_type [string], merged_type [attachment]"}],"type":"illegal_argument_exception","reason":"mapper [field_attachment_resource:file:name] of different type, current_type [string], merged_type [attachment]"},"status":400` ---
**Author:** ![silversun](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@silversun](https://discuss.elastic.co/u/silversun)\ **Post date:** [June 1, 2018, 4:14pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/10 "2018-06-01T16:14:08Z")
@cbuescher I got the same error for appindex/\_mapping/appindex - ` > {"error":{"root\_cause":[{"type":"illegal\_argument\_exception","reason":"mapper [field\_attachment\_resource:file:name] of different type, current\_type [string], merged\_type [attachment]"}],"type":"illegal\_argument\_exception","reason":"mapper [field\_attachment\_resource:file:name] of different type, current\_type [string], merged\_type [attachment]"},"status":400} ` This was the PUT command I used - ``` curl -X PUT 'http://localhost:9200/appindex/_mapping/appindex?ignore_conflicts=true' -d \ '{ "appindex": { "properties": { "field_attachment_resource:file:name": { "type": "attachment" } } } } ``` Please advise. ---
**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\ **Post date:** [June 2, 2018, 5:23pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/11 "2018-06-02T17:23:33Z")
This is a different kind of error now. Note that before you reported you get > [@silversun](#): > > "Root mapping definition has unsupported parameters while now the error message says: > [@silversun](#): > > "mapper [field\_attachment\_resource:file:name] of different type, current\_type [string], merged\_type [attachment]"}] This means the field `field_attachment_resource:file:name` was somehow defined before, which makes sense because you already mapped it to "String". You either need to use a different name now or create the mapping with the "attachment" type to begin with. ---
**Author:** ![silversun](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@silversun](https://discuss.elastic.co/u/silversun)\ **Post date:** [June 2, 2018, 6:30pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/12 "2018-06-02T18:30:01Z")
Could you please advise me on how I go about changing the type of an existing field? That error is a result of an attempt to change the mapping type from string to attachment ---
**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\ **Post date:** [June 3, 2018, 1:11pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/13 "2018-06-03T13:11:44Z")
Unfortunately you can't change an existing field after you have indexed something. You will have to put the correct mapping from the start. ---
**Author:** ![silversun](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@silversun](https://discuss.elastic.co/u/silversun)\ **Post date:** [June 3, 2018, 8:50pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/14 "2018-06-03T20:50:15Z")
Oh, really? That is highly disappointing. So, we cannot update a mapping once indexing has occurred? That is so inflexible of Elasticsearch. ---
**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\ **Post date:** [June 3, 2018, 9:26pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/15 "2018-06-03T21:26:17Z")
Sorry to disappoint here but this is a fundamental principle in Lucene, the underlying search library. What would happen otherwise to fields in documents that are already indexed? Should they be dropped? Overwritten? Ignored? Changed would be the best option, but this requires re-indexing, which we have a convenient API for. Please take a look at that. ---
**Author:** ![silversun](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@silversun](https://discuss.elastic.co/u/silversun)\ **Post date:** [June 4, 2018, 4:12pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/16 "2018-06-04T16:12:13Z")
@cbuescher I can certainly re-index it using the re-index API. [https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-reindex.html](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-reindex.html) But, I am confused - I can create a new index and call it new-index and re-index my data from old-index to new-index. But, where in this process, do I update the field type from "string" to "attachment"? Please advise. ---
**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\ **Post date:** [June 4, 2018, 6:16pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/17 "2018-06-04T18:16:07Z")
From that page: > Reindex does not attempt to set up the destination index. It does not copy the settings of the source index. You should set up the destination index prior to running a \_reindex action, including setting up mappings, shard counts, replicas, etc. ---
**Author:** ![silversun](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@silversun](https://discuss.elastic.co/u/silversun)\ **Post date:** [June 4, 2018, 6:22pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/18 "2018-06-04T18:22:35Z")
@cbuescher I created a new index and created a mapping with the same field name as that of the old index but of type "attachment" and then ran re-index API - Got this error - ``` { "index":"newindex-attachments", "type":"appindex", "id":"119", "cause":{ "type":"mapper_parsing_exception", "reason":"failed to parse", "caused_by":{ "type":"json_parse_exception", "reason":"Failed to decode VALUE_STRING as base64 (MIME-NO-LINEFEEDS): Illegal character '.' (code 0x2e) in base64 content\n at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@50ffdf13; line: 1, column: 748]" } }, "status":400 }, ``` Since the type of new field for the destination index is "attachment", it is looking for base64 text from the source index field when copying it over. I think that is why it is erroring out. Please advise how to go about this. ---
**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\ **Post date:** [June 4, 2018, 7:08pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/19 "2018-06-04T19:08:41Z")
This is exactly the reason why we don't allow changing a fields mapping on an existing index. You have documents in your old index where field "fooBar" is a String. Now you want field "fooBar" to be an attachment. You will need to decide what to do with the documents that already have strings in there (and they contain e.g. dots and other stuff). Do you want to delete them? Do that either on the source index before reindexing or filter them out from the reindex action. Do you want to replace the old value with something else? Then re-index won't do all the work for you, you will need to somehow do the replacement logic in an own client-side script. Maybe in this case its easiest to introduce a new field for the attachment and rewrite your application to use that? Lots of options, but it depends on what you want to do. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 2, 2018, 7:10pm UTC](https://discuss.elastic.co/t/extracting-full-text-from-an-attachment-field/133933/20 "2018-07-02T19:10:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.