# Attachment Plugin Questions on Storing **URL:** \ **Category:** Elasticsearch\ **Created:** [September 29, 2011, 6:44pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480 "2011-09-29T18:44:45Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Mike\_Gaffney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_gaffney/32/3108_2.png) [@Mike\_Gaffney](https://discuss.elastic.co/u/Mike_Gaffney)\ **Post date:** [September 29, 2011, 6:44pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/1 "2011-09-29T18:44:45Z")
I'm trying to make use of the attachments plugin. I've got the following Mapping: { "docs":{ "properties" : { "contents" : { "type" : "attachment", "fields" : { "contents" : {"store" : "no"} } }, "lastModified": { "type" : "long", "index" : "analyzed", "store" : "no"} } } } And the following index code: ``` XContentBuilder objectBuilder ``` = jsonBuilder().startObject(); ``` objectBuilder.startObject( ``` Index.CONTENTS); if (extension.equals("xml")){ objectBuilder.field("\_content\_type", MimeTypes.XML); } else { objectBuilder.field("\_content\_type", MimeTypes.PLAIN\_TEXT); } objectBuilder.field("\_name", file.getName()); objectBuilder.field("content", Base64.encodeBase64(FileUtils.readFileToString(file).getBytes())); objectBuilder.endObject(); objectBuilder.field(Index.LAST\_MODIFIED, file.lastModified()); objectBuilder.endObject(); IndexRequestBuilder setSource = client.prepareIndex(Index.INDEX, Index.TYPE, file.getAbsolutePath()).setSource(objectBuilder); setSource.execute().actionGet(); But when I look at the indexing on the server I see: { doc: { properties: { lastModified: { index: "analyzed" type: "long" } contents: { path: "full" type: "attachment" fields: { author: { type: "string" } title: { type: "string" } keywords: { type: "string" } contents: { type: "string" } date: { format: "dateOptionalTime" type: "date" } content\_type: { type: "string" } } } } } } Basically, I don't really want to store the contents, just index the documents and be able to search on them. I'm indexing files that are on the computer already so I don't need the contents, and in fact it's taking up a ton of space to have the contents in there. Another question is, the contents seem to just be the base64. Is that correct or am I doing something incorrectly. I'm using this as a local machine file search mechanism for a large art / document tree that each user has locally on their machines. My results look like this (sorry for the redactions, it's proprietary info: {"took":4,"timed\_out":false,"\_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1663,"max\_score":1.0,"hits":[{"\_index":"docs","\_type":"doc","\_id":"the\_art\_redacted","\_score":1.0,"fields":{"contents":{"content":"REALLY\_LONG\_BASE64\_STRING","\_name":"the\_file\_name\_redacted","\_content\_type":"text/plain"}}}]}} Any additional explanation of attachments will be quite helpful. Thanks, Mike ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [September 29, 2011, 6:52pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/2 "2011-09-29T18:52:01Z")
can you have a look in your logs for statements about mapping changes. It may be that you don't have everything specified in your mapping so it's getting overridden by dynamic mappings. You should see update mapping messages in your logs. On Thu, Sep 29, 2011 at 11:44 AM, Mike Gaffney [mr.gaffo@gmail.com](mailto:mr.gaffo@gmail.com) wrote: > I'm trying to make use of the attachments plugin. I've got the > following Mapping: > > { > "docs":{ > "properties" : { > "contents" : { > "type" : "attachment", > "fields" : { > "contents" : {"store" : "no"} > } > }, > "lastModified": { "type" : "long", "index" : > "analyzed", "store" : "no"} > } > } > } > > And the following index code: > > ``` > XContentBuilder > > ``` > > objectBuilder = jsonBuilder().startObject(); > > ``` > objectBuilder.startObject( > > ``` > > \*\*Index.CONTENTS); > if > (extension.equals("xml")){ > > objectBuilder.field("_content_\*\*type", MimeTypes.XML); > } > else { > > objectBuilder.field("_content_\*\*type", MimeTypes.PLAIN\_TEXT); > } > objectBuilder.field("\_name", > file.getName()); > > objectBuilder.field("content", > Base64.encodeBase64(FileUtils.\*\*readFileToString(file).\*\*getBytes())); > objectBuilder.endObject(); > > ``` > objectBuilder.field(Index.* > > ``` > > \*LAST\_MODIFIED, file.lastModified()); > objectBuilder.endObject(); > IndexRequestBuilder > setSource = client.prepareIndex(Index.\*\*INDEX, > Index.TYPE, file.getAbsolutePath()).\*\*setSource(objectBuilder); > > setSource.execute().actionGet(\*\*); > > But when I look at the indexing on the server I see: > { > doc: { > properties: { > lastModified: { > index: "analyzed" > type: "long" > } > contents: { > path: "full" > type: "attachment" > fields: { > author: { > type: "string" > } > title: { > type: "string" > } > keywords: { > type: "string" > } > contents: { > type: "string" > } > date: { > format: "dateOptionalTime" > type: "date" > } > content\_type: { > type: "string" > } > } > } > } > } > } > > Basically, I don't really want to store the contents, just index the > documents and be able to search on them. I'm indexing files that are > on the computer already so I don't need the contents, and in fact it's > taking up a ton of space to have the contents in there. > > Another question is, the contents seem to just be the base64. Is that > correct or am I doing something incorrectly. > > I'm using this as a local machine file search mechanism for a large > art / document tree that each user has locally on their machines. > > My results look like this (sorry for the redactions, it's proprietary info: > > {"took":4,"timed\_out":false,"_ **shards":{"total":5,"** > successful":5,"failed":0},"\*\*hits":{"total":1663,"max_\*\* > score":1.0,"hits":[{"\_index":" **docs","\_type":"doc","\_id":"** > the\_art\_redacted","_score":1. **0,"fields":{"contents":{"** > content":"REALLY\_LONG\_BASE64_ **STRING","_name":"the\_file_** > name\_redacted","\_content\_type"\*\*:"text/plain"}}}]}} > > Any additional explanation of attachments will be quite helpful. > > Thanks, > Mike ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Mike\_Gaffney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_gaffney/32/3108_2.png) [@Mike\_Gaffney](https://discuss.elastic.co/u/Mike_Gaffney)\ **Post date:** [September 29, 2011, 8:04pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/3 "2011-09-29T20:04:05Z")
On debug all I see related to mapping is: 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- pool-11-thread-1] DEBUG org.elasticsearch.index.mapper - [Ani-Mator] [docs] using dynamic[true], default mapping: location[null] and source[{ "_default_" : { } }] 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- pool-11-thread-1] DEBUG org.elasticsearch.index.cache.field.data.resident - [Ani-Mator] [docs] using [resident] field cache with max\_size [-1], expire [null] 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- pool-11-thread-1] DEBUG org.elasticsearch.index.cache - [Ani-Mator] [docs] Using stats.refresh\_interval [1s] 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- pool-11-thread-1] INFO org.elasticsearch.cluster.metadata - [Ani- Mator] [docs] creating index, cause [api], shards [5]/[1], mappings [doc] I'm guessing that using dynamic[true] means that I'm not doing it right. On Sep 29, 11:52 am, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > can you have a look in your logs for statements about mapping changes. It > may be that you don't have everything specified in your mapping so it's > getting overridden by dynamic mappings. > > You should see update mapping messages in your logs. > > On Thu, Sep 29, 2011 at 11:44 AM, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) wrote: > > > I'm trying to make use of the attachments plugin. I've got the > > following Mapping: > > > { > > "docs":{ > > "properties" : { > > "contents" : { > > "type" : "attachment", > > "fields" : { > > "contents" : {"store" : "no"} > > } > > }, > > "lastModified": { "type" : "long", "index" : > > "analyzed", "store" : "no"} > > } > > } > > } > > > And the following index code: > > > ``` > > XContentBuilder > > > > ``` > > > > objectBuilder = jsonBuilder().startObject(); > > > ``` > > objectBuilder.startObject( > > > > ``` > > > > \*\*Index.CONTENTS); > > if > > (extension.equals("xml")){ > > > objectBuilder.field("_content_\*\*type", MimeTypes.XML); > > } > > else { > > > objectBuilder.field("_content_\*\*type", MimeTypes.PLAIN\_TEXT); > > } > > objectBuilder.field("\_name", > > file.getName()); > > > objectBuilder.field("content", > > Base64.encodeBase64(FileUtils.\*\*readFileToString(file).\*\*getBytes())); > > objectBuilder.endObject(); > > > ``` > > objectBuilder.field(Index.* > > > > ``` > > > > \*LAST\_MODIFIED, file.lastModified()); > > objectBuilder.endObject(); > > IndexRequestBuilder > > setSource = client.prepareIndex(Index.\*\*INDEX, > > Index.TYPE, file.getAbsolutePath()).\*\*setSource(objectBuilder); > > > setSource.execute().actionGet(\*\*); > > > But when I look at the indexing on the server I see: > > { > > doc: { > > properties: { > > lastModified: { > > index: "analyzed" > > type: "long" > > } > > contents: { > > path: "full" > > type: "attachment" > > fields: { > > author: { > > type: "string" > > } > > title: { > > type: "string" > > } > > keywords: { > > type: "string" > > } > > contents: { > > type: "string" > > } > > date: { > > format: "dateOptionalTime" > > type: "date" > > } > > content\_type: { > > type: "string" > > } > > } > > } > > } > > } > > } > > > Basically, I don't really want to store the contents, just index the > > documents and be able to search on them. I'm indexing files that are > > on the computer already so I don't need the contents, and in fact it's > > taking up a ton of space to have the contents in there. > > > Another question is, the contents seem to just be the base64. Is that > > correct or am I doing something incorrectly. > > > I'm using this as a local machine file search mechanism for a large > > art / document tree that each user has locally on their machines. > > > My results look like this (sorry for the redactions, it's proprietary info: > > > {"took":4,"timed\_out":false,"_ **shards":{"total":5,"** > > successful":5,"failed":0},"\*\*hits":{"total":1663,"max_\*\* > > score":1.0,"hits":[{"\_index":" **docs","\_type":"doc","\_id":"** > > the\_art\_redacted","_score":1. **0,"fields":{"contents":{"** > > content":"REALLY\_LONG\_BASE64_ **STRING","_name":"the\_file_** > > name\_redacted","\_content\_type"\*\*:"text/plain"}}}]}} > > > Any additional explanation of attachments will be quite helpful. > > > Thanks, > > Mike > > ## -- > > Paul Loy > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [September 29, 2011, 8:08pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/4 "2011-09-29T20:08:36Z")
If you can gist full logs, mappings, settings, code, etc, (or as much as you can without giving away proprietary stuff) that's quite useful 😉 So at the end you have a create index[api] for docs. Do you push the mappings in there? Can I see that code? On Thu, Sep 29, 2011 at 1:04 PM, Mike Gaffney [mr.gaffo@gmail.com](mailto:mr.gaffo@gmail.com) wrote: > On debug all I see related to mapping is: > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > pool-11-thread-1] DEBUG org.elasticsearch.index.mapper - [Ani-Mator] > [docs] using dynamic[true], default mapping: location[null] and > source[{ > "_default_" : { > } > }] > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > pool-11-thread-1] DEBUG > org.elasticsearch.index.cache.field.data.resident - [Ani-Mator] > [docs] using [resident] field cache with max\_size [-1], expire [null] > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > pool-11-thread-1] DEBUG org.elasticsearch.index.cache - [Ani-Mator] > [docs] Using stats.refresh\_interval [1s] > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > pool-11-thread-1] INFO org.elasticsearch.cluster.metadata - [Ani- > Mator] [docs] creating index, cause [api], shards [5]/[1], mappings > [doc] > > I'm guessing that using dynamic[true] means that I'm not doing it > right. > > On Sep 29, 11:52 am, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > can you have a look in your logs for statements about mapping changes. It > > may be that you don't have everything specified in your mapping so it's > > getting overridden by dynamic mappings. > > > > You should see update mapping messages in your logs. > > > > On Thu, Sep 29, 2011 at 11:44 AM, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) > > wrote: > > > > > I'm trying to make use of the attachments plugin. I've got the > > > following Mapping: > > > > > { > > > "docs":{ > > > "properties" : { > > > "contents" : { > > > "type" : "attachment", > > > "fields" : { > > > "contents" : {"store" : "no"} > > > } > > > }, > > > "lastModified": { "type" : "long", "index" : > > > "analyzed", "store" : "no"} > > > } > > > } > > > } > > > > > And the following index code: > > > > > ``` > > > XContentBuilder > > > > > > ``` > > > > > > objectBuilder = jsonBuilder().startObject(); > > > > > > > objectBuilder.startObject( > > > > \*\*Index.CONTENTS); > > > if > > > (extension.equals("xml")){ > > > > > objectBuilder.field("_content_\*\*type", MimeTypes.XML); > > > } > > > else { > > > > > objectBuilder.field("_content_\*\*type", MimeTypes.PLAIN\_TEXT); > > > } > > objectBuilder.field("\_name", > > > > file.getName()); > > > > > objectBuilder.field("content", > > > Base64.encodeBase64(FileUtils.\*\*readFileToString(file).\*\*getBytes())); > > objectBuilder.endObject(); > > > > > > objectBuilder.field(Index.\* > > > > \*LAST\_MODIFIED, file.lastModified()); > > objectBuilder.endObject(); > > > > ``` > > > IndexRequestBuilder > > > > > > ``` > > > > > > setSource = client.prepareIndex(Index.\*\*INDEX, > > > Index.TYPE, file.getAbsolutePath()).\*\*setSource(objectBuilder); > > > > > setSource.execute().actionGet(\*\*); > > > > > But when I look at the indexing on the server I see: > > > { > > > doc: { > > > properties: { > > > lastModified: { > > > index: "analyzed" > > > type: "long" > > > } > > > contents: { > > > path: "full" > > > type: "attachment" > > > fields: { > > > author: { > > > type: "string" > > > } > > > title: { > > > type: "string" > > > } > > > keywords: { > > > type: "string" > > > } > > > contents: { > > > type: "string" > > > } > > > date: { > > > format: "dateOptionalTime" > > > type: "date" > > > } > > > content\_type: { > > > type: "string" > > > } > > > } > > > } > > > } > > > } > > > } > > > > > Basically, I don't really want to store the contents, just index the > > > documents and be able to search on them. I'm indexing files that are > > > on the computer already so I don't need the contents, and in fact it's > > > taking up a ton of space to have the contents in there. > > > > > Another question is, the contents seem to just be the base64. Is that > > > correct or am I doing something incorrectly. > > > > > I'm using this as a local machine file search mechanism for a large > > > art / document tree that each user has locally on their machines. > > > > > My results look like this (sorry for the redactions, it's proprietary > > > info: > > > > > {"took":4,"timed\_out":false,"_ **shards":{"total":5,"** > > > successful":5,"failed":0},"\*\*hits":{"total":1663,"max_\*\* > > > score":1.0,"hits":[{"\_index":" **docs","\_type":"doc","\_id":"** > > > the\_art\_redacted","_score":1. **0,"fields":{"contents":{"** > > > content":"REALLY\_LONG\_BASE64_ **STRING","_name":"the\_file_** > > > name\_redacted","\_content\_type"\*\*:"text/plain"}}}]}} > > > > > Any additional explanation of attachments will be quite helpful. > > > > > Thanks, > > > Mike > > > > ## -- > > > > Paul Loy > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Mike\_Gaffney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_gaffney/32/3108_2.png) [@Mike\_Gaffney](https://discuss.elastic.co/u/Mike_Gaffney)\ **Post date:** [September 29, 2011, 9:03pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/5 "2011-09-29T21:03:42Z")
> > > There are more than three files. show original On Sep 29, 1:08 pm, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > If you can gist full logs, mappings, settings, code, etc, (or as much as you > can without giving away proprietary stuff) that's quite useful 😉 > > So at the end you have a create index[api] for docs. Do you push the > mappings in there? Can I see that code? > > On Thu, Sep 29, 2011 at 1:04 PM, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) wrote: > > > On debug all I see related to mapping is: > > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > > pool-11-thread-1] DEBUG org.elasticsearch.index.mapper - [Ani-Mator] > > [docs] using dynamic[true], default mapping: location[null] and > > source[{ > > "_default_" : { > > } > > }] > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > > pool-11-thread-1] DEBUG > > org.elasticsearch.index.cache.field.data.resident - [Ani-Mator] > > [docs] using [resident] field cache with max\_size [-1], expire [null] > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > > pool-11-thread-1] DEBUG org.elasticsearch.index.cache - [Ani-Mator] > > [docs] Using stats.refresh\_interval [1s] > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > > pool-11-thread-1] INFO org.elasticsearch.cluster.metadata - [Ani- > > Mator] [docs] creating index, cause [api], shards [5]/[1], mappings > > [doc] > > > I'm guessing that using dynamic[true] means that I'm not doing it > > right. > > > On Sep 29, 11:52 am, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > can you have a look in your logs for statements about mapping changes. It > > > may be that you don't have everything specified in your mapping so it's > > > getting overridden by dynamic mappings. > > > > You should see update mapping messages in your logs. > > > > On Thu, Sep 29, 2011 at 11:44 AM, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) > > > wrote: > > > > > > > I'm trying to make use of the attachments plugin. I've got the > > > > following Mapping: > > > > > { > > > > "docs":{ > > > > "properties" : { > > > > "contents" : { > > > > "type" : "attachment", > > > > "fields" : { > > > > "contents" : {"store" : "no"} > > > > } > > > > }, > > > > "lastModified": { "type" : "long", "index" : > > > > "analyzed", "store" : "no"} > > > > } > > > > } > > > > } > > > > > And the following index code: > > > > > ``` > > > > XContentBuilder > > > > > > > > ``` > > > > > > > > objectBuilder = jsonBuilder().startObject(); > > > objectBuilder.startObject( > > > > > > \*\*Index.CONTENTS); > > > > if > > > > (extension.equals("xml")){ > > > > > objectBuilder.field("_content_\*\*type", MimeTypes.XML); > > > > } > > > > else { > > > > > objectBuilder.field("_content_\*\*type", MimeTypes.PLAIN\_TEXT); > > > > } > > > objectBuilder.field("\_name", > > > > > > file.getName()); > > > > > objectBuilder.field("content", > > > > Base64.encodeBase64(FileUtils.\*\*readFileToString(file).\*\*getBytes())); > > > objectBuilder.endObject(); > > > objectBuilder.field(Index.\* > > > > > > \*LAST\_MODIFIED, file.lastModified()); > > > objectBuilder.endObject(); > > > > > > ``` > > > > IndexRequestBuilder > > > > > > > > ``` > > > > > > > > setSource = client.prepareIndex(Index.\*\*INDEX, > > > > Index.TYPE, file.getAbsolutePath()).\*\*setSource(objectBuilder); > > > > > setSource.execute().actionGet(\*\*); > > > > > But when I look at the indexing on the server I see: > > > > { > > > > doc: { > > > > properties: { > > > > lastModified: { > > > > index: "analyzed" > > > > type: "long" > > > > } > > > > contents: { > > > > path: "full" > > > > type: "attachment" > > > > fields: { > > > > author: { > > > > type: "string" > > > > } > > > > title: { > > > > type: "string" > > > > } > > > > keywords: { > > > > type: "string" > > > > } > > > > contents: { > > > > type: "string" > > > > } > > > > date: { > > > > format: "dateOptionalTime" > > > > type: "date" > > > > } > > > > content\_type: { > > > > type: "string" > > > > } > > > > } > > > > } > > > > } > > > > } > > > > } > > > > > Basically, I don't really want to store the contents, just index the > > > > documents and be able to search on them. I'm indexing files that are > > > > on the computer already so I don't need the contents, and in fact it's > > > > taking up a ton of space to have the contents in there. > > > > > Another question is, the contents seem to just be the base64. Is that > > > > correct or am I doing something incorrectly. > > > > > I'm using this as a local machine file search mechanism for a large > > > > art / document tree that each user has locally on their machines. > > > > > My results look like this (sorry for the redactions, it's proprietary > > > > info: > > > > > {"took":4,"timed\_out":false,"_ **shards":{"total":5,"** > > > > successful":5,"failed":0},"\*\*hits":{"total":1663,"max_\*\* > > > > score":1.0,"hits":[{"\_index":" **docs","\_type":"doc","\_id":"** > > > > the\_art\_redacted","_score":1. **0,"fields":{"contents":{"** > > > > content":"REALLY\_LONG\_BASE64_ **STRING","_name":"the\_file_** > > > > name\_redacted","\_content\_type"\*\*:"text/plain"}}}]}} > > > > > Any additional explanation of attachments will be quite helpful. > > > > > Thanks, > > > > Mike > > > > ## -- > > > > > > Paul Loy > > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy > > ## -- > > Paul Loy > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [September 29, 2011, 9:08pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/6 "2011-09-29T21:08:20Z")
I would have expected that the following line would have caused put mapping, cause [api] (or something similar in the logs): InputStream docsMappings = IndexerMain.class.getResourceAsStream("/mappings/docs.json"); On Thu, Sep 29, 2011 at 2:03 PM, Mike Gaffney [mr.gaffo@gmail.com](mailto:mr.gaffo@gmail.com) wrote: > [IndexerMain.java · GitHub](https://gist.github.com/1251943) > > On Sep 29, 1:08 pm, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > If you can gist full logs, mappings, settings, code, etc, (or as much as > > you > > can without giving away proprietary stuff) that's quite useful 😉 > > > > So at the end you have a create index[api] for docs. Do you push the > > mappings in there? Can I see that code? > > > > On Thu, Sep 29, 2011 at 1:04 PM, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) > > wrote: > > > > > On debug all I see related to mapping is: > > > > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > > > pool-11-thread-1] DEBUG org.elasticsearch.index.mapper - [Ani-Mator] > > > [docs] using dynamic[true], default mapping: location[null] and > > > source[{ > > > "_default_" : { > > > } > > > }] > > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > > > pool-11-thread-1] DEBUG > > > org.elasticsearch.index.cache.field.data.resident - [Ani-Mator] > > > [docs] using [resident] field cache with max\_size [-1], expire [null] > > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > > > pool-11-thread-1] DEBUG org.elasticsearch.index.cache - [Ani-Mator] > > > [docs] Using stats.refresh\_interval [1s] > > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > > > pool-11-thread-1] INFO org.elasticsearch.cluster.metadata - [Ani- > > > Mator] [docs] creating index, cause [api], shards [5]/[1], mappings > > > [doc] > > > > > I'm guessing that using dynamic[true] means that I'm not doing it > > > right. > > > > > On Sep 29, 11:52 am, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > > > can you have a look in your logs for statements about mapping > > > > changes. It > > > > may be that you don't have everything specified in your mapping so > > > > it's > > > > getting overridden by dynamic mappings. > > > > > > You should see update mapping messages in your logs. > > > > > > On Thu, Sep 29, 2011 at 11:44 AM, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) > > > > wrote: > > > > > > > > > I'm trying to make use of the attachments plugin. I've got the > > > > > following Mapping: > > > > > > > { > > > > > "docs":{ > > > > > "properties" : { > > > > > "contents" : { > > > > > "type" : "attachment", > > > > > "fields" : { > > > > > "contents" : {"store" : > > > > > "no"} > > > > > } > > > > > }, > > > > > "lastModified": { "type" : "long", "index" : > > > > > "analyzed", "store" : "no"} > > > > > } > > > > > } > > > > > } > > > > > > > And the following index code: > > > > > > > ``` > > > > > XContentBuilder > > > > > > > > > > ``` > > > > > > > > > > objectBuilder = jsonBuilder().startObject(); > > > > > objectBuilder.startObject( > > > > > > > > \*\*Index.CONTENTS); > > > > > if > > > > > (extension.equals("xml")){ > > > > > > > objectBuilder.field("_content_\*\*type", MimeTypes.XML); > > > > > } > > > > > else { > > > > > > > objectBuilder.field("_content_\*\*type", MimeTypes.PLAIN\_TEXT); > > > > > } > > > > > objectBuilder.field("\_name", > > > > > > > > file.getName()); > > > > > > > objectBuilder.field("content", > > Base64.encodeBase64(FileUtils.\*\*readFileToString(file).\*\*getBytes())); > > > > objectBuilder.endObject(); > > > > > objectBuilder.field(Index.\* > > > > > > > > \*LAST\_MODIFIED, file.lastModified()); > > > > > objectBuilder.endObject(); > > > > > > > > ``` > > > > > IndexRequestBuilder > > > > > > > > > > ``` > > > > > > > > > > setSource = client.prepareIndex(Index.\*\*INDEX, > > > > > Index.TYPE, file.getAbsolutePath()).\*\*setSource(objectBuilder); > > > > > > > setSource.execute().actionGet(\*\*); > > > > > > > But when I look at the indexing on the server I see: > > > > > { > > > > > doc: { > > > > > properties: { > > > > > lastModified: { > > > > > index: "analyzed" > > > > > type: "long" > > > > > } > > > > > contents: { > > > > > path: "full" > > > > > type: "attachment" > > > > > fields: { > > > > > author: { > > > > > type: "string" > > > > > } > > > > > title: { > > > > > type: "string" > > > > > } > > > > > keywords: { > > > > > type: "string" > > > > > } > > > > > contents: { > > > > > type: "string" > > > > > } > > > > > date: { > > > > > format: "dateOptionalTime" > > > > > type: "date" > > > > > } > > > > > content\_type: { > > > > > type: "string" > > > > > } > > > > > } > > > > > } > > > > > } > > > > > } > > > > > } > > > > > > > Basically, I don't really want to store the contents, just index > > > > > the > > > > > documents and be able to search on them. I'm indexing files that > > > > > are > > > > > on the computer already so I don't need the contents, and in fact > > > > > it's > > > > > taking up a ton of space to have the contents in there. > > > > > > > Another question is, the contents seem to just be the base64. Is > > > > > that > > > > > correct or am I doing something incorrectly. > > > > > > > I'm using this as a local machine file search mechanism for a large > > > > > art / document tree that each user has locally on their machines. > > > > > > > My results look like this (sorry for the redactions, it's > > > > > proprietary > > > > > info: > > > > > > > {"took":4,"timed\_out":false,"_ **shards":{"total":5,"** > > > > > successful":5,"failed":0},"\*\*hits":{"total":1663,"max_\*\* > > > > > score":1.0,"hits":[{"\_index":" **docs","\_type":"doc","\_id":"** > > > > > the\_art\_redacted","_score":1. **0,"fields":{"contents":{"** > > > > > content":"REALLY\_LONG\_BASE64_ **STRING","_name":"the\_file_** > > > > > name\_redacted","\_content\_type"\*\*:"text/plain"}}}]}} > > > > > > > Any additional explanation of attachments will be quite helpful. > > > > > > > Thanks, > > > > > Mike > > > > > > ## -- > > > > > > > > Paul Loy > > > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy > > > > ## -- > > > > Paul Loy > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [September 29, 2011, 9:08pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/7 "2011-09-29T21:08:51Z")
or rather the block: ``` InputStream docsMappings = ``` IndexerMain.class.getResourceAsStream("/mappings/docs.json"); String docsMappingAsString = IOUtils.toString(docsMappings); prepareCreate.addMapping(Index.TYPE, docsMappingAsString); prepareCreate.execute().actionGet(); On Thu, Sep 29, 2011 at 2:08 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > I would have expected that the following line would have caused put > mapping, cause [api] (or something similar in the logs): > > InputStream docsMappings = IndexerMain.class.getResourceAsStream("/mappings/docs.json"); > > On Thu, Sep 29, 2011 at 2:03 PM, Mike Gaffney [mr.gaffo@gmail.com](mailto:mr.gaffo@gmail.com) wrote: > > > [IndexerMain.java · GitHub](https://gist.github.com/1251943) > > > > On Sep 29, 1:08 pm, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > If you can gist full logs, mappings, settings, code, etc, (or as much as > > > you > > > can without giving away proprietary stuff) that's quite useful 😉 > > > > > > So at the end you have a create index[api] for docs. Do you push the > > > mappings in there? Can I see that code? > > > > > > On Thu, Sep 29, 2011 at 1:04 PM, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) > > > wrote: > > > > > > > On debug all I see related to mapping is: > > > > > > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > > > > pool-11-thread-1] DEBUG org.elasticsearch.index.mapper - [Ani-Mator] > > > > [docs] using dynamic[true], default mapping: location[null] and > > > > source[{ > > > > "_default_" : { > > > > } > > > > }] > > > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > > > > pool-11-thread-1] DEBUG > > > > org.elasticsearch.index.cache.field.data.resident - [Ani-Mator] > > > > [docs] using [resident] field cache with max\_size [-1], expire [null] > > > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > > > > pool-11-thread-1] DEBUG org.elasticsearch.index.cache - [Ani-Mator] > > > > [docs] Using stats.refresh\_interval [1s] > > > > 20110929-12:59:45 [elasticsearch[Ani-Mator]clusterService#updateTask- > > > > pool-11-thread-1] INFO org.elasticsearch.cluster.metadata - [Ani- > > > > Mator] [docs] creating index, cause [api], shards [5]/[1], mappings > > > > [doc] > > > > > > > I'm guessing that using dynamic[true] means that I'm not doing it > > > > right. > > > > > > > On Sep 29, 11:52 am, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > > > > > can you have a look in your logs for statements about mapping > > > > > changes. It > > > > > may be that you don't have everything specified in your mapping so > > > > > it's > > > > > getting overridden by dynamic mappings. > > > > > > > > You should see update mapping messages in your logs. > > > > > > > > On Thu, Sep 29, 2011 at 11:44 AM, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) > > > > > wrote: > > > > > > > > > > > I'm trying to make use of the attachments plugin. I've got the > > > > > > following Mapping: > > > > > > > > > { > > > > > > "docs":{ > > > > > > "properties" : { > > > > > > "contents" : { > > > > > > "type" : "attachment", > > > > > > "fields" : { > > > > > > "contents" : {"store" : > > > > > > "no"} > > > > > > } > > > > > > }, > > > > > > "lastModified": { "type" : "long", "index" > > > > > > : > > > > > > "analyzed", "store" : "no"} > > > > > > } > > > > > > } > > > > > > } > > > > > > > > > And the following index code: > > > > > > > > > ``` > > > > > > XContentBuilder > > > > > > > > > > > > ``` > > > > > > > > > > > > objectBuilder = jsonBuilder().startObject(); > > > > > > > objectBuilder.startObject( > > > > > > > > > > \*\*Index.CONTENTS); > > > > > > if > > > > > > (extension.equals("xml")){ > > > > > > > > > objectBuilder.field("_content_\*\*type", MimeTypes.XML); > > > > > > } > > > > > > else { > > > > > > > > > objectBuilder.field("_content_\*\*type", MimeTypes.PLAIN\_TEXT); > > > > > > } > > > > > > > objectBuilder.field("\_name", > > > > > > > > > > file.getName()); > > > > > > > > > objectBuilder.field("content", > > > > Base64.encodeBase64(FileUtils.\*\*readFileToString(file).\*\*getBytes())); > > > > > > objectBuilder.endObject(); > > > > > > > objectBuilder.field(Index.\* > > > > > > > > > > \*LAST\_MODIFIED, file.lastModified()); > > > > > > > objectBuilder.endObject(); > > > > > > > > > > ``` > > > > > > IndexRequestBuilder > > > > > > > > > > > > ``` > > > > > > > > > > > > setSource = client.prepareIndex(Index.\*\*INDEX, > > > > > > Index.TYPE, file.getAbsolutePath()).\*\*setSource(objectBuilder); > > > > > > > > > setSource.execute().actionGet(\*\*); > > > > > > > > > But when I look at the indexing on the server I see: > > > > > > { > > > > > > doc: { > > > > > > properties: { > > > > > > lastModified: { > > > > > > index: "analyzed" > > > > > > type: "long" > > > > > > } > > > > > > contents: { > > > > > > path: "full" > > > > > > type: "attachment" > > > > > > fields: { > > > > > > author: { > > > > > > type: "string" > > > > > > } > > > > > > title: { > > > > > > type: "string" > > > > > > } > > > > > > keywords: { > > > > > > type: "string" > > > > > > } > > > > > > contents: { > > > > > > type: "string" > > > > > > } > > > > > > date: { > > > > > > format: "dateOptionalTime" > > > > > > type: "date" > > > > > > } > > > > > > content\_type: { > > > > > > type: "string" > > > > > > } > > > > > > } > > > > > > } > > > > > > } > > > > > > } > > > > > > } > > > > > > > > > Basically, I don't really want to store the contents, just index > > > > > > the > > > > > > documents and be able to search on them. I'm indexing files that > > > > > > are > > > > > > on the computer already so I don't need the contents, and in fact > > > > > > it's > > > > > > taking up a ton of space to have the contents in there. > > > > > > > > > Another question is, the contents seem to just be the base64. Is > > > > > > that > > > > > > correct or am I doing something incorrectly. > > > > > > > > > I'm using this as a local machine file search mechanism for a > > > > > > large > > > > > > art / document tree that each user has locally on their machines. > > > > > > > > > My results look like this (sorry for the redactions, it's > > > > > > proprietary > > > > > > info: > > > > > > > > > {"took":4,"timed\_out":false,"_ **shards":{"total":5,"** > > > > > > successful":5,"failed":0},"\*\*hits":{"total":1663,"max_\*\* > > > > > > score":1.0,"hits":[{"\_index":" **docs","\_type":"doc","\_id":"** > > > > > > the\_art\_redacted","_score":1. **0,"fields":{"contents":{"** > > > > > > content":"REALLY\_LONG\_BASE64_ **STRING","_name":"the\_file_** > > > > > > name\_redacted","\_content\_type"\*\*:"text/plain"}}}]}} > > > > > > > > > Any additional explanation of attachments will be quite helpful. > > > > > > > > > Thanks, > > > > > > Mike > > > > > > > > ## -- > > > > > > > > > > Paul Loy > > > > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy > > > > > > ## -- > > > > > > Paul Loy > > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy > > ## -- > > Paul Loy > [paul@keteracel.com](mailto:paul@keteracel.com) > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Mike\_Gaffney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_gaffney/32/3108_2.png) [@Mike\_Gaffney](https://discuss.elastic.co/u/Mike_Gaffney)\ **Post date:** [September 29, 2011, 9:25pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/8 "2011-09-29T21:25:46Z")
added the log output. There is a create by api that happens. But not much else that I can tell. ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [September 29, 2011, 9:39pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/9 "2011-09-29T21:39:59Z")
20110929-14:22:17 [elasticsearch[Xi'an Chi Xan]clusterService#updateTask-pool-11-thread-1] DEBUG org.elasticsearch.index.mapper - [Xi'an Chi Xan] [docs] using dynamic[true], default mapping: location[null] and source[{ "_default_" : { } }] So yeah, that doesn't look good. Can you try putting the mapping after creating the index? On Thu, Sep 29, 2011 at 2:25 PM, Mike Gaffney [mr.gaffo@gmail.com](mailto:mr.gaffo@gmail.com) wrote: > added the log output. There is a create by api that happens. But not > much else that I can tell. ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Mike\_Gaffney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_gaffney/32/3108_2.png) [@Mike\_Gaffney](https://discuss.elastic.co/u/Mike_Gaffney)\ **Post date:** [September 29, 2011, 11:12pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/10 "2011-09-29T23:12:33Z")
Done. I get this log output: 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] TRACE org.elasticsearch.index.shard.service - [White Tiger] [docs][4] refresh with waitForOperations[false] 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG org.elasticsearch.index.gateway - [White Tiger] [docs][4] recovery completed from local, took [6ms] index : files [0] with total\_size [0b], took[1ms] : recovered\_files [0] with total\_size [0b] : reusing\_files [0] with total\_size [0b] translog : number\_of\_operations [0], took [6ms] 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG org.elasticsearch.cluster.action.shard - [White Tiger] sending shard started for [docs][4], node[LK1HamGVSqCXrOxh7zr8yA], [P], s[INITIALIZING], reason [after recovery from gateway] 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG org.elasticsearch.cluster.action.shard - [White Tiger] received shard started for [docs][4], node[LK1HamGVSqCXrOxh7zr8yA], [P], s[INITIALIZING], reason [after recovery from gateway] 20110929-15:02:25 [elasticsearch[White Tiger]clusterService#updateTask- pool-11-thread-1] DEBUG org.elasticsearch.cluster.metadata - [White Tiger] [docs] create\_mapping [doc] with source [{"doc":{"properties": {"contents":{"type":"attachment","path":"full","fields":{"contents": {"type":"string"},"author":{"type":"string"},"title": {"type":"string"},"date": {"type":"date","format":"dateOptionalTime"},"keywords": {"type":"string"},"content\_type":{"type":"string"}}},"lastModified": {"type":"long","index":"analyzed"}}}}] ... added 20110929-15:02:25 [elasticsearch[White Tiger]clusterService#updateTask- pool-11-thread-1] TRACE org.elasticsearch.cluster.service - [White Tiger] cluster state updated: version [5], source [put-mapping [doc]] nodes: with this config: { "docs":{ "properties" : { "contents": { "type" : "attachment", "path":"full", "store": "no", "fields":{ "contents":{"type":"string", "store": "no", "index":"analyzed"}, "author":{"type":"string"}, "title":{"type":"string"}, "date": {"type":"date","store":"no","format":"dateOptionalTime"}, "keywords":{"type":"string"}, "content\_type":{"type":"string"} } }, "lastModified": { "type" : "long", "index" : "analyzed", "store" : "no"} } } } ---
**Author:** ![Mike\_Gaffney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_gaffney/32/3108_2.png) [@Mike\_Gaffney](https://discuss.elastic.co/u/Mike_Gaffney)\ **Post date:** [October 12, 2011, 6:39pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/11 "2011-10-12T18:39:14Z")
I never got the attachment system to stop storing the full document base64 in the db or from returning it in results. The results isn't that big of a deal, but I can't really store all of the documents 2x (once for real and once in the index). We're already harddrive constrained as we are. Shay do you have any thoughts? On Sep 29, 4:12 pm, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) wrote: > Done. I get this log output: > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] TRACE > org.elasticsearch.index.shard.service - [White Tiger] [docs][4] > refresh with waitForOperations[false] > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG > org.elasticsearch.index.gateway - [White Tiger] [docs][4] recovery > completed from local, took [6ms] > index : files [0] with total\_size [0b], took[1ms] > : recovered\_files [0] with total\_size [0b] > : reusing\_files [0] with total\_size [0b] > translog : number\_of\_operations [0], took [6ms] > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG > org.elasticsearch.cluster.action.shard - [White Tiger] sending shard > started for [docs][4], node[LK1HamGVSqCXrOxh7zr8yA], [P], > s[INITIALIZING], reason [after recovery from gateway] > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG > org.elasticsearch.cluster.action.shard - [White Tiger] received shard > started for [docs][4], node[LK1HamGVSqCXrOxh7zr8yA], [P], > s[INITIALIZING], reason [after recovery from gateway] > 20110929-15:02:25 [elasticsearch[White Tiger]clusterService#updateTask- > pool-11-thread-1] DEBUG org.elasticsearch.cluster.metadata - [White > Tiger] [docs] create\_mapping [doc] with source [{"doc":{"properties": > {"contents":{"type":"attachment","path":"full","fields":{"contents": > {"type":"string"},"author":{"type":"string"},"title": > {"type":"string"},"date": > {"type":"date","format":"dateOptionalTime"},"keywords": > {"type":"string"},"content\_type":{"type":"string"}}},"lastModified": > {"type":"long","index":"analyzed"}}}}] > ... added > 20110929-15:02:25 [elasticsearch[White Tiger]clusterService#updateTask- > pool-11-thread-1] TRACE org.elasticsearch.cluster.service - [White > Tiger] cluster state updated: > version [5], source [put-mapping [doc]] > nodes: > > with this config: > > { > "docs":{ > "properties" : { > "contents": { > "type" : "attachment", > "path":"full", > "store": "no", > "fields":{ > "contents":{"type":"string", "store": "no", > "index":"analyzed"}, > "author":{"type":"string"}, > "title":{"type":"string"}, > "date": > {"type":"date","store":"no","format":"dateOptionalTime"}, > "keywords":{"type":"string"}, > "content\_type":{"type":"string"} > } > }, > "lastModified": { "type" : "long", "index" : "analyzed", "store" : > "no"} > } > } > > } ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [October 12, 2011, 7:22pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/12 "2011-10-12T19:22:04Z")
Hi, may be there are other possibilities, but, you can completely disable storing the \_source [1] and you can also return only selected fields [2] in the search results. [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/source-field.html) [2] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/fields.html) However, your request to disable storing only the attachments base64 data might be reasonable. You are probably not the only user requesting this. On the other hand, this can make things more complicated later because the compete document source may not be available for re-indexing. This is probably up to Shay whether he wants to allow this or not, you can always clone mapper-attachments plugin and do your customizations and try to sent pull request. just my 2 cents. Regards, Lukas On Wed, Oct 12, 2011 at 8:39 PM, Mike Gaffney [mr.gaffo@gmail.com](mailto:mr.gaffo@gmail.com) wrote: > I never got the attachment system to stop storing the full document > base64 in the db or from returning it in results. The results isn't > that big of a deal, but I can't really store all of the documents 2x > (once for real and once in the index). We're already harddrive > constrained as we are. > > Shay do you have any thoughts? > > On Sep 29, 4:12 pm, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) wrote: > > > Done. I get this log output: > > > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] TRACE > > org.elasticsearch.index.shard.service - [White Tiger] [docs][4] > > refresh with waitForOperations[false] > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG > > org.elasticsearch.index.gateway - [White Tiger] [docs][4] recovery > > completed from local, took [6ms] > > index : files [0] with total\_size [0b], took[1ms] > > : recovered\_files [0] with total\_size [0b] > > : reusing\_files [0] with total\_size [0b] > > translog : number\_of\_operations [0], took [6ms] > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG > > org.elasticsearch.cluster.action.shard - [White Tiger] sending shard > > started for [docs][4], node[LK1HamGVSqCXrOxh7zr8yA], [P], > > s[INITIALIZING], reason [after recovery from gateway] > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG > > org.elasticsearch.cluster.action.shard - [White Tiger] received shard > > started for [docs][4], node[LK1HamGVSqCXrOxh7zr8yA], [P], > > s[INITIALIZING], reason [after recovery from gateway] > > 20110929-15:02:25 [elasticsearch[White Tiger]clusterService#updateTask- > > pool-11-thread-1] DEBUG org.elasticsearch.cluster.metadata - [White > > Tiger] [docs] create\_mapping [doc] with source [{"doc":{"properties": > > {"contents":{"type":"attachment","path":"full","fields":{"contents": > > {"type":"string"},"author":{"type":"string"},"title": > > {"type":"string"},"date": > > {"type":"date","format":"dateOptionalTime"},"keywords": > > {"type":"string"},"content\_type":{"type":"string"}}},"lastModified": > > {"type":"long","index":"analyzed"}}}}] > > ... added > > 20110929-15:02:25 [elasticsearch[White Tiger]clusterService#updateTask- > > pool-11-thread-1] TRACE org.elasticsearch.cluster.service - [White > > Tiger] cluster state updated: > > version [5], source [put-mapping [doc]] > > nodes: > > > > with this config: > > > > { > > "docs":{ > > "properties" : { > > "contents": { > > "type" : "attachment", > > "path":"full", > > "store": "no", > > "fields":{ > > "contents":{"type":"string", "store": > > "no", > > "index":"analyzed"}, > > "author":{"type":"string"}, > > "title":{"type":"string"}, > > "date": > > {"type":"date","store":"no","format":"dateOptionalTime"}, > > "keywords":{"type":"string"}, > > "content\_type":{"type":"string"} > > } > > }, > > "lastModified": { "type" : "long", "index" : > > "analyzed", "store" : > > "no"} > > } > > } > > > > } ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 12, 2011, 9:32pm UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/13 "2011-10-12T21:32:12Z")
Currently, you can disable \_source and thus the attachment will not be stored as well. There is no option to "remove" the attachment from \_source (the json doc) and store in the \_source everything _but_ the attachment. On Wed, Oct 12, 2011 at 9:22 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > Hi, > > may be there are other possibilities, but, you can completely disable > storing the \_source [1] and you can also return only selected fields [2] in > the search results. > > [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/source-field.html) > [2] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/fields.html) > > However, your request to disable storing only the attachments base64 data > might be reasonable. You are probably not the only user requesting this. On > the other hand, this can make things more complicated later because the > compete document source may not be available for re-indexing. This is > probably up to Shay whether he wants to allow this or not, you can always > clone mapper-attachments plugin and do your customizations and try to sent > pull request. > > just my 2 cents. > > Regards, > Lukas > > On Wed, Oct 12, 2011 at 8:39 PM, Mike Gaffney [mr.gaffo@gmail.com](mailto:mr.gaffo@gmail.com) wrote: > > > I never got the attachment system to stop storing the full document > > base64 in the db or from returning it in results. The results isn't > > that big of a deal, but I can't really store all of the documents 2x > > (once for real and once in the index). We're already harddrive > > constrained as we are. > > > > Shay do you have any thoughts? > > > > On Sep 29, 4:12 pm, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) wrote: > > > > > Done. I get this log output: > > > > > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] TRACE > > > org.elasticsearch.index.shard.service - [White Tiger] [docs][4] > > > refresh with waitForOperations[false] > > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG > > > org.elasticsearch.index.gateway - [White Tiger] [docs][4] recovery > > > completed from local, took [6ms] > > > index : files [0] with total\_size [0b], took[1ms] > > > : recovered\_files [0] with total\_size [0b] > > > : reusing\_files [0] with total\_size [0b] > > > translog : number\_of\_operations [0], took [6ms] > > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG > > > org.elasticsearch.cluster.action.shard - [White Tiger] sending shard > > > started for [docs][4], node[LK1HamGVSqCXrOxh7zr8yA], [P], > > > s[INITIALIZING], reason [after recovery from gateway] > > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG > > > org.elasticsearch.cluster.action.shard - [White Tiger] received shard > > > started for [docs][4], node[LK1HamGVSqCXrOxh7zr8yA], [P], > > > s[INITIALIZING], reason [after recovery from gateway] > > > 20110929-15:02:25 [elasticsearch[White Tiger]clusterService#updateTask- > > > pool-11-thread-1] DEBUG org.elasticsearch.cluster.metadata - [White > > > Tiger] [docs] create\_mapping [doc] with source [{"doc":{"properties": > > > {"contents":{"type":"attachment","path":"full","fields":{"contents": > > > {"type":"string"},"author":{"type":"string"},"title": > > > {"type":"string"},"date": > > > {"type":"date","format":"dateOptionalTime"},"keywords": > > > {"type":"string"},"content\_type":{"type":"string"}}},"lastModified": > > > {"type":"long","index":"analyzed"}}}}] > > > ... added > > > 20110929-15:02:25 [elasticsearch[White Tiger]clusterService#updateTask- > > > pool-11-thread-1] TRACE org.elasticsearch.cluster.service - [White > > > Tiger] cluster state updated: > > > version [5], source [put-mapping [doc]] > > > nodes: > > > > > > with this config: > > > > > > { > > > "docs":{ > > > "properties" : { > > > "contents": { > > > "type" : "attachment", > > > "path":"full", > > > "store": "no", > > > "fields":{ > > > "contents":{"type":"string", "store": > > > "no", > > > "index":"analyzed"}, > > > "author":{"type":"string"}, > > > "title":{"type":"string"}, > > > "date": > > > {"type":"date","store":"no","format":"dateOptionalTime"}, > > > "keywords":{"type":"string"}, > > > "content\_type":{"type":"string"} > > > } > > > }, > > > "lastModified": { "type" : "long", "index" : > > > "analyzed", "store" : > > > "no"} > > > } > > > } > > > > > > } ---
**Author:** ![Mike\_Gaffney](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_gaffney/32/3108_2.png) [@Mike\_Gaffney](https://discuss.elastic.co/u/Mike_Gaffney)\ **Post date:** [October 13, 2011, 1:24am UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/14 "2011-10-13T01:24:14Z")
Thanks for the clarification guys! Good enough for what I'm doing On Oct 12, 2:32 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Currently, you can disable \_source and thus the attachment will not be > stored as well. There is no option to "remove" the attachment from \_source > (the json doc) and store in the \_source everything _but_ the attachment. > > On Wed, Oct 12, 2011 at 9:22 PM, Lukáš Vlček [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote: > > > Hi, > > > may be there are other possibilities, but, you can completely disable > > storing the \_source [1] and you can also return only selected fields [2] in > > the search results. > > > [1][Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/source-field.html) > > [2][Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/fields.html) > > > However, your request to disable storing only the attachments base64 data > > might be reasonable. You are probably not the only user requesting this. On > > the other hand, this can make things more complicated later because the > > compete document source may not be available for re-indexing. This is > > probably up to Shay whether he wants to allow this or not, you can always > > clone mapper-attachments plugin and do your customizations and try to sent > > pull request. > > > just my 2 cents. > > > Regards, > > Lukas > > > On Wed, Oct 12, 2011 at 8:39 PM, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) wrote: > > > > I never got the attachment system to stop storing the full document > > > base64 in the db or from returning it in results. The results isn't > > > that big of a deal, but I can't really store all of the documents 2x > > > (once for real and once in the index). We're already harddrive > > > constrained as we are. > > > > Shay do you have any thoughts? > > > > On Sep 29, 4:12 pm, Mike Gaffney [mr.ga...@gmail.com](mailto:mr.ga...@gmail.com) wrote: > > > > > > > Done. I get this log output: > > > > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] TRACE > > > > org.elasticsearch.index.shard.service - [White Tiger] [docs][4] > > > > refresh with waitForOperations[false] > > > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG > > > > org.elasticsearch.index.gateway - [White Tiger] [docs][4] recovery > > > > completed from local, took [6ms] > > > > index : files [0] with total\_size [0b], took[1ms] > > > > : recovered\_files [0] with total\_size [0b] > > > > : reusing\_files [0] with total\_size [0b] > > > > translog : number\_of\_operations [0], took [6ms] > > > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG > > > > org.elasticsearch.cluster.action.shard - [White Tiger] sending shard > > > > started for [docs][4], node[LK1HamGVSqCXrOxh7zr8yA], [P], > > > > s[INITIALIZING], reason [after recovery from gateway] > > > > 20110929-15:02:25 [elasticsearch[cached]-pool-1-thread-2] DEBUG > > > > org.elasticsearch.cluster.action.shard - [White Tiger] received shard > > > > started for [docs][4], node[LK1HamGVSqCXrOxh7zr8yA], [P], > > > > s[INITIALIZING], reason [after recovery from gateway] > > > > 20110929-15:02:25 [elasticsearch[White Tiger]clusterService#updateTask- > > > > pool-11-thread-1] DEBUG org.elasticsearch.cluster.metadata - [White > > > > Tiger] [docs] create\_mapping [doc] with source [{"doc":{"properties": > > > > {"contents":{"type":"attachment","path":"full","fields":{"contents": > > > > {"type":"string"},"author":{"type":"string"},"title": > > > > {"type":"string"},"date": > > > > {"type":"date","format":"dateOptionalTime"},"keywords": > > > > {"type":"string"},"content\_type":{"type":"string"}}},"lastModified": > > > > {"type":"long","index":"analyzed"}}}}] > > > > ... added > > > > 20110929-15:02:25 [elasticsearch[White Tiger]clusterService#updateTask- > > > > pool-11-thread-1] TRACE org.elasticsearch.cluster.service - [White > > > > Tiger] cluster state updated: > > > > version [5], source [put-mapping [doc]] > > > > nodes: > > > > > with this config: > > > > > { > > > > "docs":{ > > > > "properties" : { > > > > "contents": { > > > > "type" : "attachment", > > > > "path":"full", > > > > "store": "no", > > > > "fields":{ > > > > "contents":{"type":"string", "store": > > > > "no", > > > > "index":"analyzed"}, > > > > "author":{"type":"string"}, > > > > "title":{"type":"string"}, > > > > "date": > > > > {"type":"date","store":"no","format":"dateOptionalTime"}, > > > > "keywords":{"type":"string"}, > > > > "content\_type":{"type":"string"} > > > > } > > > > }, > > > > "lastModified": { "type" : "long", "index" : > > > > "analyzed", "store" : > > > > "no"} > > > > } > > > > } > > > > > } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:52am UTC](https://discuss.elastic.co/t/attachment-plugin-questions-on-storing/5480/15 "2017-07-06T03:52:08Z")