# Attachment plugin: flexibility? **URL:** \ **Category:** Elasticsearch\ **Created:** [October 12, 2011, 10:33am UTC](https://discuss.elastic.co/t/attachment-plugin-flexibility/5576 "2011-10-12T10:33:11Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![jeremie\_bordier](https://avatars.discourse-cdn.com/v4/letter/j/919ad9/32.png) [@jeremie\_bordier](https://discuss.elastic.co/u/jeremie_bordier)\ **Post date:** [October 12, 2011, 10:33am UTC](https://discuss.elastic.co/t/attachment-plugin-flexibility/5576/1 "2011-10-12T10:33:11Z")
Hi ! First, many thanks for the great work done on the attachment plugin and on elasticsearch in general. I have a few questions about the attachment plugin: - I tried to configure the attachment mapping adding a "size" long field but it always gets dropped. I had a look at the code and it seems that the attachement type takes in account existing configuration of the standard fields, but drops the other, Why ? curl -X PUT "${host}/test/attachment/\_mapping" -d '{ "attachment" : { "properties" : { "file" : { "type" : "attachment", "fields" : { "size" : { "type" : "long", "store" : yes } "title" : { "store" : "yes" }, "file" : { "term\_vector":"with\_positions\_offsets", "store":"yes" } } } } } }' - When I index simple PDF files and try to fetch back the content\_type, I don't get the response: > - I saw in the source that we could hint the document name using \_name, but it doesn't seem to be used as an alternative title for the document if tika don't manage to extract one from the metadata. Wouldn't that make sense ? What is it used for then ? Thanks again ! Jérémie ---
**Author:** ![jeremie\_bordier](https://avatars.discourse-cdn.com/v4/letter/j/919ad9/32.png) [@jeremie\_bordier](https://discuss.elastic.co/u/jeremie_bordier)\ **Post date:** [October 12, 2011, 4:37pm UTC](https://discuss.elastic.co/t/attachment-plugin-flexibility/5576/2 "2011-10-12T16:37:54Z")
On Wed, Oct 12, 2011 at 12:33 PM, [jeremie.bordier@gmail.com](mailto:jeremie.bordier@gmail.com) [ahfeel@gmail.com](mailto:ahfeel@gmail.com) wrote: > Hi ! > > First, many thanks for the great work done on the attachment plugin > and on elasticsearch in general. > > I have a few questions about the attachment plugin: > > - I tried to configure the attachment mapping adding a "size" long > field but it always gets dropped. I had a look at the code and it > seems that the attachement type takes in account existing > configuration of the standard fields, but drops the other, Why ? > > curl -X PUT "${host}/test/attachment/\_mapping" -d '{ > "attachment" : { > "properties" : { > "file" : { > "type" : "attachment", > "fields" : { > "size" : { "type" : "long", "store" : yes } > "title" : { "store" : "yes" }, > "file" : { "term\_vector":"with\_positions\_offsets", > "store":"yes" } > } > } > } > } > }' > > - When I index simple PDF files and try to fetch back the > content\_type, I don't get the response: > > [gist:1280860 · GitHub](https://gist.github.com/1280860) > > - I saw in the source that we could hint the document name using > \_name, but it doesn't seem to be used as an alternative title for the > document if tika don't manage to extract one from the metadata. > Wouldn't that make sense ? What is it used for then ? > > Thanks again ! > > Jérémie Also, another quick question: - If my understanding is right, the original message will be stored in \_source with the "content" : base64. Will it really store the base64 encoded file without decoding it first ? This would be a HUGE storing overhead compared to the bare original binary. (I'm okay for storing the binary in ES as long as it's not stored in base64). Thanks ! -- Jérémie 'ahFeel' BORDIER ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [October 12, 2011, 8:14pm UTC](https://discuss.elastic.co/t/attachment-plugin-flexibility/5576/3 "2011-10-12T20:14:28Z")
Hi, my understanding is that as of now it only allows to store base64 encoded attachments in \_source. That is what the incoming json document looks like and thus that should be what the client gets back. Decoding it and encoding internally would mean that the search operation would take longer (but may be there are also other reasons why it is stored in base64). In the end of the day you need to apply some encoding anyway because the data are transmitted and replicated inside the cluster and between the cluster and client. Not sure if that is really problem from the disk space perspective, I do not have that much attachments content to be even near to any index size threshold. Also note, you can compress \_source content: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On Wed, Oct 12, 2011 at 6:37 PM, Jérémie BORDIER [ahfeel@gmail.com](mailto:ahfeel@gmail.com) wrote: > On Wed, Oct 12, 2011 at 12:33 PM, [jeremie.bordier@gmail.com](mailto:jeremie.bordier@gmail.com) > [ahfeel@gmail.com](mailto:ahfeel@gmail.com) wrote: > > > Hi ! > > > > First, many thanks for the great work done on the attachment plugin > > and on elasticsearch in general. > > > > I have a few questions about the attachment plugin: > > > > - I tried to configure the attachment mapping adding a "size" long > > field but it always gets dropped. I had a look at the code and it > > seems that the attachement type takes in account existing > > configuration of the standard fields, but drops the other, Why ? > > > > curl -X PUT "${host}/test/attachment/\_mapping" -d '{ > > "attachment" : { > > "properties" : { > > "file" : { > > "type" : "attachment", > > "fields" : { > > "size" : { "type" : "long", "store" : yes } > > "title" : { "store" : "yes" }, > > "file" : { "term\_vector":"with\_positions\_offsets", > > "store":"yes" } > > } > > } > > } > > } > > }' > > > > - When I index simple PDF files and try to fetch back the > > content\_type, I don't get the response: > > > > [gist:1280860 · GitHub](https://gist.github.com/1280860) I think these sounds like candidates for a ticket? I remember looking at the code back some time and noticed something similar but it was not a problem for me at that time (probably that is why this plugin is experimental). > > - I saw in the source that we could hint the document name using > > \_name, but it doesn't seem to be used as an alternative title for the > > document if tika don't manage to extract one from the metadata. > > Wouldn't that make sense ? What is it used for then ? What if the document does not really have any title (not that Tika fails to extract it) how can you then learn about this in case it is important? But I am not saying that this was the original reasoning behind this logic. However, you can always pull both fields [title, name] and do this magic on client side. > > Thanks again ! > > > > Jérémie > > Also, another quick question: > > - If my understanding is right, the original message will be stored in > \_source with the "content" : base64. Will it really store the base64 > encoded file without decoding it first ? This would be a HUGE storing > overhead compared to the bare original binary. (I'm okay for storing > the binary in ES as long as it's not stored in base64). > > Thanks ! > > -- > Jérémie 'ahFeel' BORDIER HTH, Regards, Lukas ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 12, 2011, 9:40pm UTC](https://discuss.elastic.co/t/attachment-plugin-flexibility/5576/4 "2011-10-12T21:40:35Z")
On Wed, Oct 12, 2011 at 12:33 PM, [jeremie.bordier@gmail.com](mailto:jeremie.bordier@gmail.com) \< [ahfeel@gmail.com](mailto:ahfeel@gmail.com)\> wrote: > Hi ! > > First, many thanks for the great work done on the attachment plugin > and on elasticsearch in general. > > I have a few questions about the attachment plugin: > > - I tried to configure the attachment mapping adding a "size" long > field but it always gets dropped. I had a look at the code and it > seems that the attachement type takes in account existing > configuration of the standard fields, but drops the other, Why ? Because it only support specific "extra" fields associated with the attachment, and indexing the size is not part of it. You can either add it as a field yourself to the json doc, or, we can enhance the attachment mapper to also index it. > curl -X PUT "${host}/test/attachment/\_mapping" -d '{ > "attachment" : { > "properties" : { > "file" : { > "type" : "attachment", > "fields" : { > "size" : { "type" : "long", "store" : yes } > "title" : { "store" : "yes" }, > "file" : { "term\_vector":"with\_positions\_offsets", > "store":"yes" } > } > } > } > } > }' > > - When I index simple PDF files and try to fetch back the > content\_type, I don't get the response: > > [gist:1280860 · GitHub](https://gist.github.com/1280860) You need to explicitly configure the content\_type mapping and set "store" to "yes". > - I saw in the source that we could hint the document name using > \_name, but it doesn't seem to be used as an alternative title for the > document if tika don't manage to extract one from the metadata. > Wouldn't that make sense ? What is it used for then ? Just to enhance Tika ability to guess the content type. > Thanks again ! > > Jérémie ---
**Author:** ![Jeremie\_BORDIER1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremie_bordier1/32/2860_2.png) [@Jeremie\_BORDIER1](https://discuss.elastic.co/u/Jeremie_BORDIER1)\ **Post date:** [October 12, 2011, 9:44pm UTC](https://discuss.elastic.co/t/attachment-plugin-flexibility/5576/5 "2011-10-12T21:44:23Z")
Hi Shay and Lukas, Thanks for the answers 🙂 I had a quick talk with Lukas on twitter so I'll just post back on your last comments Shay. > > - When I index simple PDF files and try to fetch back the > > content\_type, I don't get the response: > > > > [gist:1280860 · GitHub](https://gist.github.com/1280860) > > You need to explicitly configure the content\_type mapping and set "store" to > "yes". Isn't "file.content\_type" supposed to fetch it from \_source implicitly ? > > - I saw in the source that we could hint the document name using > > \_name, but it doesn't seem to be used as an alternative title for the > > document if tika don't manage to extract one from the metadata. > > Wouldn't that make sense ? What is it used for then ? > > Just to enhance Tika ability to guess the content type. Ahh.. okay 🙂 Thanks, Jérémie ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 12, 2011, 9:44pm UTC](https://discuss.elastic.co/t/attachment-plugin-flexibility/5576/6 "2011-10-12T21:44:41Z")
On Wed, Oct 12, 2011 at 6:37 PM, Jérémie BORDIER [ahfeel@gmail.com](mailto:ahfeel@gmail.com) wrote: > On Wed, Oct 12, 2011 at 12:33 PM, [jeremie.bordier@gmail.com](mailto:jeremie.bordier@gmail.com) > [ahfeel@gmail.com](mailto:ahfeel@gmail.com) wrote: > > > Hi ! > > > > First, many thanks for the great work done on the attachment plugin > > and on elasticsearch in general. > > > > I have a few questions about the attachment plugin: > > > > - I tried to configure the attachment mapping adding a "size" long > > field but it always gets dropped. I had a look at the code and it > > seems that the attachement type takes in account existing > > configuration of the standard fields, but drops the other, Why ? > > > > curl -X PUT "${host}/test/attachment/\_mapping" -d '{ > > "attachment" : { > > "properties" : { > > "file" : { > > "type" : "attachment", > > "fields" : { > > "size" : { "type" : "long", "store" : yes } > > "title" : { "store" : "yes" }, > > "file" : { "term\_vector":"with\_positions\_offsets", > > "store":"yes" } > > } > > } > > } > > } > > }' > > > > - When I index simple PDF files and try to fetch back the > > content\_type, I don't get the response: > > > > [gist:1280860 · GitHub](https://gist.github.com/1280860) > > > > - I saw in the source that we could hint the document name using > > \_name, but it doesn't seem to be used as an alternative title for the > > document if tika don't manage to extract one from the metadata. > > Wouldn't that make sense ? What is it used for then ? > > > > Thanks again ! > > > > Jérémie > > Also, another quick question: > > - If my understanding is right, the original message will be stored in > \_source with the "content" : base64. Will it really store the base64 > encoded file without decoding it first ? This would be a HUGE storing > overhead compared to the bare original binary. (I'm okay for storing > the binary in ES as long as it's not stored in base64). Yes, elasticsearch does not "touch" the \_source and stores it "as is". You can have the \_source compressed though, or completely disable it (storing the \_source, and explicitly store, in the mapping, what you want). > Thanks ! > > -- > Jérémie 'ahFeel' BORDIER ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 12, 2011, 9:46pm UTC](https://discuss.elastic.co/t/attachment-plugin-flexibility/5576/7 "2011-10-12T21:46:43Z")
On Wed, Oct 12, 2011 at 11:44 PM, Jérémie BORDIER \<[jeremie.bordier@gmail.com](mailto:jeremie.bordier@gmail.com) > wrote: > Hi Shay and Lukas, > > Thanks for the answers 🙂 I had a quick talk with Lukas on twitter so > I'll just post back on your last comments Shay. > > > > - When I index simple PDF files and try to fetch back the > > > content\_type, I don't get the response: > > > > > > [gist:1280860 · GitHub](https://gist.github.com/1280860) > > > > You need to explicitly configure the content\_type mapping and set "store" > > to > > "yes". > > Isn't "file.content\_type" supposed to fetch it from \_source implicitly ? No, \_source is the json you provided, as is, without any changes. Obviously, it does not contain the content\_type, thats why you need to explicitly store it in addition to the \_source. > > > - I saw in the source that we could hint the document name using > > > \_name, but it doesn't seem to be used as an alternative title for the > > > document if tika don't manage to extract one from the metadata. > > > Wouldn't that make sense ? What is it used for then ? > > > > Just to enhance Tika ability to guess the content type. > > Ahh.. okay 🙂 > > Thanks, > Jérémie ---
**Author:** ![Jeremie\_BORDIER1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremie_bordier1/32/2860_2.png) [@Jeremie\_BORDIER1](https://discuss.elastic.co/u/Jeremie_BORDIER1)\ **Post date:** [October 12, 2011, 9:49pm UTC](https://discuss.elastic.co/t/attachment-plugin-flexibility/5576/8 "2011-10-12T21:49:38Z")
On Wed, Oct 12, 2011 at 11:46 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > On Wed, Oct 12, 2011 at 11:44 PM, Jérémie BORDIER > [jeremie.bordier@gmail.com](mailto:jeremie.bordier@gmail.com) wrote: > > > Hi Shay and Lukas, > > > > Thanks for the answers 🙂 I had a quick talk with Lukas on twitter so > > I'll just post back on your last comments Shay. > > > > > > - When I index simple PDF files and try to fetch back the > > > > content\_type, I don't get the response: > > > > > > > > [gist:1280860 · GitHub](https://gist.github.com/1280860) > > > > > > You need to explicitly configure the content\_type mapping and set > > > "store" to > > > "yes". > > > > Isn't "file.content\_type" supposed to fetch it from \_source implicitly ? > > No, \_source is the json you provided, as is, without any changes. Obviously, > it does not contain the content\_type, thats why you need to explicitly store > it in addition to the \_source. Ah sorry, I realized I forgot to show the PUT request I made. Then, last question, if I had a content\_type in my source json, would the "file.content\_type" =\> "\_source.file.content\_type" be done automatically ? It doesn't seems like, but I though I've seen something like that in the documentation. Jérémie ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 14, 2011, 11:59am UTC](https://discuss.elastic.co/t/attachment-plugin-flexibility/5576/9 "2011-10-14T11:59:00Z")
On Wed, Oct 12, 2011 at 11:49 PM, Jérémie BORDIER \<[jeremie.bordier@gmail.com](mailto:jeremie.bordier@gmail.com) > wrote: > On Wed, Oct 12, 2011 at 11:46 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > On Wed, Oct 12, 2011 at 11:44 PM, Jérémie BORDIER > > [jeremie.bordier@gmail.com](mailto:jeremie.bordier@gmail.com) wrote: > > > > > Hi Shay and Lukas, > > > > > > Thanks for the answers 🙂 I had a quick talk with Lukas on twitter so > > > I'll just post back on your last comments Shay. > > > > > > > > - When I index simple PDF files and try to fetch back the > > > > > content\_type, I don't get the response: > > > > > > > > > > [gist:1280860 · GitHub](https://gist.github.com/1280860) > > > > > > > > You need to explicitly configure the content\_type mapping and set > > > > "store" to > > > > "yes". > > > > > > Isn't "file.content\_type" supposed to fetch it from \_source implicitly ? > > > > No, \_source is the json you provided, as is, without any changes. > > Obviously, > > it does not contain the content\_type, thats why you need to explicitly > > store > > it in addition to the \_source. > > Ah sorry, I realized I forgot to show the PUT request I made. Then, > last question, if I had a content\_type in my source json, would the > "file.content\_type" =\> "\_source.file.content\_type" be done > automatically ? It doesn't seems like, but I though I've seen > something like that in the documentation. If you have any other field in the json, then it will be indexed and you can search on it, as well as get it back from the \_source. > Jérémie ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:51am UTC](https://discuss.elastic.co/t/attachment-plugin-flexibility/5576/10 "2017-07-06T03:51:58Z")