# Re: Indexing binary files - how to store the extracted text only **URL:** \ **Category:** Elasticsearch\ **Created:** [July 16, 2013, 11:39am UTC](https://discuss.elastic.co/t/re-indexing-binary-files-how-to-store-the-extracted-text-only/12809 "2013-07-16T11:39:53Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![matthew\_ford](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@matthew\_ford](https://discuss.elastic.co/u/matthew_ford)\ **Post date:** [July 16, 2013, 11:39am UTC](https://discuss.elastic.co/t/re-indexing-binary-files-how-to-store-the-extracted-text-only/12809/1 "2013-07-16T11:39:53Z")
Hello Tom, Did you find a solution to this problem in the end? Cheers, Matt On Tuesday, 15 January 2013 23:19:07 UTC, tom wrote: > Hello, > > indexing binary files with the mapper attachments plugin works very fine. > Unfortunately, it always stores the total file content base64 encoded in > the > index where I just want to store only the text segments Tika extracts from > the file (I fetch the files from my local file system via the fsriver). Is > it possible to configure the attachment plugin appropriatley? Or are there > any other plugins/solutions for that task available? > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Indexing-binary-files-how-to-store-the-extracted-text-only-tp4028232.html](http://elasticsearch-users.115913.n3.nabble.com/Indexing-binary-files-how-to-store-the-extracted-text-only-tp4028232.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 16, 2013, 1:05pm UTC](https://discuss.elastic.co/t/re-indexing-binary-files-how-to-store-the-extracted-text-only/12809/2 "2013-07-16T13:05:23Z")
Are you really sure it does not work? From the description at > **[GitHub - elastic/elasticsearch-mapper-attachments: Mapper Attachments Type...](https://github.com/elastic/elasticsearch-mapper-attachments)** > > Mapper Attachments Type plugin for Elasticsearch. Contribute to elastic/elasticsearch-mapper-attachments development by creating an account on GitHub. "By default, 100000 characters are extracted when indexing the content. This default value can be changed by setting the index.mapping.attachment.indexed\_chars setting. It can also be provided on a per document indexed using the \_indexed\_chars parameter. -1 can be set to extract all text, but note that all the text needs to be allowed to be represented in memory." Jörg Am 16.07.13 13:39, schrieb [matthew.ford@digital.cabinet-office.gov.uk](mailto:matthew.ford@digital.cabinet-office.gov.uk): > Hello Tom, > > Did you find a solution to this problem in the end? > > Cheers, > Matt > > On Tuesday, 15 January 2013 23:19:07 UTC, tom wrote: > > ``` > Hello, > > indexing binary files with the mapper attachments plugin works > very fine. > Unfortunately, it always stores the total file content base64 > encoded in the > index where I just want to store only the text segments Tika > extracts from > the file (I fetch the files from my local file system via the > fsriver). Is > it possible to configure the attachment plugin appropriatley? Or > are there > any other plugins/solutions for that task available? > > -- > View this message in context: > http://elasticsearch-users.115913.n3.nabble.com/Indexing-binary-files-how-to-store-the-extracted-text-only-tp4028232.html > > > Sent from the ElasticSearch Users mailing list archive at Nabble.com. > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Christian\_Th](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@Christian\_Th](https://discuss.elastic.co/u/Christian_Th)\ **Post date:** [July 16, 2013, 1:36pm UTC](https://discuss.elastic.co/t/re-indexing-binary-files-how-to-store-the-extracted-text-only/12809/3 "2013-07-16T13:36:40Z")
You do not want to store the base64 encoded String, which was sent to ElasticSearch as the "content" field? I think it is not possible yet. The base64 encoded String is indexed anytime a json-document is sent with a field called "content" if (name.equals(propName)) { // that is the content [https://github.com/elasticsearch/elasticsearch-mapper-attachments/blob/master/src/main/java/org/elasticsearch/index/mapper/attachment/AttachmentMapper.java](https://github.com/elasticsearch/elasticsearch-mapper-attachments/blob/master/src/main/java/org/elasticsearch/index/mapper/attachment/AttachmentMapper.java) Please correct me, if i am wrong -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![matthew\_ford](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@matthew\_ford](https://discuss.elastic.co/u/matthew_ford)\ **Post date:** [July 16, 2013, 2:20pm UTC](https://discuss.elastic.co/t/re-indexing-binary-files-how-to-store-the-extracted-text-only/12809/4 "2013-07-16T14:20:10Z")
Hi Christian, This is what I'm trying to achieve, with close to 50GB of pfds most of which contain images, I would love to be able to just store the indexed text rather than the attachment itself. In my ideal world the base64 attachment is sent, processed but not stored in ES, sounds like this isn't possible and I'll have to do it before we send the document to the index. As an aside I'm also somewhat confused if the file field is called 'file' or 'content', file seemed to work for me, but perhaps this changed in more recent versions of the plugin. Many Thanks, Matthew Ford On Tuesday, 16 July 2013 14:36:40 UTC+1, Christian Th. wrote: > You do not want to store the base64 encoded String, which was sent to > Elasticsearch as the "content" field? > I think it is not possible yet. > > The base64 encoded String is indexed anytime a json-document is sent with > a field called "content" > > if (name.equals(propName)) { > // that is the content > > [https://github.com/elasticsearch/elasticsearch-mapper-attachments/blob/master/src/main/java/org/elasticsearch/index/mapper/attachment/AttachmentMapper.java](https://github.com/elasticsearch/elasticsearch-mapper-attachments/blob/master/src/main/java/org/elasticsearch/index/mapper/attachment/AttachmentMapper.java) > > Please correct me, if i am wrong -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Christian\_Th](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@Christian\_Th](https://discuss.elastic.co/u/Christian_Th)\ **Post date:** [July 16, 2013, 9:13pm UTC](https://discuss.elastic.co/t/re-indexing-binary-files-how-to-store-the-extracted-text-only/12809/5 "2013-07-16T21:13:03Z")
Sorry for giving the wrong answer. It is possible. The extracted content from Tika is added to the "\_all" field and added to the "file" field. The "\_source" field is filled with the base64 String. One possibility is to disable the "\_source" field for your attachment type. { "person" : { "\_source" : {"enabled" : false}, "properties" : { "file" : { "type" : "attachment", "fields" : { "file" : {"store" : "yes"} } } } } } Am Dienstag, 16. Juli 2013 16:20:10 UTC+2 schrieb [matthe...@digital.cabinet-office.gov.uk](mailto:matthe...@digital.cabinet-office.gov.uk): > Hi Christian, > > This is what I'm trying to achieve, with close to 50GB of pfds most of > which contain images, I would love to be able to just store the indexed > text rather than the attachment itself. In my ideal world the base64 > attachment is sent, processed but not stored in ES, sounds like this isn't > possible and I'll have to do it before we send the document to the index. > > As an aside I'm also somewhat confused if the file field is called 'file' > or 'content', file seemed to work for me, but perhaps this changed in more > recent versions of the plugin. > > Many Thanks, > Matthew Ford > > On Tuesday, 16 July 2013 14:36:40 UTC+1, Christian Th. wrote: > > > You do not want to store the base64 encoded String, which was sent to > > Elasticsearch as the "content" field? > > I think it is not possible yet. > > > > The base64 encoded String is indexed anytime a json-document is sent with > > a field called "content" > > > > if (name.equals(propName)) { > > // that is the content > > > > [https://github.com/elasticsearch/elasticsearch-mapper-attachments/blob/master/src/main/java/org/elasticsearch/index/mapper/attachment/AttachmentMapper.java](https://github.com/elasticsearch/elasticsearch-mapper-attachments/blob/master/src/main/java/org/elasticsearch/index/mapper/attachment/AttachmentMapper.java) > > > > Please correct me, if i am wrong -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Christian\_Th](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@Christian\_Th](https://discuss.elastic.co/u/Christian_Th)\ **Post date:** [July 16, 2013, 9:36pm UTC](https://discuss.elastic.co/t/re-indexing-binary-files-how-to-store-the-extracted-text-only/12809/6 "2013-07-16T21:36:12Z")
Sorry for the wrong answer. It is possible. The extracted content from Tika is added to the "\_all" field and added to the "file" field. The "\_source" field is filled with the base64 String. One possibility is to disable the "\_source" field for your attachment type. { "person" : { "\_source" : {"enabled" : false}, "properties" : { "file" : { "type" : "attachment", "fields" : { "file" : {"store" : "yes"} } } } } } -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 17, 2013, 7:34am UTC](https://discuss.elastic.co/t/re-indexing-binary-files-how-to-store-the-extracted-text-only/12809/7 "2013-07-17T07:34:13Z")
Hey, how about curl -X DELETE localhost:9200/test curl -X PUT localhost:9200/test curl -X PUT localhost:9200/test/person/\_mapping -d ' { "person" : { "\_source" : { "excludes" : ["my\_attachment"] }, "properties" : { "my\_attachment" : { "type" : "attachment" } } } }' # base64 string is: Hello World curl -X PUT 'localhost:9200/test/person/1?refresh=true' -d '{ "name" : "my person", "my\_attachment" : "SGVsbG8gV29ybGQK" }' # this query returns only the name field, but searches in the attachment curl -X POST 'localhost:9200/test/person/\_search' -d '{ "query" : { "match" : { "my\_attachment" : "hello" } } }' hope this helps --Alex On Tue, Jul 16, 2013 at 11:36 PM, Christian Th. [chth.exensio@gmail.com](mailto:chth.exensio@gmail.com)wrote: > Sorry for the wrong answer. It is possible. > The extracted content from Tika is added to the "\_all" field and added to > the "file" field. The "\_source" field is filled with the base64 String. > One possibility is to disable the "\_source" field for your attachment > type. > > { > "person" : { > "\_source" : {"enabled" : false}, > "properties" : { > "file" : { > "type" : "attachment", > "fields" : { > "file" : {"store" : "yes"} > } > } > } > } > } > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Christian\_Th](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@Christian\_Th](https://discuss.elastic.co/u/Christian_Th)\ **Post date:** [July 17, 2013, 8:38am UTC](https://discuss.elastic.co/t/re-indexing-binary-files-how-to-store-the-extracted-text-only/12809/8 "2013-07-17T08:38:25Z")
An other possibility is the combination of the two answers, if you want to store the field and not only search on it. Disable the \_source for the field of type attachment. But store the extracted text, so it can be retrieved. Fields are not stored by default. curl -X PUT localhost:9200/test/person/\_mapping -d ' { "person": { "\_source": { "excludes": [ "my\_attachment" ] }, "properties": { "file": { "type": "attachment", "fields": { "file": { "store": "yes" } } } } } }' -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Christian\_Th](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@Christian\_Th](https://discuss.elastic.co/u/Christian_Th)\ **Post date:** [July 17, 2013, 8:40am UTC](https://discuss.elastic.co/t/re-indexing-binary-files-how-to-store-the-extracted-text-only/12809/9 "2013-07-17T08:40:55Z")
An other possibility is the combination of the two answers, if you want to store the field and not only search on it. Disable the \_source for the field of type attachment. But store the extracted text, so it can be retrieved. Fields are not stored by default. curl -X PUT localhost:9200/test/person/\_mapping -d ' { "person": { "\_source": { "excludes": [ "file" ] }, "properties": { "file": { "type": "attachment", "fields": { "file": { "store": "yes" } } } } } }' -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:25am UTC](https://discuss.elastic.co/t/re-indexing-binary-files-how-to-store-the-extracted-text-only/12809/10 "2017-07-06T02:25:59Z")