# Customized attachment mapper plugin for streaming, smile and extra metadata support **URL:** \ **Category:** Elasticsearch\ **Created:** [August 12, 2013, 12:19pm UTC](https://discuss.elastic.co/t/customized-attachment-mapper-plugin-for-streaming-smile-and-extra-metadata-support/13174 "2013-08-12T12:19:17Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![vincentlaucy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vincentlaucy/32/2175_2.png) [@vincentlaucy](https://discuss.elastic.co/u/vincentlaucy)\ **Post date:** [August 12, 2013, 12:19pm UTC](https://discuss.elastic.co/t/customized-attachment-mapper-plugin-for-streaming-smile-and-extra-metadata-support/13174/1 "2013-08-12T12:19:17Z")
Not sure if I should post it here / in github Thanks for the work of Mapper Attachment Plugin, and recently I have created a customization for my work and I would like to share here. > **[vincentlaucy/elasticsearch-mapper-attachments](https://github.com/vincentlaucy/elasticsearch-mapper-attachments)** > > elasticsearch-mapper-attachments - Mapper Attachments Type plugin for ElasticSearch The customization try to: - Support end to end streaming without loading whole JSON into memory - Support Smile - Support Accepting Images or other binary files for indexing Existing implementation accept only String and will throw Exception otherwise. Customized implementation now accept any binary format and will use Tika for detection - Support customized fields (e.g. EXIF for images) - Calculate the file checksum for my use case Quite a lot of things and they may not applicable in general and refactoring will need to be done, I try to put things I would like to discuss one by one below and see if I can help to submit any PR ####Streaming This is quite complicated, I used a dirty hack the get the actual Jackson JSONParser and then apply the streaming method read `JsonParser.readBinaryValue(OutputStream)` > **[FasterXML/jackson-docs](https://github.com/FasterXML/jackson-docs/wiki/Presentation-Jackson-2.1-Overview#chunked-binary-value-reads-writes-jsonparser--jsongenerator)** > > jackson-docs - Documentation for the Jackson JSON processor. Then I pass it to apache commons io's TeeOutputStream, which will then use java's Pipe to buffer the content to the actual inpustream consumed by Tika. I am not a big fan of using multi Thread in these cases, but it seems whenever the PipeInputStream is to be used the problem is inherently a multi-threading one. Here we are trading off duplicating of buffer data into threads to ensure buffer is of limited size depending on the process, in general the consumer stream is still much faster than the input stream (likely network IO). Are there any streaming libraries out there that could better handling this problem?say Google's Guava ? ###Decouple index request with file content? I also did some study comparing base64 / Smile indexing where later will be more efficient After all, will there be any benefit if we could separate the json index request and the actual file content? i.e. with Multipart? That decouple the encoding which make base64/Gzip easier. ####Smile For Smile, in my scenario when the content object is being parsed will be actually `XContentParser.Token.VALUE_EMBEDDED_OBJECT`, not `XContentParser.Token.VALUE_STRING` ####Accept empty parsed content For some files, after parsing the content is still empty. e.g. Images. however metadata retrieved by Tika is more useful. Exception should not be throw for such cases ####tika type it seems currently field mapping cannot be added dynamically? i.e. when new Tika metadata will needed to be stored, that will require customization in the plugin. Tika is quite powerful and supports lots of metadata I wonder is it possible to add a type "tika" like this, for example. type:"tika", metadatakey:"Image Height:" store:true where when such key exists in Tika's Metadata object, its value will be populated and stored accordingly. Also, I found in building with context.path().add(name), fields stored are in path delimited by dot `.` but not in object format. i.e. I will in the result document with key individually "file\_attachment.author" "file\_attachment.keywords" "file\_attachment.name" can I group them into nested object?e.g. with image as an example ``` i.e. file_attachment:{ image_exif:{ height:{ ``` ####Testing - is ES0.90.3 moving to Junit or TestNG should stay? Quite a lot of stuff, very happy if got feedback from you -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:21am UTC](https://discuss.elastic.co/t/customized-attachment-mapper-plugin-for-streaming-smile-and-extra-metadata-support/13174/2 "2017-07-06T02:21:34Z")
Closed on July 6, 2017, 2:21am UTC