# Parsing and indexing documents with Apache Tika **URL:** \ **Category:** Elasticsearch\ **Created:** [May 6, 2015, 1:30pm UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292 "2015-05-06T13:30:34Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Jean\_Wisser](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@Jean\_Wisser](https://discuss.elastic.co/u/Jean_Wisser)\ **Post date:** [May 6, 2015, 1:30pm UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292/1 "2015-05-06T13:30:34Z")
Hello everyone, I'm trying to parse and index .doc files into elasticsearch with apache Tika. Actually, my project is to build a resume search engine for my company. Since we have a standardized resume format, I would like to parse these resume using apache tika in Java. Basically I have a .doc file like this : ``` Jean Wisser avenue des Ternes 75017 Paris Business Intelligence Consultant Skills : Qlikview, SAS, Cognos, ... Companies : IBM, Orange, ... ``` And I would like to extract and parse the content to index it in elasticsearch like this : ``` XContentBuilder builder = jsonBuilder() .startObject() .field("Name", "Jean") .field("Lastname", "Wisser") .startObject("Adress") .field("Street", "avenue des Ternes") .field("City", "Paris") ...... .endObject() .endObject() ``` - What is the best way to achieve this ? - Should I use Tika, POI or something else ? ---
**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\ **Post date:** [May 6, 2015, 2:36pm UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292/2 "2015-05-06T14:36:19Z")
You are looking for the Mapper Attachments Plugin [1]. With this you just need to configure it in your mapping and send the base64 encoded .doc file to elasticsearch. [1] [https://github.com/elastic/elasticsearch-mapper-attachments](https://github.com/elastic/elasticsearch-mapper-attachments) ---
**Author:** ![Jean\_Wisser](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@Jean\_Wisser](https://discuss.elastic.co/u/Jean_Wisser)\ **Post date:** [May 6, 2015, 3:03pm UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292/3 "2015-05-06T15:03:18Z")
The thing is, with the mapper attachment plugin I can only extract the content of my files in plain text. I would like to parse every .doc to create a specific elastic document that match my .doc structure. I don't want to have : (this is what I get currently with the plugin) ``` { "my_attachment" : { "_content_type" : "application/msword", "_name" : "resource/name/of/my.doc", "_language" : "en", "_content" : "... base64 encoded attachment ..." } } ``` I would like to have : ``` { "my_attachment" : { "_content_type" : "application/msword", "_name" : "resource/name/of/my.doc", "_language" : "en", "_adress": { "street": "99 avenue des ternes", "city": "Paris" } ...... } } ``` ---
**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\ **Post date:** [May 6, 2015, 3:24pm UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292/4 "2015-05-06T15:24:48Z")
Yes, the plugin only handles metadata and text content. If you have business logic that defines how to extract fields from the content then you would need to do that before submitting to elasticsearch. I guess if you really had to do it in elasticsearch you could write a transform script [1] do extract the data you want and put into fields, but even with that the source (ie. json) would not be updated. My recommendation would to use logstash with some custom filters. You would have one that uses tika to extract the content, then another one to perform your business logic that extracts fields from the content, and use the elasticsearch output to submit to elasticsearch. [1] [http://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-transform.html](http://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-transform.html) ---
**Author:** ![Jean\_Wisser](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@Jean\_Wisser](https://discuss.elastic.co/u/Jean_Wisser)\ **Post date:** [May 6, 2015, 3:57pm UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292/5 "2015-05-06T15:57:41Z")
Thanks a lot for your answer Matt. Currently I'm doing everything with the Java API, and achieved to extract the content of my files with Tika, but I'm stuck at the second step where I need to extract my fields. Do you have any idea how to get this done ? Edit : my idea for this part is to convert the content of the .doc to xhtml with tika and then use something like xpath to parse the fields. Also, since I will extract my fields and not send plain text do I still need to encode anything in base64 ? Thanks ! ---
**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\ **Post date:** [May 6, 2015, 6:56pm UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292/6 "2015-05-06T18:56:09Z")
That's the tricky part, I assumed you had a way to do that. You will probably want to use something simple like regular expressions to start and then do some research into entity extraction. Good luck. ---
**Author:** ![Yongyao\_Jiang](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@Yongyao\_Jiang](https://discuss.elastic.co/u/Yongyao_Jiang)\ **Post date:** [February 19, 2016, 3:22am UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292/7 "2016-02-19T03:22:45Z")
Hi Jean, I am having exactly the same problem. The only difference is that I am trying to index more data formats (pdf, doc, ppt...). May I ask what you ended up doing? Thanks, Cody ---
**Author:** ![Jean\_Wisser](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@Jean\_Wisser](https://discuss.elastic.co/u/Jean_Wisser)\ **Post date:** [February 22, 2016, 5:23pm UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292/8 "2016-02-22T17:23:20Z")
Hi Cody, I ended up using Talend with apache tika to extract the content of my documents and parse my fields with java functions and regex. ---
**Author:** ![Yongyao\_Jiang](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@Yongyao\_Jiang](https://discuss.elastic.co/u/Yongyao_Jiang)\ **Post date:** [February 22, 2016, 8:51pm UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292/9 "2016-02-22T20:51:15Z")
Jean, thanks for your reply. So, you are not using ES, including the mapper plugin, at all? I was also wondering if ES is a good option for primary data storage. Thanks, Cody ---
**Author:** ![Jean\_Wisser](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@Jean\_Wisser](https://discuss.elastic.co/u/Jean_Wisser)\ **Post date:** [February 22, 2016, 9:37pm UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292/10 "2016-02-22T21:37:04Z")
I used ES but without the mapper plugin. My process was like this : Documents -\> Talend (apache tika + regex) -\> MongoDB -\> ES (with mongoDB river). I think that you should consider something else (like mongoDB) to store data and plug your elasticsearch on it. ---
**Author:** ![Yongyao\_Jiang](https://avatars.discourse-cdn.com/v4/letter/y/c4cdca/32.png) [@Yongyao\_Jiang](https://discuss.elastic.co/u/Yongyao_Jiang)\ **Post date:** [February 23, 2016, 2:54am UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292/11 "2016-02-23T02:54:09Z")
Thanks a lot. I am looking at Cassandra and trying to put it together with ES for search workload. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:14pm UTC](https://discuss.elastic.co/t/parsing-and-indexing-documents-with-apache-tika/292/12 "2017-07-05T23:14:18Z")