# Extracting data from documents stored in HDFS to index in Elasticsearch **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [April 5, 2016, 12:42pm UTC](https://discuss.elastic.co/t/extracting-data-from-documents-stored-in-hdfs-to-index-in-elasticsearch/46391 "2016-04-05T12:42:04Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Sachin054](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sachin054/32/9102_2.png) [@Sachin054](https://discuss.elastic.co/u/Sachin054)\ **Post date:** [April 5, 2016, 12:42pm UTC](https://discuss.elastic.co/t/extracting-data-from-documents-stored-in-hdfs-to-index-in-elasticsearch/46391/1 "2016-04-05T12:42:04Z")
I have a HDFS archive to store variety of documents like pdf,ms word file,ppt,csv etc. I would like to build a platform using elasticsearch to search the file or text contents. I know I can use the es-hadoop plugin to index data to from HDFS to ES. I want to know the best ways that I can extract out the textual data from the docs stored in HDFS and index the same. Any help would be appreciated. ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [April 5, 2016, 3:29pm UTC](https://discuss.elastic.co/t/extracting-data-from-documents-stored-in-hdfs-to-index-in-elasticsearch/46391/2 "2016-04-05T15:29:01Z")
Take a look at the various plugins for elasticsearch. This is about content extraction and that's where the plugins help a lot. In particular ES 5.0 comes with a dedicated ingest plugin based on Apache Tika: [https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest.html](https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest.html) ---
**Author:** ![Sachin054](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sachin054/32/9102_2.png) [@Sachin054](https://discuss.elastic.co/u/Sachin054)\ **Post date:** [April 6, 2016, 5:02am UTC](https://discuss.elastic.co/t/extracting-data-from-documents-stored-in-hdfs-to-index-in-elasticsearch/46391/3 "2016-04-06T05:02:51Z")
So what is the difference between ingest plugin and _mapper attachments plugin_ ? I was referring to it as a solution. Can we point to a HDFS location using either of these ? Because I don't want to store files in ES.File should remain on HDFS ,only searchable contents should be moved to ES. ---
**Author:** ![Sachin054](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sachin054/32/9102_2.png) [@Sachin054](https://discuss.elastic.co/u/Sachin054)\ **Post date:** [April 6, 2016, 5:27am UTC](https://discuss.elastic.co/t/extracting-data-from-documents-stored-in-hdfs-to-index-in-elasticsearch/46391/4 "2016-04-06T05:27:45Z")
I haven't seen any proper documentation for the ingest plugin and alpha version of ES 5.0 is just released. So at this time what would you suggest for my use-case ? I have a HDFS archive with many types of documents and want to extract out all the information to ES while keeping the doc in Hadoop itself. Please take a look at [http://stackoverflow.com/questions/36419608/extracting-data-from-documents-stored-in-hdfs-to-index-in-elasticsearch/36436556#36436556](http://stackoverflow.com/questions/36419608/extracting-data-from-documents-stored-in-hdfs-to-index-in-elasticsearch/36436556#36436556) for detailed info. ---
**Author:** ![anon55368183](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@anon55368183](https://discuss.elastic.co/u/anon55368183)\ **Post date:** [April 6, 2016, 2:55pm UTC](https://discuss.elastic.co/t/extracting-data-from-documents-stored-in-hdfs-to-index-in-elasticsearch/46391/5 "2016-04-06T14:55:07Z")
You might want to look into ManifoldCF. [https://manifoldcf.apache.org/en\_US/index.html#What+Is+Apache+ManifoldCF%3F](https://manifoldcf.apache.org/en_US/index.html#What+Is+Apache+ManifoldCF%3F) Just point it to your HDFS repository and define ES as output - it uses Apache Tika as well. ---
**Author:** ![Sachin054](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sachin054/32/9102_2.png) [@Sachin054](https://discuss.elastic.co/u/Sachin054)\ **Post date:** [April 7, 2016, 5:09am UTC](https://discuss.elastic.co/t/extracting-data-from-documents-stored-in-hdfs-to-index-in-elasticsearch/46391/6 "2016-04-07T05:09:17Z")
Thanks for the reply! Can we use tika as it is to extract from HDFS ? Which one will be the better choice ? ---
**Author:** ![anon55368183](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@anon55368183](https://discuss.elastic.co/u/anon55368183)\ **Post date:** [April 8, 2016, 9:46am UTC](https://discuss.elastic.co/t/extracting-data-from-documents-stored-in-hdfs-to-index-in-elasticsearch/46391/7 "2016-04-08T09:46:24Z")
I may caused some confusion but Apache Tika is used by ManifoldCF. You can decide on your own whether you want Tika to use the Mapper-Attachments Plugin or not ([https://github.com/elastic/elasticsearch-mapper-attachments](https://github.com/elastic/elasticsearch-mapper-attachments)). In fact it is quite easy to set-up and does not need advanced configuration. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:25pm UTC](https://discuss.elastic.co/t/extracting-data-from-documents-stored-in-hdfs-to-index-in-elasticsearch/46391/8 "2017-07-06T13:25:16Z")