# How to read JSON files stored in HDFS via Logstash **URL:** \ **Category:** Logstash\ **Created:** [April 11, 2016, 7:42am UTC](https://discuss.elastic.co/t/how-to-read-json-files-stored-in-hdfs-via-logstash/46954 "2016-04-11T07:42:08Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![gowthamsadasivam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gowthamsadasivam/32/9043_2.png) [@gowthamsadasivam](https://discuss.elastic.co/u/gowthamsadasivam)\ **Post date:** [April 11, 2016, 7:42am UTC](https://discuss.elastic.co/t/how-to-read-json-files-stored-in-hdfs-via-logstash/46954/1 "2016-04-11T07:42:08Z")
I have my data (JSON Files) stored in HDFS and would like to push these data to ElasticSearch. I have seen output plugin for HDFS but not any input plugin. Is there a way to read the HDFS files via Logstash? or any other way to push the HDFS data to ElasticSearch other than Logstash? Any help/recommendation is appreciated. Thanks in advance 🙂 ---
**Author:** ![Vinicius\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinicius_silva/32/15754_2.png) [@Vinicius\_Silva](https://discuss.elastic.co/u/Vinicius_Silva)\ **Post date:** [February 6, 2017, 8:30pm UTC](https://discuss.elastic.co/t/how-to-read-json-files-stored-in-hdfs-via-logstash/46954/2 "2017-02-06T20:30:01Z")
Did you get an answer on this ? I have the same issue. ---
**Author:** ![gowthamsadasivam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gowthamsadasivam/32/9043_2.png) [@gowthamsadasivam](https://discuss.elastic.co/u/gowthamsadasivam)\ **Post date:** [February 7, 2017, 6:43am UTC](https://discuss.elastic.co/t/how-to-read-json-files-stored-in-hdfs-via-logstash/46954/3 "2017-02-07T06:43:09Z")
Hello Vinicius, I didn't get any suggestions from this forum yet. But I figured two ways to achieve this. **Method #1:** If you have looked at the currently available output plug-ins for logstash, there is "[webhdfs](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-webhdfs.html)" plug-in which utilizes the [HDFS's REST API over HTTP](https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/WebHDFS.html). I believe we can use this same API to read from HDFS using logstash [HTTP input plug-in](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html) available. By specifying the files location inside the HDFS and read it by HDFS REST API and process it, push it to ES. **Method #2:** _(The one I'm using in production now)_ Using HIVE to push data directly to ES without using logstash. I'm using this method currently in my production cluster. There is[JAR integration](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html) available to push data to ES. But, check the compatibility and [transform features](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Transform) available with-in HIVE before you use this method. If you use HIVE, you may not get all the transformation/manipulation methods available in logstash. Here are few issues I went through while using HIVE with ES: - [Mapping Hive Table Fields to Nested Fields in ElasticSearch](https://discuss.elastic.co/t/mapping-hive-table-fields-to-nested-fields-in-elasticsearch/47389) - [Removing '.' (DOT) character from field name using ES-Hadoop SerDe](https://discuss.elastic.co/t/removing-dot-character-from-field-name-using-es-hadoop-serde/49127) I'm also using HIVE's [pre-defined functions](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF) as well as [User Defined Functions](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateFunction) to transform my data and push them to ES. I believe these information may helped you to get started 🙂 Regards. ---
**Author:** ![Vinicius\_Silva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinicius_silva/32/15754_2.png) [@Vinicius\_Silva](https://discuss.elastic.co/u/Vinicius_Silva)\ **Post date:** [February 21, 2017, 5:52pm UTC](https://discuss.elastic.co/t/how-to-read-json-files-stored-in-hdfs-via-logstash/46954/4 "2017-02-21T17:52:30Z")
Thanks for the reply @gowthamsadasivam. I made a work around to avoid reading from HDFS 🙂 But I will keep your suggestions in mind in case I need to change. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:28am UTC](https://discuss.elastic.co/t/how-to-read-json-files-stored-in-hdfs-via-logstash/46954/5 "2017-07-06T04:28:22Z")