# How to read JSON files stored in HDFS via Logstash

**URL:** <https://discuss.elastic.co/t/how-to-read-json-files-stored-in-hdfs-via-logstash/46954>\
**Category:** Logstash\
**Created:** [April 11, 2016, 7:42am UTC](https://discuss.elastic.co/t/how-to-read-json-files-stored-in-hdfs-via-logstash/46954 "2016-04-11T07:42:08Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![gowthamsadasivam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gowthamsadasivam/32/9043_2.png) [@gowthamsadasivam](https://discuss.elastic.co/u/gowthamsadasivam)\
**Post date:** [February 7, 2017, 6:43am UTC](https://discuss.elastic.co/t/how-to-read-json-files-stored-in-hdfs-via-logstash/46954/3 "2017-02-07T06:43:09Z")

</div>

Hello Vinicius,

I didn't get any suggestions from this forum yet. But I figured two ways to achieve this.

**Method #1:**

If you have looked at the currently available output plug-ins for logstash, there is "[webhdfs](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-webhdfs.html)" plug-in which utilizes the [HDFS's REST API over HTTP](https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/WebHDFS.html).

I believe we can use this same API to read from HDFS using logstash [HTTP input plug-in](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html) available. By specifying the files location inside the HDFS and read it by HDFS REST API and process it, push it to ES.

**Method #2:** _(The one I'm using in production now)_

Using HIVE to push data directly to ES without using logstash. I'm using this method currently in my production cluster. There is[JAR integration](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html) available to push data to ES.

But, check the compatibility and [transform features](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Transform) available with-in HIVE before you use this method. If you use HIVE, you may not get all the transformation/manipulation methods available in logstash.

Here are few issues I went through while using HIVE with ES:

- [Mapping Hive Table Fields to Nested Fields in ElasticSearch](https://discuss.elastic.co/t/mapping-hive-table-fields-to-nested-fields-in-elasticsearch/47389)

- [Removing '.' (DOT) character from field name using ES-Hadoop SerDe](https://discuss.elastic.co/t/removing-dot-character-from-field-name-using-es-hadoop-serde/49127)

I'm also using HIVE's [pre-defined functions](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF) as well as [User Defined Functions](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateFunction) to transform my data and push them to ES.

I believe these information may helped you to get started 🙂

Regards.

---

_[View the full topic](https://discuss.elastic.co/t/how-to-read-json-files-stored-in-hdfs-via-logstash/46954)._