# Elasticsearch-hadoop and hadoop-streaming **URL:** \ **Category:** Elasticsearch\ **Created:** [May 5, 2014, 1:06pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-and-hadoop-streaming/17348 "2014-05-05T13:06:07Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Peter\_Sheridan](https://avatars.discourse-cdn.com/v4/letter/p/8baadc/32.png) [@Peter\_Sheridan](https://discuss.elastic.co/u/Peter_Sheridan)\ **Post date:** [May 5, 2014, 1:06pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-and-hadoop-streaming/17348/1 "2014-05-05T13:06:07Z")
Let's say I have an index, my\_twitter\_river, which has been populated by the Twitter river plugin. I want to do some analysis on the data using a Hadoop streaming job ([http://hadoop.apache.org/docs/r1.2.1/streaming.html](http://hadoop.apache.org/docs/r1.2.1/streaming.html)), where essentially command line programs can be used to write map/reduce jobs. Is there a good way for me to have the mapper for that streaming job receive parsable json using only available configuration options? What I see is this: my-macbook:misc psheridan$ hadoop jar /usr/lib/hadoop/contrib/streaming/hadoop-streaming-1.2.1.jar -D es.resource=my\_twitter\_river/status -D mapred.reduce.tasks=0 -inputformat org.elasticsearch.hadoop.mr.EsInputFormat -mapper /bin/cat -input in -output out 14/05/05 08:54:13 INFO mr.EsInputFormat: Discovered mapping {my\_twitter\_river=[mappings=[status=[created\_at=DATE, hashtag=[end=LONG, start=LONG, text=STRING], in\_reply=[status=LONG, user\_id=LONG, user\_screen\_name=STRING], language=STRING, link=[display\_url=STRING, end=LONG, expand\_url=STRING, start=LONG, url=STRING], location=GEO\_POINT, mention=[end=LONG, id=LONG, name=STRING, screen\_name=STRING, start=LONG], place=[country=STRING, country\_code=STRING, full\_name=STRING, id=STRING, name=STRING, type=STRING, url=STRING], retweet=[id=LONG, retweet\_count=LONG, user\_id=LONG, user\_screen\_name=STRING], retweet\_count=LONG, source=STRING, text=STRING, truncated=BOOLEAN, user=[description=STRING, id=LONG, location=STRING, name=STRING, profile\_image\_url=STRING, profile\_image\_url\_https=STRING, screen\_name=STRING]]]]} for [my\_twitter\_river/status] 14/05/05 08:54:13 INFO mr.EsInputFormat: Created [5] shard-splits [...output snipped...] my-macbook:misc psheridan$ hadoop dfs -ls out Found 7 items -rw-r--r-- 1 psheridan supergroup 0 2014-05-05 08:54 /user/psheridan/out/\_SUCCESS drwxr-xr-x - psheridan supergroup 0 2014-05-05 08:54 /user/psheridan/out/\_logs -rw-r--r-- 1 psheridan supergroup 10541885 2014-05-05 08:54 /user/psheridan/out/part-00000 -rw-r--r-- 1 psheridan supergroup 10252834 2014-05-05 08:54 /user/psheridan/out/part-00001 -rw-r--r-- 1 psheridan supergroup 10492008 2014-05-05 08:54 /user/psheridan/out/part-00002 -rw-r--r-- 1 psheridan supergroup 10497346 2014-05-05 08:54 /user/psheridan/out/part-00003 -rw-r--r-- 1 psheridan supergroup 10489611 2014-05-05 08:54 /user/psheridan/out/part-00004 my-macbook:misc psheridan$ dfs -cat /user/psheridan/out/part-00000 | head -2 458635348029886464 {text=RT @adrianrmante: Having lunch with my great friend OH MR. MOOSSEEBBYY!! I love that guy for life!! #thesuitelifeofzackandcody [http://t.co/…](http://t.co/%E2%80%A6), created\_at=2014-04-22T15:56:02.000Z, source=[Twitter for iPhone](http://twitter.com/download/iphone), truncated=false, language=en, mention=[{id=38270235, name=Adrian R'Mante, screen\_name=adrianrmante, start=3, end=16}], retweet\_count=0, retweet={id=456201545562873856, user\_id=38270235, user\_screen\_name=adrianrmante, retweet\_count=3279}, hashtag=[{text=thesuitelifeofzackandcody, start=100, end=126}], link=[], user={id=467951508, name=Marenna Nonya, screen\_name=ThatGirlish, location=(null), description=(null), profile\_image\_url=http://pbs.twimg.com/profile\_images/431121751456890880/pItAYIpY\_normal.jpeg, profile\_image\_url\_https=https://pbs.twimg.com/profile\_images/431121751456890880/pItAYIpY\_normal.jpeg}} 458635352232976384 {text=Dae vc sobe com seu amigão, ele fica de love e vc separando briga --' K, created\_at=2014-04-22T15:56:03.000Z, source=[Twitter for Android](http://twitter.com/download/android), truncated=false, language=pt, mention=[], retweet\_count=0, hashtag=[], link=[], user={id=2187296905, name=Carlinhos , screen\_name=AngeloChionpat0, location=(null), description=(null), profile\_image\_url=http://pbs.twimg.com/profile\_images/448849601844772864/ZNwdGDC7\_normal.jpeg, profile\_image\_url\_https=https://pbs.twimg.com/profile\_images/448849601844772864/ZNwdGDC7\_normal.jpeg}} When I cat the output files, I'd like to see valid json for the value. Thanks for any assistance...if this isn't possible, I'll likely submit a pull request when I get to it, which will not be as soon as I'd like. 🙂 --Pete -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a3fc229e-4bd3-4c31-ae0a-bec7362ba724%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3fc229e-4bd3-4c31-ae0a-bec7362ba724%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [May 5, 2014, 8:20pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-and-hadoop-streaming/17348/2 "2014-05-05T20:20:29Z")
Hi, Reading data in JSON format from ES (which I think is what you are interested in doing) is not available out of the box. Simply because you can do the same thing directly from the command line with curl or any http-like client. One of the reasons behind hadoop-streaming is to allow native clients to interact with Hadoop, primarily with HDF; since you are interacting with ES, why not talk to it directly? Am I missing something? On 5/5/14 4:06 PM, Peter Sheridan wrote: > Let's say I have an index, my\_twitter\_river, which has been populated by the Twitter river plugin. I want to do some > analysis on the data using a Hadoop streaming job ([Hadoop Streaming](http://hadoop.apache.org/docs/r1.2.1/streaming.html)), where > essentially command line programs can be used to write map/reduce jobs. > > Is there a good way for me to have the mapper for that streaming job receive parsable json using only available > configuration options? What I see is this: > > my-macbook:misc psheridan$ hadoop jar /usr/lib/hadoop/contrib/streaming/hadoop-streaming-1.2.1.jar > -D es.resource=my\_twitter\_river/status -D mapred.reduce.tasks=0 > -inputformat org.elasticsearch.hadoop.mr.EsInputFormat > -mapper /bin/cat -input in -output out > > 14/05/05 08:54:13 INFO mr.EsInputFormat: Discovered mapping {my\_twitter\_river=[mappings=[status=[created\_at=DATE, > hashtag=[end=LONG, start=LONG, text=STRING], in\_reply=[status=LONG, user\_id=LONG, user\_screen\_name=STRING], > language=STRING, link=[display\_url=STRING, end=LONG, expand\_url=STRING, start=LONG, url=STRING], location=GEO\_POINT, > mention=[end=LONG, id=LONG, name=STRING, screen\_name=STRING, start=LONG], place=[country=STRING, country\_code=STRING, > full\_name=STRING, id=STRING, name=STRING, type=STRING, url=STRING], retweet=[id=LONG, retweet\_count=LONG, user\_id=LONG, > user\_screen\_name=STRING], retweet\_count=LONG, source=STRING, text=STRING, truncated=BOOLEAN, user=[description=STRING, > id=LONG, location=STRING, name=STRING, profile\_image\_url=STRING, profile\_image\_url\_https=STRING, screen\_name=STRING]]]]} > for [my\_twitter\_river/status] > 14/05/05 08:54:13 INFO mr.EsInputFormat: Created [5] shard-splits > > [...output snipped...] > > my-macbook:misc psheridan$ hadoop dfs -ls out > Found 7 items > -rw-r--r-- 1 psheridan supergroup 0 2014-05-05 08:54 /user/psheridan/out/\_SUCCESS > drwxr-xr-x - psheridan supergroup 0 2014-05-05 08:54 /user/psheridan/out/\_logs > -rw-r--r-- 1 psheridan supergroup 10541885 2014-05-05 08:54 /user/psheridan/out/part-00000 > -rw-r--r-- 1 psheridan supergroup 10252834 2014-05-05 08:54 /user/psheridan/out/part-00001 > -rw-r--r-- 1 psheridan supergroup 10492008 2014-05-05 08:54 /user/psheridan/out/part-00002 > -rw-r--r-- 1 psheridan supergroup 10497346 2014-05-05 08:54 /user/psheridan/out/part-00003 > -rw-r--r-- 1 psheridan supergroup 10489611 2014-05-05 08:54 /user/psheridan/out/part-00004 > > my-macbook:misc psheridan$ dfs -cat /user/psheridan/out/part-00000 | head -2 > > 458635348029886464{text=RT @adrianrmante: Having lunch with my great friend OH MR. MOOSSEEBBYY!! I love that guy for > life!! #thesuitelifeofzackandcody [http://t.co/…](http://t.co/%E2%80%A6), created\_at=2014-04-22T15:56:02.000Z, source=[Twitter for iPhone](http://twitter.com/download/iphone), truncated=false, language=en, > mention=[{id=38270235, name=Adrian R'Mante, screen\_name=adrianrmante, start=3, end=16}], retweet\_count=0, > retweet={id=456201545562873856, user\_id=38270235, user\_screen\_name=adrianrmante, retweet\_count=3279}, > hashtag=[{text=thesuitelifeofzackandcody, start=100, end=126}], link=, user={id=467951508, name=Marenna Nonya, > screen\_name=ThatGirlish, location=(null), description=(null), > profile\_image\_url=[http://pbs.twimg.com/profile\_images/431121751456890880/pItAYIpY\_normal.jpeg](http://pbs.twimg.com/profile_images/431121751456890880/pItAYIpY_normal.jpeg), > profile\_image\_url\_https=[https://pbs.twimg.com/profile\_images/431121751456890880/pItAYIpY\_normal.jpeg](https://pbs.twimg.com/profile_images/431121751456890880/pItAYIpY_normal.jpeg)}} > 458635352232976384{text=Dae vc sobe com seu amigão, ele fica de love e vc separando briga --' K, > created\_at=2014-04-22T15:56:03.000Z, source=[Twitter for > Android](http://twitter.com/download/android), truncated=false, language=pt, mention=, retweet\_count=0, hashtag=, link=, user={id=2187296905, > name=Carlinhos , screen\_name=AngeloChionpat0, location=(null), description=(null), > profile\_image\_url=[http://pbs.twimg.com/profile\_images/448849601844772864/ZNwdGDC7\_normal.jpeg](http://pbs.twimg.com/profile_images/448849601844772864/ZNwdGDC7_normal.jpeg), > profile\_image\_url\_https=[https://pbs.twimg.com/profile\_images/448849601844772864/ZNwdGDC7\_normal.jpeg](https://pbs.twimg.com/profile_images/448849601844772864/ZNwdGDC7_normal.jpeg)}} > > When I cat the output files, I'd like to see valid json for the value. > > Thanks for any assistance...if this isn't possible, I'll likely submit a pull request when I get to it, which will not > be as soon as I'd like. 🙂 > > --Pete > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/a3fc229e-4bd3-4c31-ae0a-bec7362ba724%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3fc229e-4bd3-4c31-ae0a-bec7362ba724%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/a3fc229e-4bd3-4c31-ae0a-bec7362ba724%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a3fc229e-4bd3-4c31-ae0a-bec7362ba724%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5367F28D.1080502%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/5367F28D.1080502%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Peter\_Sheridan](https://avatars.discourse-cdn.com/v4/letter/p/8baadc/32.png) [@Peter\_Sheridan](https://discuss.elastic.co/u/Peter_Sheridan)\ **Post date:** [May 5, 2014, 8:35pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-and-hadoop-streaming/17348/3 "2014-05-05T20:35:54Z")
This is a simple example to illustrate the point. The real use case: 1. Is a rather large amount of data which I'd like to handle in parallel & also take advantage of es-hadoop's handling of shards. 2. Uses an existing job execution framework & toolset based on hadoop-streaming; I would rather not have a special case to handle it. Thanks for your feedback & nice work on es-hadoop, started using the Hive integration today as well. --Pete On Monday, May 5, 2014 4:20:29 PM UTC-4, Costin Leau wrote: > Hi, > > Reading data in JSON format from ES (which I think is what you are > interested in doing) is not available out of the box. > Simply because you can do the same thing directly from the command line > with curl or any http-like client. > One of the reasons behind hadoop-streaming is to allow native clients to > interact with Hadoop, primarily with HDF; since > you are > interacting with ES, why not talk to it directly? > > Am I missing something? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3bc7c0cb-fd97-402d-9644-97a60b2aa4b7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3bc7c0cb-fd97-402d-9644-97a60b2aa4b7%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [May 5, 2014, 9:21pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-and-hadoop-streaming/17348/4 "2014-05-05T21:21:45Z")
How would the json format help you though? If you plan parse the output, using a simpler text representation (such as TextOutputFormat) should be a lot easier. On 5/5/14 11:35 PM, Peter Sheridan wrote: > This is a simple example to illustrate the point. The real use case: > > 1. Is a rather large amount of data which I'd like to handle in parallel & also take advantage of es-hadoop's handling > of shards. > 2. Uses an existing job execution framework & toolset based on hadoop-streaming; I would rather not have a special case > to handle it. > > Thanks for your feedback & nice work on es-hadoop, started using the Hive integration today as well. > > --Pete > > On Monday, May 5, 2014 4:20:29 PM UTC-4, Costin Leau wrote: > > ``` > Hi, > > Reading data in JSON format from ES (which I think is what you are interested in doing) is not available out of the > box. > Simply because you can do the same thing directly from the command line with curl or any http-like client. > One of the reasons behind hadoop-streaming is to allow native clients to interact with Hadoop, primarily with HDF; > since > you are > interacting with ES, why not talk to it directly? > > Am I missing something? > > ``` > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/3bc7c0cb-fd97-402d-9644-97a60b2aa4b7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3bc7c0cb-fd97-402d-9644-97a60b2aa4b7%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/3bc7c0cb-fd97-402d-9644-97a60b2aa4b7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/3bc7c0cb-fd97-402d-9644-97a60b2aa4b7%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/536800E9.8090805%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/536800E9.8090805%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:31am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-and-hadoop-streaming/17348/5 "2017-07-06T01:31:34Z")