# Elasticsearch with Hadoop
**URL:** https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623
**Category:** Elasticsearch
**Created:** [May 9, 2012, 11:18pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623 "2012-05-09T23:18:58Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)
#### Post date: [May 9, 2012, 11:18pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/1 "2012-05-09T23:18:58Z")
Is anyone using elasticsearch with Hadoop? Would like to know how it's used
and any suggestions that you can provide would be helpful.
---
### Author:  [@Jerome](https://discuss.elastic.co/u/Jerome)
#### Post date: [May 10, 2012, 8:02am UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/2 "2012-05-10T08:02:51Z")
Aha, i follow this topic, have to do the same thing, you need it for
what exactly ?
On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> Is anyone using elasticsearch with Hadoop? Would like to know how it's used
> and any suggestions that you can provide would be helpful.
---
### Author:  [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)
#### Post date: [May 10, 2012, 2:21pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/3 "2012-05-10T14:21:21Z")
We have a hadoop cluster and I would like to use HDFS as the storage so
that I have one place where everything is stored and tracked.
On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com) wrote:
> Aha, i follow this topic, have to do the same thing, you need it for
> what exactly ?
>
> On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
>
> > Is anyone using elasticsearch with Hadoop? Would like to know how it's
> > used
> > and any suggestions that you can provide would be helpful.
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [May 10, 2012, 9:08pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/4 "2012-05-10T21:08:25Z")
I have never used it, but have you looked at the Hadoop plugin?
> **[GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and...](https://github.com/elastic/elasticsearch-hadoop)**
>
> :elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop - GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated wi...
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote:
> We have a hadoop cluster and I would like to use HDFS as the storage so that
> I have one place where everything is stored and tracked.
>
> On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com) wrote:
>
> > Aha, i follow this topic, have to do the same thing, you need it for
> > what exactly ?
> >
> > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> >
> > > Is anyone using elasticsearch with Hadoop? Would like to know how it's
> > > used
> > > and any suggestions that you can provide would be helpful.
---
### Author:  [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)
#### Post date: [May 10, 2012, 9:20pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/5 "2012-05-10T21:20:01Z")
Looks like that plugin is to use it for backups? It's not clear if it
stores all the data in HDFS.
On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> I have never used it, but have you looked at the Hadoop plugin?
>
> [GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
>
> On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> wrote:
>
> > We have a hadoop cluster and I would like to use HDFS as the storage so
> > that
> > I have one place where everything is stored and tracked.
> >
> > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com) wrote:
> >
> > > Aha, i follow this topic, have to do the same thing, you need it for
> > > what exactly ?
> > >
> > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > >
> > > > Is anyone using elasticsearch with Hadoop? Would like to know how it's
> > > > used
> > > > and any suggestions that you can provide would be helpful.
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [May 10, 2012, 9:26pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/6 "2012-05-10T21:26:39Z")
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote:
> Looks like that plugin is to use it for backups? It's not clear if it stores
> all the data in HDFS.
>
> On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > I have never used it, but have you looked at the Hadoop plugin?
> >
> > [GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> >
> > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > wrote:
> >
> > > We have a hadoop cluster and I would like to use HDFS as the storage so
> > > that
> > > I have one place where everything is stored and tracked.
> > >
> > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com) wrote:
> > >
> > > > Aha, i follow this topic, have to do the same thing, you need it for
> > > > what exactly ?
> > > >
> > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > >
> > > > > Is anyone using elasticsearch with Hadoop? Would like to know how
> > > > > it's
> > > > > used
> > > > > and any suggestions that you can provide would be helpful.
---
### Author:  [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)
#### Post date: [May 10, 2012, 9:30pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/7 "2012-05-10T21:30:36Z")
Is there something that elasticsearch stores locally? I see we still need
to configure workdir.
On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> HDFS will be the gateway for ES, storing both the index and metadata.
>
> On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> wrote:
>
> > Looks like that plugin is to use it for backups? It's not clear if it
> > stores
> > all the data in HDFS.
> >
> > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> >
> > > I have never used it, but have you looked at the Hadoop plugin?
> > >
> > > [GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > >
> > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > wrote:
> > >
> > > > We have a hadoop cluster and I would like to use HDFS as the storage
> > > > so
> > > > that
> > > > I have one place where everything is stored and tracked.
> > > >
> > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > wrote:
> > > >
> > > > > Aha, i follow this topic, have to do the same thing, you need it for
> > > > > what exactly ?
> > > > >
> > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > >
> > > > > > Is anyone using elasticsearch with Hadoop? Would like to know how
> > > > > > it's
> > > > > > used
> > > > > > and any suggestions that you can provide would be helpful.
---
### Author:  [@Andrew\_at\_DataFeedFi](https://discuss.elastic.co/u/Andrew_at_DataFeedFi)
#### Post date: [May 10, 2012, 10:40pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/8 "2012-05-10T22:40:36Z")
Yes, it is call Gateway Local (basically your local filesystem)
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> Is there something that elasticsearch stores locally? I see we still need
> to configure workdir.
>
> On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > HDFS will be the gateway for ES, storing both the index and metadata.
> >
> > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > wrote:
> >
> > > Looks like that plugin is to use it for backups? It's not clear if it
> > > stores
> > > all the data in HDFS.
> > >
> > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > >
> > > > I have never used it, but have you looked at the Hadoop plugin?
> > > >
> > > > [GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > >
> > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<[mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > >
> > > > wrote:
> > > >
> > > > > We have a hadoop cluster and I would like to use HDFS as the storage
> > > > > so
> > > > > that
> > > > > I have one place where everything is stored and tracked.
> > > > >
> > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > Aha, i follow this topic, have to do the same thing, you need it for
> > > > > > what exactly ?
> > > > > >
> > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > >
> > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know how
> > > > > > > it's
> > > > > > > used
> > > > > > > and any suggestions that you can provide would be helpful.
On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> Is there something that elasticsearch stores locally? I see we still need
> to configure workdir.
>
> On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > HDFS will be the gateway for ES, storing both the index and metadata.
> >
> > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > wrote:
> >
> > > Looks like that plugin is to use it for backups? It's not clear if it
> > > stores
> > > all the data in HDFS.
> > >
> > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > >
> > > > I have never used it, but have you looked at the Hadoop plugin?
> > > >
> > > > [GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > >
> > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<[mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > >
> > > > wrote:
> > > >
> > > > > We have a hadoop cluster and I would like to use HDFS as the storage
> > > > > so
> > > > > that
> > > > > I have one place where everything is stored and tracked.
> > > > >
> > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > Aha, i follow this topic, have to do the same thing, you need it for
> > > > > > what exactly ?
> > > > > >
> > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > >
> > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know how
> > > > > > > it's
> > > > > > > used
> > > > > > > and any suggestions that you can provide would be helpful.
On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> Is there something that elasticsearch stores locally? I see we still need
> to configure workdir.
>
> On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > HDFS will be the gateway for ES, storing both the index and metadata.
> >
> > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > wrote:
> >
> > > Looks like that plugin is to use it for backups? It's not clear if it
> > > stores
> > > all the data in HDFS.
> > >
> > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > >
> > > > I have never used it, but have you looked at the Hadoop plugin?
> > > >
> > > > [GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > >
> > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<[mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > >
> > > > wrote:
> > > >
> > > > > We have a hadoop cluster and I would like to use HDFS as the storage
> > > > > so
> > > > > that
> > > > > I have one place where everything is stored and tracked.
> > > > >
> > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > Aha, i follow this topic, have to do the same thing, you need it for
> > > > > > what exactly ?
> > > > > >
> > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > >
> > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know how
> > > > > > > it's
> > > > > > > used
> > > > > > > and any suggestions that you can provide would be helpful.
On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> Is there something that elasticsearch stores locally? I see we still need
> to configure workdir.
>
> On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > HDFS will be the gateway for ES, storing both the index and metadata.
> >
> > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > wrote:
> >
> > > Looks like that plugin is to use it for backups? It's not clear if it
> > > stores
> > > all the data in HDFS.
> > >
> > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > >
> > > > I have never used it, but have you looked at the Hadoop plugin?
> > > >
> > > > [GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > >
> > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<[mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > >
> > > > wrote:
> > > >
> > > > > We have a hadoop cluster and I would like to use HDFS as the storage
> > > > > so
> > > > > that
> > > > > I have one place where everything is stored and tracked.
> > > > >
> > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > Aha, i follow this topic, have to do the same thing, you need it for
> > > > > > what exactly ?
> > > > > >
> > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > >
> > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know how
> > > > > > > it's
> > > > > > > used
> > > > > > > and any suggestions that you can provide would be helpful.
On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> Is there something that elasticsearch stores locally? I see we still need
> to configure workdir.
>
> On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
>
> > HDFS will be the gateway for ES, storing both the index and metadata.
> >
> > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > wrote:
> >
> > > Looks like that plugin is to use it for backups? It's not clear if it
> > > stores
> > > all the data in HDFS.
> > >
> > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > >
> > > > I have never used it, but have you looked at the Hadoop plugin?
> > > >
> > > > [GitHub - elastic/elasticsearch-hadoop: Elasticsearch real-time search and analytics natively integrated with Hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > >
> > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<[mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > >
> > > > wrote:
> > > >
> > > > > We have a hadoop cluster and I would like to use HDFS as the storage
> > > > > so
> > > > > that
> > > > > I have one place where everything is stored and tracked.
> > > > >
> > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > Aha, i follow this topic, have to do the same thing, you need it for
> > > > > > what exactly ?
> > > > > >
> > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > >
> > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know how
> > > > > > > it's
> > > > > > > used
> > > > > > > and any suggestions that you can provide would be helpful.
---
### Author:  [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)
#### Post date: [May 10, 2012, 11:48pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/9 "2012-05-10T23:48:03Z")
Actually I was referring to "data" and "work" paths. What are these for? I
am still confused what exactly gets stored locally. I also read on the page
that "gateway" is used for backups. If that's the case then everything is
also getting stored locally?
path:
home: /usr/local/share/elasticsearch
conf: /etc/elasticsearch
logs: /var/log/elasticsearch
# data: /mnt/elasticsearch/data
# work: /mnt/elasticsearch/work
On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
[andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
> Yes, it is call Gateway Local (basically your local filesystem)
>
> [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/modules/gateway/local.html)
>
> On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
>
> > Is there something that elasticsearch stores locally? I see we still need
> > to configure workdir.
> >
> > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> >
> > > HDFS will be the gateway for ES, storing both the index and metadata.
> > >
> > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > wrote:
> > >
> > > > Looks like that plugin is to use it for backups? It's not clear if it
> > > > stores
> > > > all the data in HDFS.
> > > >
> > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > >
> > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > >
> > > > > [https://github.com/\*\*elasticsearch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsearch/elasticsearch-** hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > >
> > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > wrote:
> > > > >
> > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > storage so
> > > > > > that
> > > > > > I have one place where everything is stored and tracked.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > wrote:
> > > > > >
> > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > for
> > > > > > > what exactly ?
> > > > > > >
> > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > >
> > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know how
> > > > > > > > it's
> > > > > > > > used
> > > > > > > > and any suggestions that you can provide would be helpful.
>
> On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
>
> > Is there something that elasticsearch stores locally? I see we still need
> > to configure workdir.
> >
> > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> >
> > > HDFS will be the gateway for ES, storing both the index and metadata.
> > >
> > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > wrote:
> > >
> > > > Looks like that plugin is to use it for backups? It's not clear if it
> > > > stores
> > > > all the data in HDFS.
> > > >
> > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > >
> > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > >
> > > > > [https://github.com/\*\*elasticsearch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsearch/elasticsearch-** hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > >
> > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > wrote:
> > > > >
> > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > storage so
> > > > > > that
> > > > > > I have one place where everything is stored and tracked.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > wrote:
> > > > > >
> > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > for
> > > > > > > what exactly ?
> > > > > > >
> > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > >
> > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know how
> > > > > > > > it's
> > > > > > > > used
> > > > > > > > and any suggestions that you can provide would be helpful.
>
> On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
>
> > Is there something that elasticsearch stores locally? I see we still need
> > to configure workdir.
> >
> > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> >
> > > HDFS will be the gateway for ES, storing both the index and metadata.
> > >
> > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > wrote:
> > >
> > > > Looks like that plugin is to use it for backups? It's not clear if it
> > > > stores
> > > > all the data in HDFS.
> > > >
> > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > >
> > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > >
> > > > > [https://github.com/\*\*elasticsearch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsearch/elasticsearch-** hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > >
> > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > wrote:
> > > > >
> > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > storage so
> > > > > > that
> > > > > > I have one place where everything is stored and tracked.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > wrote:
> > > > > >
> > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > for
> > > > > > > what exactly ?
> > > > > > >
> > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > >
> > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know how
> > > > > > > > it's
> > > > > > > > used
> > > > > > > > and any suggestions that you can provide would be helpful.
>
> On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
>
> > Is there something that elasticsearch stores locally? I see we still need
> > to configure workdir.
> >
> > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> >
> > > HDFS will be the gateway for ES, storing both the index and metadata.
> > >
> > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > wrote:
> > >
> > > > Looks like that plugin is to use it for backups? It's not clear if it
> > > > stores
> > > > all the data in HDFS.
> > > >
> > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > >
> > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > >
> > > > > [https://github.com/\*\*elasticsearch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsearch/elasticsearch-** hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > >
> > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > wrote:
> > > > >
> > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > storage so
> > > > > > that
> > > > > > I have one place where everything is stored and tracked.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > wrote:
> > > > > >
> > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > for
> > > > > > > what exactly ?
> > > > > > >
> > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > >
> > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know how
> > > > > > > > it's
> > > > > > > > used
> > > > > > > > and any suggestions that you can provide would be helpful.
>
> On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
>
> > Is there something that elasticsearch stores locally? I see we still need
> > to configure workdir.
> >
> > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> >
> > > HDFS will be the gateway for ES, storing both the index and metadata.
> > >
> > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > wrote:
> > >
> > > > Looks like that plugin is to use it for backups? It's not clear if it
> > > > stores
> > > > all the data in HDFS.
> > > >
> > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > >
> > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > >
> > > > > [https://github.com/\*\*elasticsearch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsearch/elasticsearch-** hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > >
> > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > wrote:
> > > > >
> > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > storage so
> > > > > > that
> > > > > > I have one place where everything is stored and tracked.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > wrote:
> > > > > >
> > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > for
> > > > > > > what exactly ?
> > > > > > >
> > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > >
> > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know how
> > > > > > > > it's
> > > > > > > > used
> > > > > > > > and any suggestions that you can provide would be helpful.
---
### Author:  [@Andrew\_at\_DataFeedFi](https://discuss.elastic.co/u/Andrew_at_DataFeedFi)
#### Post date: [May 11, 2012, 4:36pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/10 "2012-05-11T16:36:12Z")
Mo,
There are many types of Gateway, I think the one you are interested in is
the LOCAL gateway only.
Because each Gateway has different purpose, settings and behavior, I just
want to be clear that I am only describing LOCAL gateway here.
For local gateway, your ES indices and metadata are all stored to whatever
directory you set your data path to. Most of the time it is probably:
\/data
As for the work path, I am not sure what that is for, probably for
temporary working area (scratchpad?), I have not seen it being filled in
with any there lately (after version 0.18).
Actually on some of my installs 0.18+ we do not even have work path
anymore... so perhaps it is no longer needed? Maybe someone can confirm on
this?
Thanks
--Andrew
On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
> Actually I was referring to "data" and "work" paths. What are these for? I
> am still confused what exactly gets stored locally. I also read on the page
> that "gateway" is used for backups. If that's the case then everything is
> also getting stored locally?
>
> path:
> home: /usr/local/share/elasticsearch
> conf: /etc/elasticsearch
> logs: /var/log/elasticsearch
>
> # data: /mnt/elasticsearch/data
>
> # work: /mnt/elasticsearch/work
>
> On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> [andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
>
> > Yes, it is call Gateway Local (basically your local filesystem)
> >
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/gateway/local.html)
> >
> > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> >
> > > Is there something that elasticsearch stores locally? I see we still
> > > need to configure workdir.
> > >
> > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > >
> > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > >
> > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > > wrote:
> > > >
> > > > > Looks like that plugin is to use it for backups? It's not clear if it
> > > > > stores
> > > > > all the data in HDFS.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > >
> > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > >
> > > > > > [https://github.com/\*\*elasticsearch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsearch/elasticsearch-** hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > >
> > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > wrote:
> > > > > >
> > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > storage so
> > > > > > > that
> > > > > > > I have one place where everything is stored and tracked.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > > for
> > > > > > > > what exactly ?
> > > > > > > >
> > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > >
> > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > how
> > > > > > > > > it's
> > > > > > > > > used
> > > > > > > > > and any suggestions that you can provide would be helpful.
> >
> > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> >
> > > Is there something that elasticsearch stores locally? I see we still
> > > need to configure workdir.
> > >
> > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > >
> > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > >
> > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > > wrote:
> > > >
> > > > > Looks like that plugin is to use it for backups? It's not clear if it
> > > > > stores
> > > > > all the data in HDFS.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > >
> > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > >
> > > > > > [https://github.com/\*\*elasticsearch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsearch/elasticsearch-** hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > >
> > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > wrote:
> > > > > >
> > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > storage so
> > > > > > > that
> > > > > > > I have one place where everything is stored and tracked.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > > for
> > > > > > > > what exactly ?
> > > > > > > >
> > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > >
> > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > how
> > > > > > > > > it's
> > > > > > > > > used
> > > > > > > > > and any suggestions that you can provide would be helpful.
> >
> > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> >
> > > Is there something that elasticsearch stores locally? I see we still
> > > need to configure workdir.
> > >
> > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > >
> > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > >
> > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > > wrote:
> > > >
> > > > > Looks like that plugin is to use it for backups? It's not clear if it
> > > > > stores
> > > > > all the data in HDFS.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > >
> > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > >
> > > > > > [https://github.com/\*\*elasticsearch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsearch/elasticsearch-** hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > >
> > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > wrote:
> > > > > >
> > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > storage so
> > > > > > > that
> > > > > > > I have one place where everything is stored and tracked.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > > for
> > > > > > > > what exactly ?
> > > > > > > >
> > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > >
> > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > how
> > > > > > > > > it's
> > > > > > > > > used
> > > > > > > > > and any suggestions that you can provide would be helpful.
> >
> > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> >
> > > Is there something that elasticsearch stores locally? I see we still
> > > need to configure workdir.
> > >
> > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > >
> > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > >
> > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > > wrote:
> > > >
> > > > > Looks like that plugin is to use it for backups? It's not clear if it
> > > > > stores
> > > > > all the data in HDFS.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > >
> > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > >
> > > > > > [https://github.com/\*\*elasticsearch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsearch/elasticsearch-** hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > >
> > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > wrote:
> > > > > >
> > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > storage so
> > > > > > > that
> > > > > > > I have one place where everything is stored and tracked.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > > for
> > > > > > > > what exactly ?
> > > > > > > >
> > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > >
> > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > how
> > > > > > > > > it's
> > > > > > > > > used
> > > > > > > > > and any suggestions that you can provide would be helpful.
> >
> > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> >
> > > Is there something that elasticsearch stores locally? I see we still
> > > need to configure workdir.
> > >
> > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > >
> > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > >
> > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > > wrote:
> > > >
> > > > > Looks like that plugin is to use it for backups? It's not clear if it
> > > > > stores
> > > > > all the data in HDFS.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > >
> > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > >
> > > > > > [https://github.com/\*\*elasticsearch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsearch/elasticsearch-** hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > >
> > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > wrote:
> > > > > >
> > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > storage so
> > > > > > > that
> > > > > > > I have one place where everything is stored and tracked.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > > for
> > > > > > > > what exactly ?
> > > > > > > >
> > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > >
> > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > how
> > > > > > > > > it's
> > > > > > > > > used
> > > > > > > > > and any suggestions that you can provide would be helpful.
---
### Author:  [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)
#### Post date: [May 11, 2012, 4:53pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/11 "2012-05-11T16:53:47Z")
So if I understand correctly you are saying if I point my gateway to HDFS
"everything" including "documents", "data", "indexes", "terms" gets stored
in HDFS?
There is nothing that would get stored locally? Is that correct?
On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
[andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
> Mo,
>
> There are many types of Gateway, I think the one you are interested in is
> the LOCAL gateway only.
> Because each Gateway has different purpose, settings and behavior, I just
> want to be clear that I am only describing LOCAL gateway here.
>
> For local gateway, your ES indices and metadata are all stored to whatever
> directory you set your data path to. Most of the time it is probably:
>
> \/data
>
> As for the work path, I am not sure what that is for, probably for
> temporary working area (scratchpad?), I have not seen it being filled in
> with any there lately (after version 0.18).
>
> Actually on some of my installs 0.18+ we do not even have work path
> anymore... so perhaps it is no longer needed? Maybe someone can confirm on
> this?
>
> Thanks
>
> --Andrew
>
> On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
>
> > Actually I was referring to "data" and "work" paths. What are these for?
> > I am still confused what exactly gets stored locally. I also read on the
> > page that "gateway" is used for backups. If that's the case then everything
> > is also getting stored locally?
> >
> > path:
> > home: /usr/local/share/elasticsearch
> > conf: /etc/elasticsearch
> > logs: /var/log/elasticsearch
> >
> > # data: /mnt/elasticsearch/data
> >
> > # work: /mnt/elasticsearch/work
> >
> > On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> > [andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
> >
> > > Yes, it is call Gateway Local (basically your local filesystem)
> > >
> > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/modules/)\*\*
> > > gateway/local.html[http://www.elasticsearch.org/guide/reference/modules/gateway/local.html](http://www.elasticsearch.org/guide/reference/modules/gateway/local.html)
> > >
> > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > >
> > > > Is there something that elasticsearch stores locally? I see we still
> > > > need to configure workdir.
> > > >
> > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > >
> > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > it stores
> > > > > > all the data in HDFS.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > wrote:
> > > > > >
> > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > >
> > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > storage so
> > > > > > > > that
> > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > > > for
> > > > > > > > > what exactly ?
> > > > > > > > >
> > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > > >
> > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > > how
> > > > > > > > > > it's
> > > > > > > > > > used
> > > > > > > > > > and any suggestions that you can provide would be helpful.
> > >
> > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > >
> > > > Is there something that elasticsearch stores locally? I see we still
> > > > need to configure workdir.
> > > >
> > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > >
> > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > it stores
> > > > > > all the data in HDFS.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > wrote:
> > > > > >
> > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > >
> > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > storage so
> > > > > > > > that
> > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > > > for
> > > > > > > > > what exactly ?
> > > > > > > > >
> > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > > >
> > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > > how
> > > > > > > > > > it's
> > > > > > > > > > used
> > > > > > > > > > and any suggestions that you can provide would be helpful.
> > >
> > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > >
> > > > Is there something that elasticsearch stores locally? I see we still
> > > > need to configure workdir.
> > > >
> > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > >
> > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > it stores
> > > > > > all the data in HDFS.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > wrote:
> > > > > >
> > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > >
> > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > storage so
> > > > > > > > that
> > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > > > for
> > > > > > > > > what exactly ?
> > > > > > > > >
> > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > > >
> > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > > how
> > > > > > > > > > it's
> > > > > > > > > > used
> > > > > > > > > > and any suggestions that you can provide would be helpful.
> > >
> > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > >
> > > > Is there something that elasticsearch stores locally? I see we still
> > > > need to configure workdir.
> > > >
> > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > >
> > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > it stores
> > > > > > all the data in HDFS.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > wrote:
> > > > > >
> > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > >
> > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > storage so
> > > > > > > > that
> > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > > > for
> > > > > > > > > what exactly ?
> > > > > > > > >
> > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > > >
> > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > > how
> > > > > > > > > > it's
> > > > > > > > > > used
> > > > > > > > > > and any suggestions that you can provide would be helpful.
> > >
> > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > >
> > > > Is there something that elasticsearch stores locally? I see we still
> > > > need to configure workdir.
> > > >
> > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > >
> > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> > > > > wrote:
> > > > >
> > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > it stores
> > > > > > all the data in HDFS.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > wrote:
> > > > > >
> > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > >
> > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > storage so
> > > > > > > > that
> > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Aha, i follow this topic, have to do the same thing, you need it
> > > > > > > > > for
> > > > > > > > > what exactly ?
> > > > > > > > >
> > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > > >
> > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > > how
> > > > > > > > > > it's
> > > > > > > > > > used
> > > > > > > > > > and any suggestions that you can provide would be helpful.
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 15, 2012, 8:00pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/12 "2012-05-15T20:00:12Z")
No, the HDFS gateway will still mean that data will be stored locally (all
the indexes) on each node, and they will be snapshotted to HDFS. Using HDFS
as the main store is pointeless performance wise.
On Fri, May 11, 2012 at 7:53 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote:
> So if I understand correctly you are saying if I point my gateway to HDFS
> "everything" including "documents", "data", "indexes", "terms" gets stored
> in HDFS?
>
> There is nothing that would get stored locally? Is that correct?
> On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> [andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
>
> > Mo,
> >
> > There are many types of Gateway, I think the one you are interested in is
> > the LOCAL gateway only.
> > Because each Gateway has different purpose, settings and behavior, I just
> > want to be clear that I am only describing LOCAL gateway here.
> >
> > For local gateway, your ES indices and metadata are all stored to
> > whatever directory you set your data path to. Most of the time it is
> > probably:
> >
> > \/data
> >
> > As for the work path, I am not sure what that is for, probably for
> > temporary working area (scratchpad?), I have not seen it being filled in
> > with any there lately (after version 0.18).
> >
> > Actually on some of my installs 0.18+ we do not even have work path
> > anymore... so perhaps it is no longer needed? Maybe someone can confirm on
> > this?
> >
> > Thanks
> >
> > --Andrew
> >
> > On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
> >
> > > Actually I was referring to "data" and "work" paths. What are these for?
> > > I am still confused what exactly gets stored locally. I also read on the
> > > page that "gateway" is used for backups. If that's the case then everything
> > > is also getting stored locally?
> > >
> > > path:
> > > home: /usr/local/share/elasticsearch
> > > conf: /etc/elasticsearch
> > > logs: /var/log/elasticsearch
> > >
> > > # data: /mnt/elasticsearch/data
> > >
> > > # work: /mnt/elasticsearch/work
> > >
> > > On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> > > [andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
> > >
> > > > Yes, it is call Gateway Local (basically your local filesystem)
> > > >
> > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/modules/)\*\*
> > > > gateway/local.html[http://www.elasticsearch.org/guide/reference/modules/gateway/local.html](http://www.elasticsearch.org/guide/reference/modules/gateway/local.html)
> > > >
> > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > >
> > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > need to configure workdir.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > >
> > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > >
> > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > it stores
> > > > > > > all the data in HDFS.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > >
> > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > storage so
> > > > > > > > > that
> > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > it for
> > > > > > > > > > what exactly ?
> > > > > > > > > >
> > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > > > >
> > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > > > how
> > > > > > > > > > > it's
> > > > > > > > > > > used
> > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > >
> > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > >
> > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > need to configure workdir.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > >
> > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > >
> > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > it stores
> > > > > > > all the data in HDFS.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > >
> > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > storage so
> > > > > > > > > that
> > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > it for
> > > > > > > > > > what exactly ?
> > > > > > > > > >
> > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > > > >
> > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > > > how
> > > > > > > > > > > it's
> > > > > > > > > > > used
> > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > >
> > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > >
> > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > need to configure workdir.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > >
> > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > >
> > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > it stores
> > > > > > > all the data in HDFS.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > >
> > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > storage so
> > > > > > > > > that
> > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > it for
> > > > > > > > > > what exactly ?
> > > > > > > > > >
> > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > > > >
> > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > > > how
> > > > > > > > > > > it's
> > > > > > > > > > > used
> > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > >
> > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > >
> > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > need to configure workdir.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > >
> > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > >
> > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > it stores
> > > > > > > all the data in HDFS.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > >
> > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > storage so
> > > > > > > > > that
> > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > it for
> > > > > > > > > > what exactly ?
> > > > > > > > > >
> > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > > > >
> > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > > > how
> > > > > > > > > > > it's
> > > > > > > > > > > used
> > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > >
> > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > >
> > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > need to configure workdir.
> > > > >
> > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > >
> > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > >
> > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > it stores
> > > > > > > all the data in HDFS.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > >
> > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > storage so
> > > > > > > > > that
> > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > it for
> > > > > > > > > > what exactly ?
> > > > > > > > > >
> > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com) wrote:
> > > > > > > > > >
> > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to know
> > > > > > > > > > > how
> > > > > > > > > > > it's
> > > > > > > > > > > used
> > > > > > > > > > > and any suggestions that you can provide would be helpful.
---
### Author:  [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)
#### Post date: [May 15, 2012, 8:39pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/13 "2012-05-15T20:39:36Z")
thanks, how often does it get snapshot?
Also, how often does elasticsearch refresh data to disk from memory?
On Tue, May 15, 2012 at 1:00 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> No, the HDFS gateway will still mean that data will be stored locally (all
> the indexes) on each node, and they will be snapshotted to HDFS. Using HDFS
> as the main store is pointeless performance wise.
>
> On Fri, May 11, 2012 at 7:53 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote:
>
> > So if I understand correctly you are saying if I point my gateway to HDFS
> > "everything" including "documents", "data", "indexes", "terms" gets stored
> > in HDFS?
> >
> > There is nothing that would get stored locally? Is that correct?
> > On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> > [andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
> >
> > > Mo,
> > >
> > > There are many types of Gateway, I think the one you are interested in
> > > is the LOCAL gateway only.
> > > Because each Gateway has different purpose, settings and behavior, I
> > > just want to be clear that I am only describing LOCAL gateway here.
> > >
> > > For local gateway, your ES indices and metadata are all stored to
> > > whatever directory you set your data path to. Most of the time it is
> > > probably:
> > >
> > > \/data
> > >
> > > As for the work path, I am not sure what that is for, probably for
> > > temporary working area (scratchpad?), I have not seen it being filled in
> > > with any there lately (after version 0.18).
> > >
> > > Actually on some of my installs 0.18+ we do not even have work path
> > > anymore... so perhaps it is no longer needed? Maybe someone can confirm on
> > > this?
> > >
> > > Thanks
> > >
> > > --Andrew
> > >
> > > On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
> > >
> > > > Actually I was referring to "data" and "work" paths. What are these
> > > > for? I am still confused what exactly gets stored locally. I also read on
> > > > the page that "gateway" is used for backups. If that's the case then
> > > > everything is also getting stored locally?
> > > >
> > > > path:
> > > > home: /usr/local/share/elasticsearch
> > > > conf: /etc/elasticsearch
> > > > logs: /var/log/elasticsearch
> > > >
> > > > # data: /mnt/elasticsearch/data
> > > >
> > > > # work: /mnt/elasticsearch/work
> > > >
> > > > On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> > > > [andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
> > > >
> > > > > Yes, it is call Gateway Local (basically your local filesystem)
> > > > >
> > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/modules/)\*\*
> > > > > gateway/local.html[http://www.elasticsearch.org/guide/reference/modules/gateway/local.html](http://www.elasticsearch.org/guide/reference/modules/gateway/local.html)
> > > > >
> > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > >
> > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > need to configure workdir.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > > >
> > > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > >
> > > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > > it stores
> > > > > > > > all the data in HDFS.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > >
> > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > storage so
> > > > > > > > > > that
> > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > it for
> > > > > > > > > > > what exactly ?
> > > > > > > > > > >
> > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > know how
> > > > > > > > > > > > it's
> > > > > > > > > > > > used
> > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > >
> > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > >
> > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > need to configure workdir.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > > >
> > > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > >
> > > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > > it stores
> > > > > > > > all the data in HDFS.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > >
> > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > storage so
> > > > > > > > > > that
> > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > it for
> > > > > > > > > > > what exactly ?
> > > > > > > > > > >
> > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > know how
> > > > > > > > > > > > it's
> > > > > > > > > > > > used
> > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > >
> > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > >
> > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > need to configure workdir.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > > >
> > > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > >
> > > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > > it stores
> > > > > > > > all the data in HDFS.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > >
> > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > storage so
> > > > > > > > > > that
> > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > it for
> > > > > > > > > > > what exactly ?
> > > > > > > > > > >
> > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > know how
> > > > > > > > > > > > it's
> > > > > > > > > > > > used
> > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > >
> > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > >
> > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > need to configure workdir.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > > >
> > > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > >
> > > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > > it stores
> > > > > > > > all the data in HDFS.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > >
> > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > storage so
> > > > > > > > > > that
> > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > it for
> > > > > > > > > > > what exactly ?
> > > > > > > > > > >
> > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > know how
> > > > > > > > > > > > it's
> > > > > > > > > > > > used
> > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > >
> > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > >
> > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > need to configure workdir.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> > > > > >
> > > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > >
> > > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > > it stores
> > > > > > > > all the data in HDFS.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > >
> > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > storage so
> > > > > > > > > > that
> > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > it for
> > > > > > > > > > > what exactly ?
> > > > > > > > > > >
> > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > know how
> > > > > > > > > > > > it's
> > > > > > > > > > > > used
> > > > > > > > > > > > and any suggestions that you can provide would be helpful.
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 16, 2012, 7:48pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/14 "2012-05-16T19:48:07Z")
On Tue, May 15, 2012 at 11:39 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote:
> thanks, how often does it get snapshot?
By default every 10 seconds, though snapshot can take a long time
potentially. Note though, during recovery, it will always recover from the
last snapshot. Thats why the local gateway is preferable (and what is
missing is a "backup" API to backup the data to HDFS, for example).
> Also, how often does elasticsearch refresh data to disk from memory?
What do you mean? How often it gets "refreshed" for new data indexed to
become searchable?
> On Tue, May 15, 2012 at 1:00 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
>
> > No, the HDFS gateway will still mean that data will be stored locally
> > (all the indexes) on each node, and they will be snapshotted to HDFS. Using
> > HDFS as the main store is pointeless performance wise.
> >
> > On Fri, May 11, 2012 at 7:53 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote:
> >
> > > So if I understand correctly you are saying if I point my gateway to
> > > HDFS "everything" including "documents", "data", "indexes", "terms" gets
> > > stored in HDFS?
> > >
> > > There is nothing that would get stored locally? Is that correct?
> > > On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> > > [andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
> > >
> > > > Mo,
> > > >
> > > > There are many types of Gateway, I think the one you are interested in
> > > > is the LOCAL gateway only.
> > > > Because each Gateway has different purpose, settings and behavior, I
> > > > just want to be clear that I am only describing LOCAL gateway here.
> > > >
> > > > For local gateway, your ES indices and metadata are all stored to
> > > > whatever directory you set your data path to. Most of the time it is
> > > > probably:
> > > >
> > > > \/data
> > > >
> > > > As for the work path, I am not sure what that is for, probably for
> > > > temporary working area (scratchpad?), I have not seen it being filled in
> > > > with any there lately (after version 0.18).
> > > >
> > > > Actually on some of my installs 0.18+ we do not even have work path
> > > > anymore... so perhaps it is no longer needed? Maybe someone can confirm on
> > > > this?
> > > >
> > > > Thanks
> > > >
> > > > --Andrew
> > > >
> > > > On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
> > > >
> > > > > Actually I was referring to "data" and "work" paths. What are these
> > > > > for? I am still confused what exactly gets stored locally. I also read on
> > > > > the page that "gateway" is used for backups. If that's the case then
> > > > > everything is also getting stored locally?
> > > > >
> > > > > path:
> > > > > home: /usr/local/share/elasticsearch
> > > > > conf: /etc/elasticsearch
> > > > > logs: /var/log/elasticsearch
> > > > >
> > > > > # data: /mnt/elasticsearch/data
> > > > >
> > > > > # work: /mnt/elasticsearch/work
> > > > >
> > > > > On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> > > > > [andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
> > > > >
> > > > > > Yes, it is call Gateway Local (basically your local filesystem)
> > > > > >
> > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/modules/)\*\*
> > > > > > gateway/local.html[http://www.elasticsearch.org/guide/reference/modules/gateway/local.html](http://www.elasticsearch.org/guide/reference/modules/gateway/local.html)
> > > > > >
> > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > >
> > > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > > need to configure workdir.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > >
> > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > metadata.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > >
> > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > if it stores
> > > > > > > > > all the data in HDFS.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > >
> > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > storage so
> > > > > > > > > > > that
> > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > > it for
> > > > > > > > > > > > what exactly ?
> > > > > > > > > > > >
> > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > know how
> > > > > > > > > > > > > it's
> > > > > > > > > > > > > used
> > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > > >
> > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > >
> > > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > > need to configure workdir.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > >
> > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > metadata.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > >
> > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > if it stores
> > > > > > > > > all the data in HDFS.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > >
> > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > storage so
> > > > > > > > > > > that
> > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > > it for
> > > > > > > > > > > > what exactly ?
> > > > > > > > > > > >
> > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > know how
> > > > > > > > > > > > > it's
> > > > > > > > > > > > > used
> > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > > >
> > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > >
> > > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > > need to configure workdir.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > >
> > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > metadata.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > >
> > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > if it stores
> > > > > > > > > all the data in HDFS.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > >
> > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > storage so
> > > > > > > > > > > that
> > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > > it for
> > > > > > > > > > > > what exactly ?
> > > > > > > > > > > >
> > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > know how
> > > > > > > > > > > > > it's
> > > > > > > > > > > > > used
> > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > > >
> > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > >
> > > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > > need to configure workdir.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > >
> > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > metadata.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > >
> > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > if it stores
> > > > > > > > > all the data in HDFS.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > >
> > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > storage so
> > > > > > > > > > > that
> > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > > it for
> > > > > > > > > > > > what exactly ?
> > > > > > > > > > > >
> > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > know how
> > > > > > > > > > > > > it's
> > > > > > > > > > > > > used
> > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > > >
> > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > >
> > > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > > need to configure workdir.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > >
> > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > metadata.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > >
> > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > if it stores
> > > > > > > > > all the data in HDFS.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > >
> > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > storage so
> > > > > > > > > > > that
> > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > > it for
> > > > > > > > > > > > what exactly ?
> > > > > > > > > > > >
> > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > know how
> > > > > > > > > > > > > it's
> > > > > > > > > > > > > used
> > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
---
### Author:  [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)
#### Post date: [May 16, 2012, 10:17pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/15 "2012-05-16T22:17:54Z")
On Wed, May 16, 2012 at 12:48 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> On Tue, May 15, 2012 at 11:39 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote:
>
> > thanks, how often does it get snapshot?
>
> By default every 10 seconds, though snapshot can take a long time
> potentially. Note though, during recovery, it will always recover from the
> last snapshot. Thats why the local gateway is preferable (and what is
> missing is a "backup" API to backup the data to HDFS, for example).
>
> > Also, how often does elasticsearch refresh data to disk from memory?
>
> What do you mean? How often it gets "refreshed" for new data indexed to
> become searchable?
Is data writen to disk immediately as soon as it's writen?
> > On Tue, May 15, 2012 at 1:00 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> >
> > > No, the HDFS gateway will still mean that data will be stored locally
> > > (all the indexes) on each node, and they will be snapshotted to HDFS. Using
> > > HDFS as the main store is pointeless performance wise.
> > >
> > > On Fri, May 11, 2012 at 7:53 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote:
> > >
> > > > So if I understand correctly you are saying if I point my gateway to
> > > > HDFS "everything" including "documents", "data", "indexes", "terms" gets
> > > > stored in HDFS?
> > > >
> > > > There is nothing that would get stored locally? Is that correct?
> > > > On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> > > > [andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
> > > >
> > > > > Mo,
> > > > >
> > > > > There are many types of Gateway, I think the one you are interested in
> > > > > is the LOCAL gateway only.
> > > > > Because each Gateway has different purpose, settings and behavior, I
> > > > > just want to be clear that I am only describing LOCAL gateway here.
> > > > >
> > > > > For local gateway, your ES indices and metadata are all stored to
> > > > > whatever directory you set your data path to. Most of the time it is
> > > > > probably:
> > > > >
> > > > > \/data
> > > > >
> > > > > As for the work path, I am not sure what that is for, probably for
> > > > > temporary working area (scratchpad?), I have not seen it being filled in
> > > > > with any there lately (after version 0.18).
> > > > >
> > > > > Actually on some of my installs 0.18+ we do not even have work path
> > > > > anymore... so perhaps it is no longer needed? Maybe someone can confirm on
> > > > > this?
> > > > >
> > > > > Thanks
> > > > >
> > > > > --Andrew
> > > > >
> > > > > On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
> > > > >
> > > > > > Actually I was referring to "data" and "work" paths. What are these
> > > > > > for? I am still confused what exactly gets stored locally. I also read on
> > > > > > the page that "gateway" is used for backups. If that's the case then
> > > > > > everything is also getting stored locally?
> > > > > >
> > > > > > path:
> > > > > > home: /usr/local/share/elasticsearch
> > > > > > conf: /etc/elasticsearch
> > > > > > logs: /var/log/elasticsearch
> > > > > >
> > > > > > # data: /mnt/elasticsearch/data
> > > > > >
> > > > > > # work: /mnt/elasticsearch/work
> > > > > >
> > > > > > On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> > > > > > [andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
> > > > > >
> > > > > > > Yes, it is call Gateway Local (basically your local filesystem)
> > > > > > >
> > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/modules/)\*\*
> > > > > > > gateway/local.html[http://www.elasticsearch.org/guide/reference/modules/gateway/local.html](http://www.elasticsearch.org/guide/reference/modules/gateway/local.html)
> > > > > > >
> > > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > > >
> > > > > > > > Is there something that elasticsearch stores locally? I see we
> > > > > > > > still need to configure workdir.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > > >
> > > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > > metadata.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > > >
> > > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > > if it stores
> > > > > > > > > > all the data in HDFS.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > > >
> > > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > > storage so
> > > > > > > > > > > > that
> > > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you
> > > > > > > > > > > > > need it for
> > > > > > > > > > > > > what exactly ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > > know how
> > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > used
> > > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > > > >
> > > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > > >
> > > > > > > > Is there something that elasticsearch stores locally? I see we
> > > > > > > > still need to configure workdir.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > > >
> > > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > > metadata.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > > >
> > > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > > if it stores
> > > > > > > > > > all the data in HDFS.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > > >
> > > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > > storage so
> > > > > > > > > > > > that
> > > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you
> > > > > > > > > > > > > need it for
> > > > > > > > > > > > > n \>\> what exactly ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > > know how
> > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > used
> > > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > > > >
> > > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > > >
> > > > > > > > Is there something that elasticsearch stores locally? I see we
> > > > > > > > still need to configure workdir.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > > >
> > > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > > metadata.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > > >
> > > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > > if it stores
> > > > > > > > > > all the data in HDFS.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > > >
> > > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > > storage so
> > > > > > > > > > > > that
> > > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you
> > > > > > > > > > > > > need it for
> > > > > > > > > > > > > what exactly ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > > know how
> > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > used
> > > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > > > >
> > > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > > >
> > > > > > > > Is there something that elasticsearch stores locally? I see we
> > > > > > > > still need to configure workdir.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > > >
> > > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > > metadata.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > > >
> > > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > > if it stores
> > > > > > > > > > all the data in HDFS.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > > >
> > > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > > storage so
> > > > > > > > > > > > that
> > > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you
> > > > > > > > > > > > > need it for
> > > > > > > > > > > > > what exactly ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > > know how
> > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > used
> > > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > > > >
> > > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > > >
> > > > > > > > Is there something that elasticsearch stores locally? I see we
> > > > > > > > still need to configure workdir.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > > >
> > > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > > metadata.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > > >
> > > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > > if it stores
> > > > > > > > > > all the data in HDFS.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > > >
> > > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > > storage so
> > > > > > > > > > > > that
> > > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you
> > > > > > > > > > > > > need it for
> > > > > > > > > > > > > what exactly ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > > know how
> > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > used
> > > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [May 18, 2012, 4:42pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/16 "2012-05-18T16:42:15Z")
Mohit,
Have you looked at the Wonderdog project? I have never used it, but it
might be worthwhile to check out:
> **[GitHub - infochimps-labs/wonderdog: Bulk loading for elastic search](https://github.com/infochimps-labs/wonderdog)**
>
> Bulk loading for elastic search. Contribute to infochimps-labs/wonderdog development by creating an account on GitHub.
--
Ivan
On Wed, May 9, 2012 at 4:18 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote:
> Is anyone using elasticsearch with Hadoop? Would like to know how it's used
> and any suggestions that you can provide would be helpful.
---
### Author:  [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)
#### Post date: [May 18, 2012, 5:09pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/17 "2012-05-18T17:09:27Z")
Yes I just built the source and went through one round of testing. I think
it needs some work to make it more robust, I am currently looking at the
source.
On Fri, May 18, 2012 at 9:42 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> Mohit,
>
> Have you looked at the Wonderdog project? I have never used it, but it
> might be worthwhile to check out:
>
> [GitHub - infochimps-labs/wonderdog: Bulk loading for elastic search](https://github.com/infochimps-labs/wonderdog)
>
> --
> Ivan
>
> On Wed, May 9, 2012 at 4:18 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)
> wrote:
>
> > Is anyone using elasticsearch with Hadoop? Would like to know how it's
> > used
> > and any suggestions that you can provide would be helpful.
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [May 20, 2012, 8:27pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/18 "2012-05-20T20:27:08Z")
Yes, data is written to a transaction log before the operation returns.
On Thu, May 17, 2012 at 12:17 AM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote:
> On Wed, May 16, 2012 at 12:48 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
>
> > On Tue, May 15, 2012 at 11:39 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote:
> >
> > > thanks, how often does it get snapshot?
> >
> > By default every 10 seconds, though snapshot can take a long time
> > potentially. Note though, during recovery, it will always recover from the
> > last snapshot. Thats why the local gateway is preferable (and what is
> > missing is a "backup" API to backup the data to HDFS, for example).
> >
> > > Also, how often does elasticsearch refresh data to disk from memory?
> >
> > What do you mean? How often it gets "refreshed" for new data indexed to
> > become searchable?
>
> Is data writen to disk immediately as soon as it's writen?
>
> > > On Tue, May 15, 2012 at 1:00 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> > >
> > > > No, the HDFS gateway will still mean that data will be stored locally
> > > > (all the indexes) on each node, and they will be snapshotted to HDFS. Using
> > > > HDFS as the main store is pointeless performance wise.
> > > >
> > > > On Fri, May 11, 2012 at 7:53 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote:
> > > >
> > > > > So if I understand correctly you are saying if I point my gateway to
> > > > > HDFS "everything" including "documents", "data", "indexes", "terms" gets
> > > > > stored in HDFS?
> > > > >
> > > > > There is nothing that would get stored locally? Is that correct?
> > > > > On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> > > > > [andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
> > > > >
> > > > > > Mo,
> > > > > >
> > > > > > There are many types of Gateway, I think the one you are interested
> > > > > > in is the LOCAL gateway only.
> > > > > > Because each Gateway has different purpose, settings and behavior, I
> > > > > > just want to be clear that I am only describing LOCAL gateway here.
> > > > > >
> > > > > > For local gateway, your ES indices and metadata are all stored to
> > > > > > whatever directory you set your data path to. Most of the time it is
> > > > > > probably:
> > > > > >
> > > > > > \/data
> > > > > >
> > > > > > As for the work path, I am not sure what that is for, probably for
> > > > > > temporary working area (scratchpad?), I have not seen it being filled in
> > > > > > with any there lately (after version 0.18).
> > > > > >
> > > > > > Actually on some of my installs 0.18+ we do not even have work path
> > > > > > anymore... so perhaps it is no longer needed? Maybe someone can confirm on
> > > > > > this?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > --Andrew
> > > > > >
> > > > > > On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
> > > > > >
> > > > > > > Actually I was referring to "data" and "work" paths. What are these
> > > > > > > for? I am still confused what exactly gets stored locally. I also read on
> > > > > > > the page that "gateway" is used for backups. If that's the case then
> > > > > > > everything is also getting stored locally?
> > > > > > >
> > > > > > > path:
> > > > > > > home: /usr/local/share/elasticsearch
> > > > > > > conf: /etc/elasticsearch
> > > > > > > logs: /var/log/elasticsearch
> > > > > > >
> > > > > > > # data: /mnt/elasticsearch/data
> > > > > > >
> > > > > > > # work: /mnt/elasticsearch/work
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> > > > > > > [andrew@datafeedfile.com](mailto:andrew@datafeedfile.com)\> wrote:
> > > > > > >
> > > > > > > > Yes, it is call Gateway Local (basically your local filesystem)
> > > > > > > >
> > > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/modules/)\*\*
> > > > > > > > gateway/local.html[http://www.elasticsearch.org/guide/reference/modules/gateway/local.html](http://www.elasticsearch.org/guide/reference/modules/gateway/local.html)
> > > > > > > >
> > > > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > > > >
> > > > > > > > > Is there something that elasticsearch stores locally? I see we
> > > > > > > > > still need to configure workdir.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > > > >
> > > > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > > > metadata.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > > > >
> > > > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > > > if it stores
> > > > > > > > > > > all the data in HDFS.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > > > >
> > > > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > > > storage so
> > > > > > > > > > > > > that
> > > > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you
> > > > > > > > > > > > > > need it for
> > > > > > > > > > > > > > what exactly ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > > > know how
> > > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > > > > >
> > > > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > > > >
> > > > > > > > > Is there something that elasticsearch stores locally? I see we
> > > > > > > > > still need to configure workdir.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > > > >
> > > > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > > > metadata.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > > > >
> > > > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > > > if it stores
> > > > > > > > > > > all the data in HDFS.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > > > >
> > > > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > > > storage so
> > > > > > > > > > > > > that
> > > > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you
> > > > > > > > > > > > > > need it for
> > > > > > > > > > > > > > n \>\> what exactly ?
> > > > > > > > > >
> > > > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > > > know how
> > > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > > > > >
> > > > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > > > >
> > > > > > > > > Is there something that elasticsearch stores locally? I see we
> > > > > > > > > still need to configure workdir.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > > > >
> > > > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > > > metadata.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > > > >
> > > > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > > > if it stores
> > > > > > > > > > > all the data in HDFS.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > > > >
> > > > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > > > storage so
> > > > > > > > > > > > > that
> > > > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you
> > > > > > > > > > > > > > need it for
> > > > > > > > > > > > > > what exactly ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > > > know how
> > > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > > > > >
> > > > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > > > >
> > > > > > > > > Is there something that elasticsearch stores locally? I see we
> > > > > > > > > still need to configure workdir.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > > > >
> > > > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > > > metadata.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > > > >
> > > > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > > > if it stores
> > > > > > > > > > > all the data in HDFS.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > > > >
> > > > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > > > storage so
> > > > > > > > > > > > > that
> > > > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you
> > > > > > > > > > > > > > need it for
> > > > > > > > > > > > > > what exactly ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > > > know how
> > > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > > > > >
> > > > > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > > > > >
> > > > > > > > > Is there something that elasticsearch stores locally? I see we
> > > > > > > > > still need to configure workdir.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)wrote:
> > > > > > > > >
> > > > > > > > > > HDFS will be the gateway for ES, storing both the index and
> > > > > > > > > > metadata.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<
> > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\> wrote:
> > > > > > > > > >
> > > > > > > > > > > Looks like that plugin is to use it for backups? It's not clear
> > > > > > > > > > > if it stores
> > > > > > > > > > > all the data in HDFS.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > > > > >
> > > > > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > > > > [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)\>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > > > > storage so
> > > > > > > > > > > > > that
> > > > > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<
> > > > > > > > > > > > > [pivertjerome@gmail.com](mailto:pivertjerome@gmail.com)\> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you
> > > > > > > > > > > > > > need it for
> > > > > > > > > > > > > > what exactly ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > > > > know how
> > > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > and any suggestions that you can provide would be helpful.
---
### Author:  [@Subramanian\_Narayana](https://discuss.elastic.co/u/Subramanian_Narayana)
#### Post date: [October 18, 2012, 5:20am UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/19 "2012-10-18T05:20:09Z")
Hi Kimchy,
Does the data get copied from local store to HDFS? Even if i use HDFS
gateway, data will be stored both locally and in HDFS?
What is the level of performance degrade we will encounter if we use HDFS
as the main data store.
Thanks,
Subbu
On Wednesday, May 16, 2012 1:30:12 AM UTC+5:30, kimchy wrote:
> No, the HDFS gateway will still mean that data will be stored locally (all
> the indexes) on each node, and they will be snapshotted to HDFS. Using HDFS
> as the main store is pointeless performance wise.
>
> On Fri, May 11, 2012 at 7:53 PM, Mohit Anchlia \<[mohita...@gmail.com](mailto:mohita...@gmail.com)\
>
> > wrote:
>
> > So if I understand correctly you are saying if I point my gateway to HDFS
> > "everything" including "documents", "data", "indexes", "terms" gets stored
> > in HDFS?
> >
> > There is nothing that would get stored locally? Is that correct?
> > On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> > [and...@datafeedfile.com](mailto:and...@datafeedfile.com) \\> wrote:
> >
> > > Mo,
> > >
> > > There are many types of Gateway, I think the one you are interested in
> > > is the LOCAL gateway only.
> > > Because each Gateway has different purpose, settings and behavior, I
> > > just want to be clear that I am only describing LOCAL gateway here.
> > >
> > > For local gateway, your ES indices and metadata are all stored to
> > > whatever directory you set your data path to. Most of the time it is
> > > probably:
> > >
> > > \/data
> > >
> > > As for the work path, I am not sure what that is for, probably for
> > > temporary working area (scratchpad?), I have not seen it being filled in
> > > with any there lately (after version 0.18).
> > >
> > > Actually on some of my installs 0.18+ we do not even have work path
> > > anymore... so perhaps it is no longer needed? Maybe someone can confirm on
> > > this?
> > >
> > > Thanks
> > >
> > > --Andrew
> > >
> > > On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
> > >
> > > > Actually I was referring to "data" and "work" paths. What are these
> > > > for? I am still confused what exactly gets stored locally. I also read on
> > > > the page that "gateway" is used for backups. If that's the case then
> > > > everything is also getting stored locally?
> > > >
> > > > path:
> > > > home: /usr/local/share/elasticsearch
> > > > conf: /etc/elasticsearch
> > > > logs: /var/log/elasticsearch
> > > >
> > > > # data: /mnt/elasticsearch/data
> > > >
> > > > # work: /mnt/elasticsearch/work
> > > >
> > > > On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.][DataFeedFile.com](http://DataFeedFile.com) \<
> > > > [and...@datafeedfile.com](mailto:and...@datafeedfile.com) \\> wrote:
> > > >
> > > > > Yes, it is call Gateway Local (basically your local filesystem)
> > > > >
> > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/modules/)\*\*
> > > > > gateway/local.html[http://www.elasticsearch.org/guide/reference/modules/gateway/local.html](http://www.elasticsearch.org/guide/reference/modules/gateway/local.html)
> > > > >
> > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > >
> > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > need to configure workdir.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com)\
> > > > > >
> > > > > > > wrote:
> > > > > >
> > > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<[mohita...@gmail.com](mailto:mohita...@gmail.com)\\>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > > it stores
> > > > > > > > all the data in HDFS.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com)\\>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > >
> > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > [mohita...@gmail.com](mailto:mohita...@gmail.com) \\>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > storage so
> > > > > > > > > > that
> > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<[pivert...@gmail.com](mailto:pivert...@gmail.com)\\>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > it for
> > > > > > > > > > > what exactly ?
> > > > > > > > > > >
> > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > know how
> > > > > > > > > > > > it's
> > > > > > > > > > > > used
> > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > >
> > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > >
> > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > need to configure workdir.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com)\
> > > > > >
> > > > > > > wrote:
> > > > > >
> > > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<[mohita...@gmail.com](mailto:mohita...@gmail.com)\\>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > > it stores
> > > > > > > > all the data in HDFS.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com)\\>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > >
> > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > [mohita...@gmail.com](mailto:mohita...@gmail.com) \\>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > storage so
> > > > > > > > > > that
> > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<[pivert...@gmail.com](mailto:pivert...@gmail.com)\\>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > it for
> > > > > > > > > > > what exactly ?
> > > > > > > > > > >
> > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > know how
> > > > > > > > > > > > it's
> > > > > > > > > > > > used
> > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > >
> > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > >
> > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > need to configure workdir.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com)\
> > > > > >
> > > > > > > wrote:
> > > > > >
> > > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<[mohita...@gmail.com](mailto:mohita...@gmail.com)\\>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > > it stores
> > > > > > > > all the data in HDFS.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com)\\>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > >
> > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > [mohita...@gmail.com](mailto:mohita...@gmail.com) \\>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > storage so
> > > > > > > > > > that
> > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<[pivert...@gmail.com](mailto:pivert...@gmail.com)\\>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > it for
> > > > > > > > > > > what exactly ?
> > > > > > > > > > >
> > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > know how
> > > > > > > > > > > > it's
> > > > > > > > > > > > used
> > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > >
> > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > >
> > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > need to configure workdir.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com)\
> > > > > >
> > > > > > > wrote:
> > > > > >
> > > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<[mohita...@gmail.com](mailto:mohita...@gmail.com)\\>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > > it stores
> > > > > > > > all the data in HDFS.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com)\\>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > >
> > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > [mohita...@gmail.com](mailto:mohita...@gmail.com) \\>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > storage so
> > > > > > > > > > that
> > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<[pivert...@gmail.com](mailto:pivert...@gmail.com)\\>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > it for
> > > > > > > > > > > what exactly ?
> > > > > > > > > > >
> > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > know how
> > > > > > > > > > > > it's
> > > > > > > > > > > > used
> > > > > > > > > > > > and any suggestions that you can provide would be helpful.
> > > > >
> > > > > On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
> > > > >
> > > > > > Is there something that elasticsearch stores locally? I see we still
> > > > > > need to configure workdir.
> > > > > >
> > > > > > On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com)\
> > > > > >
> > > > > > > wrote:
> > > > > >
> > > > > > > HDFS will be the gateway for ES, storing both the index and metadata.
> > > > > > >
> > > > > > > On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia \<[mohita...@gmail.com](mailto:mohita...@gmail.com)\\>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Looks like that plugin is to use it for backups? It's not clear if
> > > > > > > > it stores
> > > > > > > > all the data in HDFS.
> > > > > > > >
> > > > > > > > On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com)\\>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have never used it, but have you looked at the Hadoop plugin?
> > > > > > > > >
> > > > > > > > > [https://github.com/\*\*elasticsear\*\*ch/elasticsearch-\*\*hadoop](https://github.com/ **elasticsear** ch/elasticsearch-**hadoop)[https://github.com/elasticsearch/elasticsearch-hadoop](https://github.com/elasticsearch/elasticsearch-hadoop)
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia \<
> > > > > > > > > [mohita...@gmail.com](mailto:mohita...@gmail.com) \\>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We have a hadoop cluster and I would like to use HDFS as the
> > > > > > > > > > storage so
> > > > > > > > > > that
> > > > > > > > > > I have one place where everything is stored and tracked.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 10, 2012 at 1:02 AM, Jérome \<[pivert...@gmail.com](mailto:pivert...@gmail.com)\\>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Aha, i follow this topic, have to do the same thing, you need
> > > > > > > > > > > it for
> > > > > > > > > > > what exactly ?
> > > > > > > > > > >
> > > > > > > > > > > On 10 mai, 01:18, Mohit Anchlia [mohitanch...@gmail.com](mailto:mohitanch...@gmail.com)
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Is anyone using elasticsearch with Hadoop? Would like to
> > > > > > > > > > > > know how
> > > > > > > > > > > > it's
> > > > > > > > > > > > used
> > > > > > > > > > > > and any suggestions that you can provide would be helpful.
--
---
### Author:  [@Phani](https://discuss.elastic.co/u/Phani)
#### Post date: [August 14, 2013, 11:09pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623/20 "2013-08-14T23:09:42Z")
Hi all,
I see that in the latest releases, it is mentioned that the hadoop
gateway is deprecated and will be removed. Does that mean that, there won't
be any support for storing the indexes on HDFS ?
And, late in the thread it is mentioned that indexes are only snapshotted
to HDFS - so indexes are first built on local FS and then copied to HDFS
every 10 seconds or so ?
It is also mentioned that using HDFS directly as a data storage for indexes
is not good performance-wise, can one give more insights into the
performance issues and how degrading it will be compared to local FS ?
On Wednesday, May 9, 2012 4:18:58 PM UTC-7, Mo wrote:
> Is anyone using elasticsearch with Hadoop? Would like to know how it's
> used and any suggestions that you can provide would be helpful.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
[Next page](https://discuss.elastic.co/t/elasticsearch-with-hadoop/7623.md?page=2)