Is anyone using elasticsearch with Hadoop? Would like to know how it's used
and any suggestions that you can provide would be helpful.
Aha, i follow this topic, have to do the same thing, you need it for
what exactly ?
On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how it's used
and any suggestions that you can provide would be helpful.
We have a hadoop cluster and I would like to use HDFS as the storage so
that I have one place where everything is stored and tracked.
On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com wrote:
Aha, i follow this topic, have to do the same thing, you need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how it's
used
and any suggestions that you can provide would be helpful.
I have never used it, but have you looked at the Hadoop plugin?
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia mohitanchlia@gmail.com wrote:
We have a hadoop cluster and I would like to use HDFS as the storage so that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com wrote:
Aha, i follow this topic, have to do the same thing, you need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how it's
used
and any suggestions that you can provide would be helpful.
Looks like that plugin is to use it for backups? It's not clear if it
stores all the data in HDFS.
On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia mohitanchlia@gmail.com
wrote:We have a hadoop cluster and I would like to use HDFS as the storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com wrote:
Aha, i follow this topic, have to do the same thing, you need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how it's
used
and any suggestions that you can provide would be helpful.
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com wrote:
Looks like that plugin is to use it for backups? It's not clear if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia mohitanchlia@gmail.com
wrote:We have a hadoop cluster and I would like to use HDFS as the storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com wrote:
Aha, i follow this topic, have to do the same thing, you need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how
it's
used
and any suggestions that you can provide would be helpful.
Is there something that elasticsearch stores locally? I see we still need
to configure workdir.
On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia mohitanchlia@gmail.com
wrote:We have a hadoop cluster and I would like to use HDFS as the storage
so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how
it's
used
and any suggestions that you can provide would be helpful.
Yes, it is call Gateway Local (basically your local filesystem)
On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still need
to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <mohitanchlia@gmail.com
wrote:
We have a hadoop cluster and I would like to use HDFS as the storage
so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how
it's
used
and any suggestions that you can provide would be helpful.
On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still need
to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <mohitanchlia@gmail.com
wrote:
We have a hadoop cluster and I would like to use HDFS as the storage
so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how
it's
used
and any suggestions that you can provide would be helpful.
On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still need
to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <mohitanchlia@gmail.com
wrote:
We have a hadoop cluster and I would like to use HDFS as the storage
so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how
it's
used
and any suggestions that you can provide would be helpful.
On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still need
to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <mohitanchlia@gmail.com
wrote:
We have a hadoop cluster and I would like to use HDFS as the storage
so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how
it's
used
and any suggestions that you can provide would be helpful.
On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still need
to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <mohitanchlia@gmail.com
wrote:
We have a hadoop cluster and I would like to use HDFS as the storage
so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how
it's
used
and any suggestions that you can provide would be helpful.
Actually I was referring to "data" and "work" paths. What are these for? I
am still confused what exactly gets stored locally. I also read on the page
that "gateway" is used for backups. If that's the case then everything is
also getting stored locally?
path:
home: /usr/local/share/elasticsearch
conf: /etc/elasticsearch
logs: /var/log/elasticsearch
data: /mnt/elasticsearch/data
work: /mnt/elasticsearch/work
On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:
Yes, it is call Gateway Local (basically your local filesystem)
Elasticsearch Platform — Find real-time answers at scale | Elastic
On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still need
to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsearch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still need
to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsearch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still need
to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsearch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still need
to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsearch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still need
to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsearch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how
it's
used
and any suggestions that you can provide would be helpful.
Mo,
There are many types of Gateway, I think the one you are interested in is
the LOCAL gateway only.
Because each Gateway has different purpose, settings and behavior, I just
want to be clear that I am only describing LOCAL gateway here.
For local gateway, your ES indices and metadata are all stored to whatever
directory you set your data path to. Most of the time it is probably:
<ES_HOME>/data
As for the work path, I am not sure what that is for, probably for
temporary working area (scratchpad?), I have not seen it being filled in
with any there lately (after version 0.18).
Actually on some of my installs 0.18+ we do not even have work path
anymore... so perhaps it is no longer needed? Maybe someone can confirm on
this?
Thanks
--Andrew
On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
Actually I was referring to "data" and "work" paths. What are these for? I
am still confused what exactly gets stored locally. I also read on the page
that "gateway" is used for backups. If that's the case then everything is
also getting stored locally?path:
home: /usr/local/share/elasticsearch
conf: /etc/elasticsearch
logs: /var/log/elasticsearchdata: /mnt/elasticsearch/data
work: /mnt/elasticsearch/work
On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:Yes, it is call Gateway Local (basically your local filesystem)
Elasticsearch Platform — Find real-time answers at scale | Elastic
On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsearch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsearch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsearch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsearch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if it
stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com wrote:
I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsearch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.
So if I understand correctly you are saying if I point my gateway to HDFS
"everything" including "documents", "data", "indexes", "terms" gets stored
in HDFS?
There is nothing that would get stored locally? Is that correct?
On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:
Mo,
There are many types of Gateway, I think the one you are interested in is
the LOCAL gateway only.
Because each Gateway has different purpose, settings and behavior, I just
want to be clear that I am only describing LOCAL gateway here.For local gateway, your ES indices and metadata are all stored to whatever
directory you set your data path to. Most of the time it is probably:<ES_HOME>/data
As for the work path, I am not sure what that is for, probably for
temporary working area (scratchpad?), I have not seen it being filled in
with any there lately (after version 0.18).Actually on some of my installs 0.18+ we do not even have work path
anymore... so perhaps it is no longer needed? Maybe someone can confirm on
this?Thanks
--Andrew
On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
Actually I was referring to "data" and "work" paths. What are these for?
I am still confused what exactly gets stored locally. I also read on the
page that "gateway" is used for backups. If that's the case then everything
is also getting stored locally?path:
home: /usr/local/share/elasticsearch
conf: /etc/elasticsearch
logs: /var/log/elasticsearchdata: /mnt/elasticsearch/data
work: /mnt/elasticsearch/work
On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:Yes, it is call Gateway Local (basically your local filesystem)
Elasticsearch Platform — Find real-time answers at scale | Elastic**
gateway/local.htmlhttp://www.elasticsearch.org/guide/reference/modules/gateway/local.htmlOn Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need it
for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.
No, the HDFS gateway will still mean that data will be stored locally (all
the indexes) on each node, and they will be snapshotted to HDFS. Using HDFS
as the main store is pointeless performance wise.
On Fri, May 11, 2012 at 7:53 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:
So if I understand correctly you are saying if I point my gateway to HDFS
"everything" including "documents", "data", "indexes", "terms" gets stored
in HDFS?There is nothing that would get stored locally? Is that correct?
On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:Mo,
There are many types of Gateway, I think the one you are interested in is
the LOCAL gateway only.
Because each Gateway has different purpose, settings and behavior, I just
want to be clear that I am only describing LOCAL gateway here.For local gateway, your ES indices and metadata are all stored to
whatever directory you set your data path to. Most of the time it is
probably:<ES_HOME>/data
As for the work path, I am not sure what that is for, probably for
temporary working area (scratchpad?), I have not seen it being filled in
with any there lately (after version 0.18).Actually on some of my installs 0.18+ we do not even have work path
anymore... so perhaps it is no longer needed? Maybe someone can confirm on
this?Thanks
--Andrew
On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
Actually I was referring to "data" and "work" paths. What are these for?
I am still confused what exactly gets stored locally. I also read on the
page that "gateway" is used for backups. If that's the case then everything
is also getting stored locally?path:
home: /usr/local/share/elasticsearch
conf: /etc/elasticsearch
logs: /var/log/elasticsearchdata: /mnt/elasticsearch/data
work: /mnt/elasticsearch/work
On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:Yes, it is call Gateway Local (basically your local filesystem)
Elasticsearch Platform — Find real-time answers at scale | Elastic**
gateway/local.htmlhttp://www.elasticsearch.org/guide/reference/modules/gateway/local.htmlOn Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know
how
it's
used
and any suggestions that you can provide would be helpful.
thanks, how often does it get snapshot?
Also, how often does elasticsearch refresh data to disk from memory?
On Tue, May 15, 2012 at 1:00 PM, Shay Banon kimchy@gmail.com wrote:
No, the HDFS gateway will still mean that data will be stored locally (all
the indexes) on each node, and they will be snapshotted to HDFS. Using HDFS
as the main store is pointeless performance wise.On Fri, May 11, 2012 at 7:53 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:
So if I understand correctly you are saying if I point my gateway to HDFS
"everything" including "documents", "data", "indexes", "terms" gets stored
in HDFS?There is nothing that would get stored locally? Is that correct?
On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:Mo,
There are many types of Gateway, I think the one you are interested in
is the LOCAL gateway only.
Because each Gateway has different purpose, settings and behavior, I
just want to be clear that I am only describing LOCAL gateway here.For local gateway, your ES indices and metadata are all stored to
whatever directory you set your data path to. Most of the time it is
probably:<ES_HOME>/data
As for the work path, I am not sure what that is for, probably for
temporary working area (scratchpad?), I have not seen it being filled in
with any there lately (after version 0.18).Actually on some of my installs 0.18+ we do not even have work path
anymore... so perhaps it is no longer needed? Maybe someone can confirm on
this?Thanks
--Andrew
On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
Actually I was referring to "data" and "work" paths. What are these
for? I am still confused what exactly gets stored locally. I also read on
the page that "gateway" is used for backups. If that's the case then
everything is also getting stored locally?path:
home: /usr/local/share/elasticsearch
conf: /etc/elasticsearch
logs: /var/log/elasticsearchdata: /mnt/elasticsearch/data
work: /mnt/elasticsearch/work
On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:Yes, it is call Gateway Local (basically your local filesystem)
Elasticsearch Platform — Find real-time answers at scale | Elastic**
gateway/local.htmlhttp://www.elasticsearch.org/guide/reference/modules/gateway/local.htmlOn Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.com wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome pivertjerome@gmail.com
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.
On Tue, May 15, 2012 at 11:39 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:
thanks, how often does it get snapshot?
By default every 10 seconds, though snapshot can take a long time
potentially. Note though, during recovery, it will always recover from the
last snapshot. Thats why the local gateway is preferable (and what is
missing is a "backup" API to backup the data to HDFS, for example).
Also, how often does elasticsearch refresh data to disk from memory?
What do you mean? How often it gets "refreshed" for new data indexed to
become searchable?
On Tue, May 15, 2012 at 1:00 PM, Shay Banon kimchy@gmail.com wrote:
No, the HDFS gateway will still mean that data will be stored locally
(all the indexes) on each node, and they will be snapshotted to HDFS. Using
HDFS as the main store is pointeless performance wise.On Fri, May 11, 2012 at 7:53 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:
So if I understand correctly you are saying if I point my gateway to
HDFS "everything" including "documents", "data", "indexes", "terms" gets
stored in HDFS?There is nothing that would get stored locally? Is that correct?
On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:Mo,
There are many types of Gateway, I think the one you are interested in
is the LOCAL gateway only.
Because each Gateway has different purpose, settings and behavior, I
just want to be clear that I am only describing LOCAL gateway here.For local gateway, your ES indices and metadata are all stored to
whatever directory you set your data path to. Most of the time it is
probably:<ES_HOME>/data
As for the work path, I am not sure what that is for, probably for
temporary working area (scratchpad?), I have not seen it being filled in
with any there lately (after version 0.18).Actually on some of my installs 0.18+ we do not even have work path
anymore... so perhaps it is no longer needed? Maybe someone can confirm on
this?Thanks
--Andrew
On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
Actually I was referring to "data" and "work" paths. What are these
for? I am still confused what exactly gets stored locally. I also read on
the page that "gateway" is used for backups. If that's the case then
everything is also getting stored locally?path:
home: /usr/local/share/elasticsearch
conf: /etc/elasticsearch
logs: /var/log/elasticsearchdata: /mnt/elasticsearch/data
work: /mnt/elasticsearch/work
On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:Yes, it is call Gateway Local (basically your local filesystem)
Elasticsearch Platform — Find real-time answers at scale | Elastic**
gateway/local.htmlhttp://www.elasticsearch.org/guide/reference/modules/gateway/local.htmlOn Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.
On Wed, May 16, 2012 at 12:48 PM, Shay Banon kimchy@gmail.com wrote:
On Tue, May 15, 2012 at 11:39 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:
thanks, how often does it get snapshot?
By default every 10 seconds, though snapshot can take a long time
potentially. Note though, during recovery, it will always recover from the
last snapshot. Thats why the local gateway is preferable (and what is
missing is a "backup" API to backup the data to HDFS, for example).Also, how often does elasticsearch refresh data to disk from memory?
What do you mean? How often it gets "refreshed" for new data indexed to
become searchable?
Is data writen to disk immediately as soon as it's writen?
On Tue, May 15, 2012 at 1:00 PM, Shay Banon kimchy@gmail.com wrote:
No, the HDFS gateway will still mean that data will be stored locally
(all the indexes) on each node, and they will be snapshotted to HDFS. Using
HDFS as the main store is pointeless performance wise.On Fri, May 11, 2012 at 7:53 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:
So if I understand correctly you are saying if I point my gateway to
HDFS "everything" including "documents", "data", "indexes", "terms" gets
stored in HDFS?There is nothing that would get stored locally? Is that correct?
On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:Mo,
There are many types of Gateway, I think the one you are interested in
is the LOCAL gateway only.
Because each Gateway has different purpose, settings and behavior, I
just want to be clear that I am only describing LOCAL gateway here.For local gateway, your ES indices and metadata are all stored to
whatever directory you set your data path to. Most of the time it is
probably:<ES_HOME>/data
As for the work path, I am not sure what that is for, probably for
temporary working area (scratchpad?), I have not seen it being filled in
with any there lately (after version 0.18).Actually on some of my installs 0.18+ we do not even have work path
anymore... so perhaps it is no longer needed? Maybe someone can confirm on
this?Thanks
--Andrew
On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
Actually I was referring to "data" and "work" paths. What are these
for? I am still confused what exactly gets stored locally. I also read on
the page that "gateway" is used for backups. If that's the case then
everything is also getting stored locally?path:
home: /usr/local/share/elasticsearch
conf: /etc/elasticsearch
logs: /var/log/elasticsearchdata: /mnt/elasticsearch/data
work: /mnt/elasticsearch/work
On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:Yes, it is call Gateway Local (basically your local filesystem)
Elasticsearch Platform — Find real-time answers at scale | Elastic**
gateway/local.htmlhttp://www.elasticsearch.org/guide/reference/modules/gateway/local.htmlOn Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we
still need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you
need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we
still need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you
need it for
n >> what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we
still need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you
need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we
still need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you
need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we
still need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you
need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.
Mohit,
Have you looked at the Wonderdog project? I have never used it, but it
might be worthwhile to check out:
--
Ivan
On Wed, May 9, 2012 at 4:18 PM, Mohit Anchlia mohitanchlia@gmail.com wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how it's used
and any suggestions that you can provide would be helpful.
Yes I just built the source and went through one round of testing. I think
it needs some work to make it more robust, I am currently looking at the
source.
On Fri, May 18, 2012 at 9:42 AM, Ivan Brusic ivan@brusic.com wrote:
Mohit,
Have you looked at the Wonderdog project? I have never used it, but it
might be worthwhile to check out:GitHub - infochimps-labs/wonderdog: Bulk loading for elastic search
--
IvanOn Wed, May 9, 2012 at 4:18 PM, Mohit Anchlia mohitanchlia@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to know how it's
used
and any suggestions that you can provide would be helpful.
Yes, data is written to a transaction log before the operation returns.
On Thu, May 17, 2012 at 12:17 AM, Mohit Anchlia mohitanchlia@gmail.comwrote:
On Wed, May 16, 2012 at 12:48 PM, Shay Banon kimchy@gmail.com wrote:
On Tue, May 15, 2012 at 11:39 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:
thanks, how often does it get snapshot?
By default every 10 seconds, though snapshot can take a long time
potentially. Note though, during recovery, it will always recover from the
last snapshot. Thats why the local gateway is preferable (and what is
missing is a "backup" API to backup the data to HDFS, for example).Also, how often does elasticsearch refresh data to disk from memory?
What do you mean? How often it gets "refreshed" for new data indexed to
become searchable?Is data writen to disk immediately as soon as it's writen?
On Tue, May 15, 2012 at 1:00 PM, Shay Banon kimchy@gmail.com wrote:
No, the HDFS gateway will still mean that data will be stored locally
(all the indexes) on each node, and they will be snapshotted to HDFS. Using
HDFS as the main store is pointeless performance wise.On Fri, May 11, 2012 at 7:53 PM, Mohit Anchlia mohitanchlia@gmail.comwrote:
So if I understand correctly you are saying if I point my gateway to
HDFS "everything" including "documents", "data", "indexes", "terms" gets
stored in HDFS?There is nothing that would get stored locally? Is that correct?
On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:Mo,
There are many types of Gateway, I think the one you are interested
in is the LOCAL gateway only.
Because each Gateway has different purpose, settings and behavior, I
just want to be clear that I am only describing LOCAL gateway here.For local gateway, your ES indices and metadata are all stored to
whatever directory you set your data path to. Most of the time it is
probably:<ES_HOME>/data
As for the work path, I am not sure what that is for, probably for
temporary working area (scratchpad?), I have not seen it being filled in
with any there lately (after version 0.18).Actually on some of my installs 0.18+ we do not even have work path
anymore... so perhaps it is no longer needed? Maybe someone can confirm on
this?Thanks
--Andrew
On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
Actually I was referring to "data" and "work" paths. What are these
for? I am still confused what exactly gets stored locally. I also read on
the page that "gateway" is used for backups. If that's the case then
everything is also getting stored locally?path:
home: /usr/local/share/elasticsearch
conf: /etc/elasticsearch
logs: /var/log/elasticsearchdata: /mnt/elasticsearch/data
work: /mnt/elasticsearch/work
On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.]DataFeedFile.com <
andrew@datafeedfile.com> wrote:Yes, it is call Gateway Local (basically your local filesystem)
Elasticsearch Platform — Find real-time answers at scale | Elastic**
gateway/local.htmlhttp://www.elasticsearch.org/guide/reference/modules/gateway/local.htmlOn Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we
still need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you
need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we
still need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you
need it for
n >> what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we
still need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you
need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we
still need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you
need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we
still need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic ivan@brusic.comwrote:
HDFS will be the gateway for ES, storing both the index and
metadata.On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <
mohitanchlia@gmail.com> wrote:Looks like that plugin is to use it for backups? It's not clear
if it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic ivan@brusic.com
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohitanchlia@gmail.com>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <
pivertjerome@gmail.com> wrote:Aha, i follow this topic, have to do the same thing, you
need it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.
Hi Kimchy,
Does the data get copied from local store to HDFS? Even if i use HDFS
gateway, data will be stored both locally and in HDFS?
What is the level of performance degrade we will encounter if we use HDFS
as the main data store.
Thanks,
Subbu
On Wednesday, May 16, 2012 1:30:12 AM UTC+5:30, kimchy wrote:
No, the HDFS gateway will still mean that data will be stored locally (all
the indexes) on each node, and they will be snapshotted to HDFS. Using HDFS
as the main store is pointeless performance wise.On Fri, May 11, 2012 at 7:53 PM, Mohit Anchlia <mohita...@gmail.com<javascript:>
wrote:
So if I understand correctly you are saying if I point my gateway to HDFS
"everything" including "documents", "data", "indexes", "terms" gets stored
in HDFS?There is nothing that would get stored locally? Is that correct?
On Fri, May 11, 2012 at 9:36 AM, Andrew[.:at:.]DataFeedFile.com <
and...@datafeedfile.com <javascript:>> wrote:Mo,
There are many types of Gateway, I think the one you are interested in
is the LOCAL gateway only.
Because each Gateway has different purpose, settings and behavior, I
just want to be clear that I am only describing LOCAL gateway here.For local gateway, your ES indices and metadata are all stored to
whatever directory you set your data path to. Most of the time it is
probably:<ES_HOME>/data
As for the work path, I am not sure what that is for, probably for
temporary working area (scratchpad?), I have not seen it being filled in
with any there lately (after version 0.18).Actually on some of my installs 0.18+ we do not even have work path
anymore... so perhaps it is no longer needed? Maybe someone can confirm on
this?Thanks
--Andrew
On Thursday, May 10, 2012 6:48:03 PM UTC-5, Mo wrote:
Actually I was referring to "data" and "work" paths. What are these
for? I am still confused what exactly gets stored locally. I also read on
the page that "gateway" is used for backups. If that's the case then
everything is also getting stored locally?path:
home: /usr/local/share/elasticsearch
conf: /etc/elasticsearch
logs: /var/log/elasticsearchdata: /mnt/elasticsearch/data
work: /mnt/elasticsearch/work
On Thu, May 10, 2012 at 3:40 PM, Andrew[.:at:.]DataFeedFile.com <
and...@datafeedfile.com <javascript:>> wrote:Yes, it is call Gateway Local (basically your local filesystem)
Elasticsearch Platform — Find real-time answers at scale | Elastic**
gateway/local.htmlhttp://www.elasticsearch.org/guide/reference/modules/gateway/local.htmlOn Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic <iv...@brusic.com<javascript:>
wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <mohita...@gmail.com<javascript:>>
wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic <iv...@brusic.com<javascript:>>
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohita...@gmail.com <javascript:>>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <pivert...@gmail.com<javascript:>>
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic <iv...@brusic.com<javascript:>
wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <mohita...@gmail.com<javascript:>>
wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic <iv...@brusic.com<javascript:>>
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohita...@gmail.com <javascript:>>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <pivert...@gmail.com<javascript:>>
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic <iv...@brusic.com<javascript:>
wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <mohita...@gmail.com<javascript:>>
wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic <iv...@brusic.com<javascript:>>
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohita...@gmail.com <javascript:>>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <pivert...@gmail.com<javascript:>>
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic <iv...@brusic.com<javascript:>
wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <mohita...@gmail.com<javascript:>>
wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic <iv...@brusic.com<javascript:>>
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohita...@gmail.com <javascript:>>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <pivert...@gmail.com<javascript:>>
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.On Thursday, May 10, 2012 4:30:36 PM UTC-5, Mo wrote:
Is there something that elasticsearch stores locally? I see we still
need to configure workdir.On Thu, May 10, 2012 at 2:26 PM, Ivan Brusic <iv...@brusic.com<javascript:>
wrote:
HDFS will be the gateway for ES, storing both the index and metadata.
On Thu, May 10, 2012 at 2:20 PM, Mohit Anchlia <mohita...@gmail.com<javascript:>>
wrote:Looks like that plugin is to use it for backups? It's not clear if
it stores
all the data in HDFS.On Thu, May 10, 2012 at 2:08 PM, Ivan Brusic <iv...@brusic.com<javascript:>>
wrote:I have never used it, but have you looked at the Hadoop plugin?
https://github.com/**elasticsear**ch/elasticsearch-**hadoophttps://github.com/elasticsearch/elasticsearch-hadoop
On Thu, May 10, 2012 at 7:21 AM, Mohit Anchlia <
mohita...@gmail.com <javascript:>>
wrote:We have a hadoop cluster and I would like to use HDFS as the
storage so
that
I have one place where everything is stored and tracked.On Thu, May 10, 2012 at 1:02 AM, Jérome <pivert...@gmail.com<javascript:>>
wrote:Aha, i follow this topic, have to do the same thing, you need
it for
what exactly ?On 10 mai, 01:18, Mohit Anchlia mohitanch...@gmail.com
wrote:Is anyone using elasticsearch with Hadoop? Would like to
know how
it's
used
and any suggestions that you can provide would be helpful.
--
Hi all,
I see that in the latest releases, it is mentioned that the hadoop
gateway is deprecated and will be removed. Does that mean that, there won't
be any support for storing the indexes on HDFS ?
And, late in the thread it is mentioned that indexes are only snapshotted
to HDFS - so indexes are first built on local FS and then copied to HDFS
every 10 seconds or so ?
It is also mentioned that using HDFS directly as a data storage for indexes
is not good performance-wise, can one give more insights into the
performance issues and how degrading it will be compared to local FS ?
On Wednesday, May 9, 2012 4:18:58 PM UTC-7, Mo wrote:
Is anyone using elasticsearch with Hadoop? Would like to know how it's
used and any suggestions that you can provide would be helpful.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.