Where is Elasticsearch storing data

I cannot find any info on where Elasticsearch is storing it's data.
I would like to manage this potentially valuable information.
Where, how big, how to back it up, restore it, etc.
Then on to clusters, etc..

Thanks,
John

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

by default in path.data dir.
In zip or tar.gz distribution, it’s in "data" dir

--
David Pilato - Developer | Evangelist

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 27 mai 2015 à 17:25, John Adams jadams007@gmail.com a écrit :

I cannot find any info on where Elasticsearch is storing it's data.
I would like to manage this potentially valuable information.
Where, how big, how to back it up, restore it, etc.
Then on to clusters, etc..

Thanks,
John

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/ https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4836BD52-F0A0-4507-BE1D-14E3CA4B0FD7%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Check out

PS - We're moving to https://discuss.elastic.co/, please join us there for
any future discussions!

On 28 May 2015 at 01:27, David Pilato david@pilato.fr wrote:

by default in path.data dir.
In zip or tar.gz distribution, it’s in "data" dir

--
David Pilato - Developer | Evangelist
elastic.co http://elastic.co
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 27 mai 2015 à 17:25, John Adams jadams007@gmail.com a écrit :

I cannot find any info on where Elasticsearch is storing it's data.
I would like to manage this potentially valuable information.
Where, how big, how to back it up, restore it, etc.
Then on to clusters, etc..

Thanks,
John

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4836BD52-F0A0-4507-BE1D-14E3CA4B0FD7%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/4836BD52-F0A0-4507-BE1D-14E3CA4B0FD7%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9BS-BBEy5W4Q1odAX7hjU%3DspAqOz8q0rTTQUb%2BU4294g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I similarly can't find where the data is stored. We have it running with
ES-yarn though. This article says that the data is only stored in
container storage by default (we haven't changed any settings):

https://www.elastic.co/guide/en/elasticsearch/hadoop/master/ey-setup.html

However, the data is definitely persisting somehow between restarts so it
has to be stored somewhere. We just can't find where. There is no
"path.data" dir and also there's no "/var/lib/elasticsearch/data" dir like
I've seen on other posts. And we can't find it in hdfs either.

So can anyone please point me to where we might be able to find the data?
If it helps, we are loading indexes from both hive and from logstash
directly.

Thanks,
Jim

On Wednesday, May 27, 2015 at 6:56:54 PM UTC-4, Mark Walkom wrote:

Check out
Directory layout | Elasticsearch Guide [8.11] | Elastic

PS - We're moving to https://discuss.elastic.co/, please join us there
for any future discussions!

On 28 May 2015 at 01:27, David Pilato <da...@pilato.fr <javascript:>>
wrote:

by default in path.data dir.
In zip or tar.gz distribution, it’s in "data" dir

--
David Pilato - Developer | Evangelist
elastic.co http://elastic.co
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 27 mai 2015 à 17:25, John Adams <jada...@gmail.com <javascript:>> a
écrit :

I cannot find any info on where Elasticsearch is storing it's data.
I would like to manage this potentially valuable information.
Where, how big, how to back it up, restore it, etc.
Then on to clusters, etc..

Thanks,
John

--
Please update your bookmarks! We have moved to
https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to
https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4836BD52-F0A0-4507-BE1D-14E3CA4B0FD7%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/4836BD52-F0A0-4507-BE1D-14E3CA4B0FD7%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bc0ad06d-21b3-4ca7-aade-876f15b379a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thank you, I will check it out.

On Wednesday, May 27, 2015 at 8:27:23 AM UTC-7, David Pilato wrote:

by default in path.data dir.
In zip or tar.gz distribution, it’s in "data" dir

--
David Pilato - Developer | Evangelist
elastic.co http://elastic.co
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 27 mai 2015 à 17:25, John Adams <jada...@gmail.com <javascript:>> a
écrit :

I cannot find any info on where Elasticsearch is storing it's data.
I would like to manage this potentially valuable information.
Where, how big, how to back it up, restore it, etc.
Then on to clusters, etc..

Thanks,
John

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/81307403-978f-45f2-972f-07595b9df337%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

If you query "_nodes?pretty | grep data" you will see a value, that's where
the data is.

PS - We're moving to https://discuss.elastic.co/, please join us there for
any future discussions!

On 29 May 2015 at 00:31, Jim Malone malonej7@gmail.com wrote:

I similarly can't find where the data is stored. We have it running with
ES-yarn though. This article says that the data is only stored in
container storage by default (we haven't changed any settings):

https://www.elastic.co/guide/en/elasticsearch/hadoop/master/ey-setup.html

However, the data is definitely persisting somehow between restarts so it
has to be stored somewhere. We just can't find where. There is no
"path.data" dir and also there's no "/var/lib/elasticsearch/data" dir
like I've seen on other posts. And we can't find it in hdfs either.

So can anyone please point me to where we might be able to find the data?
If it helps, we are loading indexes from both hive and from logstash
directly.

Thanks,
Jim

On Wednesday, May 27, 2015 at 6:56:54 PM UTC-4, Mark Walkom wrote:

Check out
Directory layout | Elasticsearch Guide [8.11] | Elastic

PS - We're moving to https://discuss.elastic.co/, please join us there
for any future discussions!

On 28 May 2015 at 01:27, David Pilato da...@pilato.fr wrote:

by default in path.data dir.
In zip or tar.gz distribution, it’s in "data" dir

--
David Pilato - Developer | Evangelist
elastic.co http://elastic.co
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 27 mai 2015 à 17:25, John Adams jada...@gmail.com a écrit :

I cannot find any info on where Elasticsearch is storing it's data.
I would like to manage this potentially valuable information.
Where, how big, how to back it up, restore it, etc.
Then on to clusters, etc..

Thanks,
John

--
Please update your bookmarks! We have moved to
https://discuss.elastic.co/

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to
https://discuss.elastic.co/

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4836BD52-F0A0-4507-BE1D-14E3CA4B0FD7%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/4836BD52-F0A0-4507-BE1D-14E3CA4B0FD7%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/


You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bc0ad06d-21b3-4ca7-aade-876f15b379a3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bc0ad06d-21b3-4ca7-aade-876f15b379a3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9pYUtnr%2B4U35t1oNu%3Dwbom4UqZyJEs0iB96CikSOiD7g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Thank you, that little query is very helpful. Although there isn't any
actual "data" property anywhere in the response. I'm guessing it's cause
our data is still just cached, and it appears to be in the "home"
directory. Now we just have to figure out how to set the data dir using
es-yarn so that it's stored on the local nodes permanently. Haven't had
luck finding that yet.

And I will use the new forum going forward, just wanted to continue on this
thread instead of starting a new one.

Jim

On Thursday, May 28, 2015 at 6:10:49 PM UTC-4, Mark Walkom wrote:

If you query "_nodes?pretty | grep data" you will see a value, that's
where the data is.

PS - We're moving to https://discuss.elastic.co/, please join us there
for any future discussions!

On 29 May 2015 at 00:31, Jim Malone <malo...@gmail.com <javascript:>>
wrote:

I similarly can't find where the data is stored. We have it running with
ES-yarn though. This article says that the data is only stored in
container storage by default (we haven't changed any settings):

https://www.elastic.co/guide/en/elasticsearch/hadoop/master/ey-setup.html

However, the data is definitely persisting somehow between restarts so
it has to be stored somewhere. We just can't find where. There is no
"path.data" dir and also there's no "/var/lib/elasticsearch/data" dir
like I've seen on other posts. And we can't find it in hdfs either.

So can anyone please point me to where we might be able to find the data?
If it helps, we are loading indexes from both hive and from logstash
directly.

Thanks,
Jim

On Wednesday, May 27, 2015 at 6:56:54 PM UTC-4, Mark Walkom wrote:

Check out
Directory layout | Elasticsearch Guide [8.11] | Elastic

PS - We're moving to https://discuss.elastic.co/, please join us there
for any future discussions!

On 28 May 2015 at 01:27, David Pilato da...@pilato.fr wrote:

by default in path.data dir.
In zip or tar.gz distribution, it’s in "data" dir

--
David Pilato - Developer | Evangelist
elastic.co http://elastic.co
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 27 mai 2015 à 17:25, John Adams jada...@gmail.com a écrit :

I cannot find any info on where Elasticsearch is storing it's data.
I would like to manage this potentially valuable information.
Where, how big, how to back it up, restore it, etc.
Then on to clusters, etc..

Thanks,
John

--
Please update your bookmarks! We have moved to
https://discuss.elastic.co/

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to
https://discuss.elastic.co/

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4836BD52-F0A0-4507-BE1D-14E3CA4B0FD7%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/4836BD52-F0A0-4507-BE1D-14E3CA4B0FD7%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to
https://discuss.elastic.co/


You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bc0ad06d-21b3-4ca7-aade-876f15b379a3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bc0ad06d-21b3-4ca7-aade-876f15b379a3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/91f7509c-2cc8-4d57-8fac-f271c27fd75f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

To answer to the initial question, I would not really worry about having acces to the data dir.

There are APIs for that. Nodes Stats and Nodes Info should give you all metrics you need.
Snapshot and restore should help for backups.

HTH

David

Le 29 mai 2015 à 03:04, Jim Malone malonej7@gmail.com a écrit :

Thank you, that little query is very helpful. Although there isn't any actual "data" property anywhere in the response. I'm guessing it's cause our data is still just cached, and it appears to be in the "home" directory. Now we just have to figure out how to set the data dir using es-yarn so that it's stored on the local nodes permanently. Haven't had luck finding that yet.

And I will use the new forum going forward, just wanted to continue on this thread instead of starting a new one.

Jim

On Thursday, May 28, 2015 at 6:10:49 PM UTC-4, Mark Walkom wrote:
If you query "_nodes?pretty | grep data" you will see a value, that's where the data is.

PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions!

On 29 May 2015 at 00:31, Jim Malone malo...@gmail.com wrote:
I similarly can't find where the data is stored. We have it running with ES-yarn though. This article says that the data is only stored in container storage by default (we haven't changed any settings):

https://www.elastic.co/guide/en/elasticsearch/hadoop/master/ey-setup.html

However, the data is definitely persisting somehow between restarts so it has to be stored somewhere. We just can't find where. There is no "path.data" dir and also there's no "/var/lib/elasticsearch/data" dir like I've seen on other posts. And we can't find it in hdfs either.

So can anyone please point me to where we might be able to find the data? If it helps, we are loading indexes from both hive and from logstash directly.

Thanks,
Jim

On Wednesday, May 27, 2015 at 6:56:54 PM UTC-4, Mark Walkom wrote:
Check out Directory layout | Elasticsearch Guide [8.11] | Elastic

PS - We're moving to https://discuss.elastic.co/, please join us there for any future discussions!

On 28 May 2015 at 01:27, David Pilato da...@pilato.fr wrote:
by default in path.data dir.
In zip or tar.gz distribution, it’s in "data" dir

--
David Pilato - Developer | Evangelist
elastic.co
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 27 mai 2015 à 17:25, John Adams jada...@gmail.com a écrit :

I cannot find any info on where Elasticsearch is storing it's data.
I would like to manage this potentially valuable information.
Where, how big, how to back it up, restore it, etc.
Then on to clusters, etc..

Thanks,
John

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4893b923-dbf2-45c3-a0e3-63c6d0b37b1c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4836BD52-F0A0-4507-BE1D-14E3CA4B0FD7%40pilato.fr.

For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bc0ad06d-21b3-4ca7-aade-876f15b379a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/91f7509c-2cc8-4d57-8fac-f271c27fd75f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8664DEDC-87EE-4304-AED1-27E10FE5E78E%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.