Transactions logs stored at the wrong place

Hello Community,

I noticed a strange behavior while setting up my ES 1.4.4 cluster: the
transactions logs seem to be stored at the wrong place.

I start ES using the option --default.path.data=/var/lib/elasticsearch
(default setting when using deb packages).
Then, in /etc/elasticsearch/elasticsearch.yml, I change the data path for 4
SSD :

path.data:
- /data/self/1
- /data/self/2
- /data/self/3
- /data/self/4

When I create an index and start pushing data, the shards are correctly
created on my SSD
(/data/self//clustername/nodes/0/indices/indexname//index/) but the
transaction logs are created in
/var/lib/elasticsearch/clustername/nodes/0/indices/indexname/
/translog/translog-*.

It seems like the config file did correctly override the data path setting
for the shards but not for the transactions logs.

Did I miss something or is it a bug ?

Regards,

Hervé BRY
System Administrator
www.geneanet.org

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/72e5ff83-c4e0-42c5-b3ff-2dd3ef41b352%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Well the translog isn't really data like an index is, however you raise a
valid point if this is the behaviour.

Is it a problem for you to have them there or just a point of interest?

On 27 February 2015 at 01:58, Hervé Bry herve.bry@geneanet.org wrote:

Hello Community,

I noticed a strange behavior while setting up my ES 1.4.4 cluster: the
transactions logs seem to be stored at the wrong place.

I start ES using the option --default.path.data=/var/lib/elasticsearch
(default setting when using deb packages).
Then, in /etc/elasticsearch/elasticsearch.yml, I change the data path for
4 SSD :

path.data:
- /data/self/1
- /data/self/2
- /data/self/3
- /data/self/4

When I create an index and start pushing data, the shards are correctly
created on my SSD
(/data/self//clustername/nodes/0/indices/indexname//index/) but the
transaction logs are created in
/var/lib/elasticsearch/clustername/nodes/0/indices/indexname/
/translog/translog-*.

It seems like the config file did correctly override the data path setting
for the shards but not for the transactions logs.

Did I miss something or is it a bug ?

Regards,

Hervé BRY
System Administrator
www.geneanet.org

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/72e5ff83-c4e0-42c5-b3ff-2dd3ef41b352%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/72e5ff83-c4e0-42c5-b3ff-2dd3ef41b352%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9%3DFTtgUULodnm_OVSJDDY%3D%3Dqpgio%3DALQ618hv830gY%3DA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Well, my /var/lib partition is small while I have 4 SSD with plenty of
space that I use to store indices.
So I have ran several times into a "no more space on device" situation when
a developer decided to increase index.translog.flush_threshold_size to
speed up indexing. And then I could not even flush the index because there
was not enough space to create new empty translogs.

I managed to move the translogs to the path.data configured in
elasticsearch.yml by removing the --default.path.data option from the init
script. However this is not a good solution as it involves altering a
packaged file.

It would be nice to either store translogs correctly with the corresponding
indices or have a setting to change the storage path.

Regards,

Hervé BRY
System Administrator

Le dimanche 1 mars 2015 09:15:08 UTC+1, Mark Walkom a écrit :

Well the translog isn't really data like an index is, however you raise a
valid point if this is the behaviour.

Is it a problem for you to have them there or just a point of interest?

On 27 February 2015 at 01:58, Hervé Bry <herv...@geneanet.org
<javascript:>> wrote:

Hello Community,

I noticed a strange behavior while setting up my ES 1.4.4 cluster: the
transactions logs seem to be stored at the wrong place.

I start ES using the option --default.path.data=/var/lib/elasticsearch
(default setting when using deb packages).
Then, in /etc/elasticsearch/elasticsearch.yml, I change the data path for
4 SSD :

path.data:
- /data/self/1
- /data/self/2
- /data/self/3
- /data/self/4

When I create an index and start pushing data, the shards are correctly
created on my SSD
(/data/self//clustername/nodes/0/indices/indexname//index/) but the
transaction logs are created in
/var/lib/elasticsearch/clustername/nodes/0/indices/indexname/
/translog/translog-*.

It seems like the config file did correctly override the data path
setting for the shards but not for the transactions logs.

Did I miss something or is it a bug ?

Regards,

Hervé BRY
System Administrator
www.geneanet.org

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/72e5ff83-c4e0-42c5-b3ff-2dd3ef41b352%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/72e5ff83-c4e0-42c5-b3ff-2dd3ef41b352%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/285ac8c0-b0bd-4885-943c-e088a5989673%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Understood! You should definitely raise this as an issue on Github -

On 2 March 2015 at 21:10, Hervé Bry herve.bry@geneanet.org wrote:

Well, my /var/lib partition is small while I have 4 SSD with plenty of
space that I use to store indices.
So I have ran several times into a "no more space on device" situation
when a developer decided to increase index.translog.flush_threshold_size to
speed up indexing. And then I could not even flush the index because there
was not enough space to create new empty translogs.

I managed to move the translogs to the path.data configured in
elasticsearch.yml by removing the --default.path.data option from the init
script. However this is not a good solution as it involves altering a
packaged file.

It would be nice to either store translogs correctly with the
corresponding indices or have a setting to change the storage path.

Regards,

Hervé BRY
System Administrator
www.geneanet.org

Le dimanche 1 mars 2015 09:15:08 UTC+1, Mark Walkom a écrit :

Well the translog isn't really data like an index is, however you raise a
valid point if this is the behaviour.

Is it a problem for you to have them there or just a point of interest?

On 27 February 2015 at 01:58, Hervé Bry herv...@geneanet.org wrote:

Hello Community,

I noticed a strange behavior while setting up my ES 1.4.4 cluster: the
transactions logs seem to be stored at the wrong place.

I start ES using the option --default.path.data=/var/lib/elasticsearch
(default setting when using deb packages).
Then, in /etc/elasticsearch/elasticsearch.yml, I change the data path
for 4 SSD :

path.data:
- /data/self/1
- /data/self/2
- /data/self/3
- /data/self/4

When I create an index and start pushing data, the shards are correctly
created on my SSD (/data/self//clustername/nodes/0/indices/indexname//index/)
but the transaction logs are created in /var/lib/elasticsearch/
clustername/nodes/0/indices/indexname/
/translog/translog-*.

It seems like the config file did correctly override the data path
setting for the shards but not for the transactions logs.

Did I miss something or is it a bug ?

Regards,

Hervé BRY
System Administrator
www.geneanet.org

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/72e5ff83-c4e0-42c5-b3ff-2dd3ef41b352%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/72e5ff83-c4e0-42c5-b3ff-2dd3ef41b352%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/285ac8c0-b0bd-4885-943c-e088a5989673%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/285ac8c0-b0bd-4885-943c-e088a5989673%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-0dFvCO1nyJupJ25avLXC_bbPwT_LNMZtFoNqzPfU%3DzA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.