Update durability vs Solr


(Michael McCandless) #1

What is the default update durability with ElasticSearch...?

Meaning, on OS crash or power loss of one or more (or all) nodes in
a cluster, will all updates be replayed on the next start up? (Is the
transaction log sync'd after every update? That seems incredibly
costly...).

Peeking at the sources it looks like there is a buffering option (add
to the log after N changes/MB have buffered up in RAM) and a
non-buffering option ... so I'm wondering what the default is.

It looks like (not sure though) Solr does not sync the transaction log
after every update (see
http://lucene.markmail.org/thread/qfpv7sq5pe4tuwq6 ) so I'm trying to
compare the two...

Thanks.

Mike

http://blog.mikemccandless.com


(Clinton Gormley) #2

Hi Mike

Peeking at the sources it looks like there is a buffering option (add
to the log after N changes/MB have buffered up in RAM) and a
non-buffering option ... so I'm wondering what the default is.

It looks like the default is 64kB:

clint


(Michael McCandless) #3

On Mon, Jul 16, 2012 at 1:21 PM, Clinton Gormley clint@traveljury.com wrote:

Hi Mike

Peeking at the sources it looks like there is a buffering option (add
to the log after N changes/MB have buffered up in RAM) and a
non-buffering option ... so I'm wondering what the default is.

It looks like the default is 64kB:
https://github.com/elasticsearch/elasticsearch/issues/1549

OK, thanks Clinton.

So the default buffer size when not sync'ing on each update is 64 KB
... does that mean after 64 KB it's flushed and sync'd?

So at worse you'd lose the last 64 KB updates if the cluster goes down?

Separately: is buffering (not sync'ing on each update) the default?

Mike

http://blog.mikemccandless.com


(Jörg Prante) #4

Hi Mike,

the default setting of the translog buffering depends on the setting of the
parameter index.gateway.local.sync (which defaults to "5s")

index.gateway.local.sync > 0: the translog is not sync'd on each operation
but after n milliseconds, so buffering is used
index.gateway.local.sync = 0: the translog is sync'd on each operation,
buffering is disabled
index.gateway.local.sync < 0: the translog is never sync'd

Best regards,

Jörg


(Michael McCandless) #5

Awesome thanks Jörg: that answers my question.

So changes committed earlier than 5s ago should not be lost if cluster
goes down, by default.

When comparing indexing throughput against Solr (or other search apps)
it's important to match the translog configuration, if possible,
because sync is so costly.

Mike

http://blog.mikemccandless.com

On Tue, Jul 17, 2012 at 3:45 PM, Jörg Prante joergprante@gmail.com wrote:

Hi Mike,

the default setting of the translog buffering depends on the setting of the
parameter index.gateway.local.sync (which defaults to "5s")

index.gateway.local.sync > 0: the translog is not sync'd on each operation
but after n milliseconds, so buffering is used
index.gateway.local.sync = 0: the translog is sync'd on each operation,
buffering is disabled
index.gateway.local.sync < 0: the translog is never sync'd

Best regards,

Jörg


(Shay Banon) #6

Note also, that elasticsearch does (by default) sync replication to all
shard copies of the data, each including its own transaction log.

On Tue, Jul 17, 2012 at 1:03 PM, Michael McCandless <mail@mikemccandless.com

wrote:

Awesome thanks Jörg: that answers my question.

So changes committed earlier than 5s ago should not be lost if cluster
goes down, by default.

When comparing indexing throughput against Solr (or other search apps)
it's important to match the translog configuration, if possible,
because sync is so costly.

Mike

http://blog.mikemccandless.com

On Tue, Jul 17, 2012 at 3:45 PM, Jörg Prante joergprante@gmail.com
wrote:

Hi Mike,

the default setting of the translog buffering depends on the setting of
the
parameter index.gateway.local.sync (which defaults to "5s")

index.gateway.local.sync > 0: the translog is not sync'd on each
operation
but after n milliseconds, so buffering is used
index.gateway.local.sync = 0: the translog is sync'd on each operation,
buffering is disabled
index.gateway.local.sync < 0: the translog is never sync'd

Best regards,

Jörg


(system) #7