Losing data if I don't flush

Elastic Stack Elasticsearch
1

During development I frequently restart the application that I have
elasticsearch embedded in. I noticed that the last update to an index
is often lost--unless the indexes are flushed. Shouldn't close() be
enough?

public void init() {
Settings settings = ImmutableSettings.settingsBuilder()
.put("index.mapper.dynamic", false)
.put("index.cache.filter.type", "none")
.put("action.auto_create_index", false).build();
node = NodeBuilder.nodeBuilder().settings(settings).node();
client = node.client();
recover();
}

public void close() {

client.admin().indices().prepareFlush("_all").execute().actionGet();
client.close();
node.close();
}

2

On Mar 6, 12:01 pm, Eric Jain eric.j...@gmail.com wrote:

During development I frequently restart the application that I have
elasticsearch embedded in. I noticed that the last update to an index
is often lost--unless the indexes are flushed. Shouldn't close() be
enough?

Looking at the translog dir, here's what I see:

translog-1331003921694 (0KB)
(update index)
translog-1331003921694 (3KB)
(unchanged after a minute; restart)
translog-1331003921695 (0KB)
translog-1331003921694.recovering (3KB)
(unchanged after a minute; restart)
translog-1331003921695 (0KB)
translog-1331003921695.recovering (0KB)
(unchanged after a minute; restart)
translog-1331003921695 (0KB)
translog-1331003921695.recovering (0KB)
(update index)
translog-1331003921695 (7KB)
translog-1331003921695.recovering (0KB)
(unchanged after a minute; restart)
translog-1331003921695 (0KB)
translog-1331003921695.recovering (0KB)
...

3

It shouldn't happen, can you create a simple test case that shows it? Also, which version are you using?

On Tuesday, March 6, 2012 at 10:39 PM, Eric Jain wrote:

On Mar 6, 12:01 pm, Eric Jain <eric.j...@gmail.com (http://gmail.com)> wrote:

During development I frequently restart the application that I have
elasticsearch embedded in. I noticed that the last update to an index
is often lost--unless the indexes are flushed. Shouldn't close() be
enough?

Looking at the translog dir, here's what I see:

translog-1331003921694 (0KB)
(update index)
translog-1331003921694 (3KB)
(unchanged after a minute; restart)
translog-1331003921695 (0KB)
translog-1331003921694.recovering (3KB)
(unchanged after a minute; restart)
translog-1331003921695 (0KB)
translog-1331003921695.recovering (0KB)
(unchanged after a minute; restart)
translog-1331003921695 (0KB)
translog-1331003921695.recovering (0KB)
(update index)
translog-1331003921695 (7KB)
translog-1331003921695.recovering (0KB)
(unchanged after a minute; restart)
translog-1331003921695 (0KB)
translog-1331003921695.recovering (0KB)
...

4

On Wed, Mar 7, 2012 at 03:16, Shay Banon kimchy@gmail.com wrote:

It shouldn't happen, can you create a simple test case that shows it? Also,
which version are you using?

0.19.0

I haven't been able to reproduce the issue with a simple test case
yet. The index only gets into the "updates are lost unless flushed"
after a while...

While writing a test case, I encountered another oddity that I haven't
seen otherwise: If the number_of_replicas for an index is set to more
than 0, the cluster state doesn't ever seem to get past YELLOW:

gist:1995058 · GitHub

What's wrong with this code?

5

If you have a single node, and create an index with index.number_of_replicas > 0, the health will be yellow (since not all replicas are allocated). It has always been like that.

On Wednesday, March 7, 2012 at 8:58 PM, Eric Jain wrote:

On Wed, Mar 7, 2012 at 03:16, Shay Banon <kimchy@gmail.com (mailto:kimchy@gmail.com)> wrote:

It shouldn't happen, can you create a simple test case that shows it? Also,
which version are you using?

0.19.0

I haven't been able to reproduce the issue with a simple test case
yet. The index only gets into the "updates are lost unless flushed"
after a while...

While writing a test case, I encountered another oddity that I haven't
seen otherwise: If the number_of_replicas for an index is set to more
than 0, the cluster state doesn't ever seem to get past YELLOW:

gist:1995058 · GitHub

What's wrong with this code?