Data vanishes a few seconds after indexing

For some of our types, indexing appears to work, but data vanishes a few
seconds later. Specifically:

  1. PUT this data https://gist.github.com/dbarlett/9695d8180da3aeafa65cto http://localhost:9200/app35/template314/212608 (via our app or the Head
    plugin)
  2. Response:
    {
    ok: true
    _index: app35
    _type: template314
    _id: 212608
    _version: 1
    }
  3. GET http://localhost:9200/app35/template314/212608
  4. Receive expected responsehttps://gist.github.com/dbarlett/edf7e181f24ed202060f
  5. Wait a few seconds and GET same URL
  6. Data is gone:{

"_index" : "app35",
"_type" : "template314",
"_id" : "212608",
"exists" : false
}

Environment:

  • es 0.20.5
  • JVM 1.7.0_17 64-bit
  • Window Server 2008 R2 64-bit
  • Four cluster nodes (2 frontends with node.data: false and Jetty
    plugin, 2 backends with node.data: true, all have node.master: true)

There are no error messages in the logs, and other data is retained
indefintely. We haven't found any meaningful distinction between data that
stays and data that vanishes. Has anyone seen this before?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

The behavior occurs regardless of the node that data is PUT to.

On Wednesday, April 3, 2013 12:18:48 PM UTC-4, Dylan Barlett wrote:

For some of our types, indexing appears to work, but data vanishes a few
seconds later. Specifically:

  1. PUT this datahttps://gist.github.com/dbarlett/9695d8180da3aeafa65cto
    http://localhost:9200/app35/template314/212608 (via our app or the
    Head plugin)
  2. Response:
    {
    ok: true
    _index: app35
    _type: template314
    _id: 212608
    _version: 1
    }
  3. GET http://localhost:9200/app35/template314/212608
  4. Receive expected responsehttps://gist.github.com/dbarlett/edf7e181f24ed202060f
  5. Wait a few seconds and GET same URL
  6. Data is gone:{

"_index" : "app35",
"_type" : "template314",
"_id" : "212608",
"exists" : false
}

Environment:

  • es 0.20.5
  • JVM 1.7.0_17 64-bit
  • Window Server 2008 R2 64-bit
  • Four cluster nodes (2 frontends with node.data: false and Jetty
    plugin, 2 backends with node.data: true, all have node.master: true)

There are no error messages in the logs, and other data is retained
indefintely. We haven't found any meaningful distinction between data that
stays and data that vanishes. Has anyone seen this before?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Do you have _ttl enabled by any chance in the mappings for the types where
records disappear?
What happens when you run get several times?
Try running

curl "localhost:9200/app35/_stats?pretty=true"

while you index data. What happens to counters in count and deleted?

"indices" : {
"app35" : {
"primaries" : {
"docs" : {
"count" : ????,
"deleted" : ????

On Wednesday, April 3, 2013 12:23:44 PM UTC-4, Dylan Barlett wrote:

The behavior occurs regardless of the node that data is PUT to.

On Wednesday, April 3, 2013 12:18:48 PM UTC-4, Dylan Barlett wrote:

For some of our types, indexing appears to work, but data vanishes a few
seconds later. Specifically:

  1. PUT this datahttps://gist.github.com/dbarlett/9695d8180da3aeafa65cto
    http://localhost:9200/app35/template314/212608 (via our app or the
    Head plugin)
  2. Response:
    {
    ok: true
    _index: app35
    _type: template314
    _id: 212608
    _version: 1
    }
  3. GET http://localhost:9200/app35/template314/212608
  4. Receive expected responsehttps://gist.github.com/dbarlett/edf7e181f24ed202060f
  5. Wait a few seconds and GET same URL
  6. Data is gone:{

"_index" : "app35",
"_type" : "template314",
"_id" : "212608",
"exists" : false
}

Environment:

  • es 0.20.5
  • JVM 1.7.0_17 64-bit
  • Window Server 2008 R2 64-bit
  • Four cluster nodes (2 frontends with node.data: false and Jetty
    plugin, 2 backends with node.data: true, all have node.master: true)

There are no error messages in the logs, and other data is retained
indefintely. We haven't found any meaningful distinction between data that
stays and data that vanishes. Has anyone seen this before?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks for cluing me in about _stats. _ttl is not enabled.

Before PUT: "count" : 138, "deleted" : 15

Immediately after PUT: "count": 139, "deleted" : 15
A few seconds after PUT: "count" : 138, "deleted" : 15

On Wednesday, April 3, 2013 3:13:04 PM UTC-4, Igor Motov wrote:

Do you have _ttl enabled by any chance in the mappings for the types where
records disappear?
What happens when you run get several times?
Try running

curl "localhost:9200/app35/_stats?pretty=true"

while you index data. What happens to counters in count and deleted?

"indices" : {
"app35" : {
"primaries" : {
"docs" : {
"count" : ????,
"deleted" : ????

On Wednesday, April 3, 2013 12:23:44 PM UTC-4, Dylan Barlett wrote:

The behavior occurs regardless of the node that data is PUT to.

On Wednesday, April 3, 2013 12:18:48 PM UTC-4, Dylan Barlett wrote:

For some of our types, indexing appears to work, but data vanishes a few
seconds later. Specifically:

  1. PUT this datahttps://gist.github.com/dbarlett/9695d8180da3aeafa65cto
    http://localhost:9200/app35/template314/212608 (via our app or the
    Head plugin)
  2. Response:
    {
    ok: true
    _index: app35
    _type: template314
    _id: 212608
    _version: 1
    }
  3. GET http://localhost:9200/app35/template314/212608
  4. Receive expected responsehttps://gist.github.com/dbarlett/edf7e181f24ed202060f
  5. Wait a few seconds and GET same URL
  6. Data is gone:{

"_index" : "app35",
"_type" : "template314",
"_id" : "212608",
"exists" : false
}

Environment:

  • es 0.20.5
  • JVM 1.7.0_17 64-bit
  • Window Server 2008 R2 64-bit
  • Four cluster nodes (2 frontends with node.data: false and Jetty
    plugin, 2 backends with node.data: true, all have node.master: true)

There are no error messages in the logs, and other data is retained
indefintely. We haven't found any meaningful distinction between data that
stays and data that vanishes. Has anyone seen this before?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey,

very strange problem. Can you reduce complexity and still reproduce this
behaviour?
Does this happen, when you only use one node?
Can you remove all plugins to ensure this is a barebones elasticsearch
problem?

I dont use windows. Are there any mechanisms preventing it to open more
files/file descriptors at some stage?

Last question: Is the document searchable at any time or are you only able
to GET it (which means it might not yet be indexed for search)?

On Wed, Apr 3, 2013 at 9:39 PM, Dylan Barlett dylan.barlett@gmail.comwrote:

Thanks for cluing me in about _stats. _ttl is not enabled.

Before PUT: "count" : 138, "deleted" : 15

Immediately after PUT: "count": 139, "deleted" : 15
A few seconds after PUT: "count" : 138, "deleted" : 15

On Wednesday, April 3, 2013 3:13:04 PM UTC-4, Igor Motov wrote:

Do you have _ttl enabled by any chance in the mappings for the types
where records disappear?
What happens when you run get several times?
Try running

curl "localhost:9200/app35/_stats?**pretty=true"

while you index data. What happens to counters in count and deleted?

"indices" : {
"app35" : {
"primaries" : {
"docs" : {
"count" : ????,
"deleted" : ????

On Wednesday, April 3, 2013 12:23:44 PM UTC-4, Dylan Barlett wrote:

The behavior occurs regardless of the node that data is PUT to.

On Wednesday, April 3, 2013 12:18:48 PM UTC-4, Dylan Barlett wrote:

For some of our types, indexing appears to work, but data vanishes a
few seconds later. Specifically:

  1. PUT this datahttps://gist.github.com/dbarlett/9695d8180da3aeafa65cto
    http://localhost:9200/app35/**template314/212608http://localhost:9200/app35/template314/212608(via our app or the Head plugin)
  2. Response:
    {
    ok: true
    _index: app35
    _type: template314
    _id: 212608
    _version: 1
    }
  3. GET http://localhost:9200/app35/**template314/212608http://localhost:9200/app35/template314/212608
  4. Receive expected responsehttps://gist.github.com/dbarlett/edf7e181f24ed202060f
  5. Wait a few seconds and GET same URL
  6. Data is gone:{

"_index" : "app35",
"_type" : "template314",
"_id" : "212608",
"exists" : false
}

Environment:

  • es 0.20.5
  • JVM 1.7.0_17 64-bit
  • Window Server 2008 R2 64-bit
  • Four cluster nodes (2 frontends with node.data: false and Jetty
    plugin, 2 backends with node.data: true, all have node.master: true)

There are no error messages in the logs, and other data is retained
indefintely. We haven't found any meaningful distinction between data that
stays and data that vanishes. Has anyone seen this before?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Can you try to GET the document from a backend data node instead of frontend node?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 4 avr. 2013 à 08:21, Alexander Reelsen alr@spinscale.de a écrit :

Hey,

very strange problem. Can you reduce complexity and still reproduce this behaviour?
Does this happen, when you only use one node?
Can you remove all plugins to ensure this is a barebones elasticsearch problem?

I dont use windows. Are there any mechanisms preventing it to open more files/file descriptors at some stage?

Last question: Is the document searchable at any time or are you only able to GET it (which means it might not yet be indexed for search)?

On Wed, Apr 3, 2013 at 9:39 PM, Dylan Barlett dylan.barlett@gmail.com wrote:
Thanks for cluing me in about _stats. _ttl is not enabled.

Before PUT: "count" : 138, "deleted" : 15
Immediately after PUT: "count": 139, "deleted" : 15
A few seconds after PUT: "count" : 138, "deleted" : 15

On Wednesday, April 3, 2013 3:13:04 PM UTC-4, Igor Motov wrote:
Do you have _ttl enabled by any chance in the mappings for the types where records disappear?
What happens when you run get several times?
Try running

curl "localhost:9200/app35/_stats?pretty=true"

while you index data. What happens to counters in count and deleted?

"indices" : {
"app35" : {
"primaries" : {
"docs" : {
"count" : ????,
"deleted" : ????

On Wednesday, April 3, 2013 12:23:44 PM UTC-4, Dylan Barlett wrote:
The behavior occurs regardless of the node that data is PUT to.

On Wednesday, April 3, 2013 12:18:48 PM UTC-4, Dylan Barlett wrote:
For some of our types, indexing appears to work, but data vanishes a few seconds later. Specifically:
PUT this data to http://localhost:9200/app35/template314/212608 (via our app or the Head plugin)
Response:
{
ok: true
_index: app35
_type: template314
_id: 212608
_version: 1
}
GET http://localhost:9200/app35/template314/212608
Receive expected response
Wait a few seconds and GET same URL
Data is gone:{

"_index" : "app35",

"_type" : "template314",

"_id" : "212608",

"exists" : false
}

Environment:

es 0.20.5
JVM 1.7.0_17 64-bit
Window Server 2008 R2 64-bit
Four cluster nodes (2 frontends with node.data: false and Jetty plugin, 2 backends with node.data: true, all have node.master: true)
There are no error messages in the logs, and other data is retained indefintely. We haven't found any meaningful distinction between data that stays and data that vanishes. Has anyone seen this before?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.