Doc's are getting deleted automatically

kish · July 6, 2018, 9:30am

Doc in the index are getting deleted automatically from one particular index alone i.e immediately . I am able to see in indices output and very next second it is getting deleted.

One the ES logs i am to see as below when deletion happened.

[2018-07-06T14:51:47,067][DEBUG][o.e.i.r.TransportDeleteByQueryAction] [cscfhcalloam002] executing initial scroll against [register][register]
[2018-07-06T14:51:47,072][DEBUG][o.e.i.r.TransportDeleteByQueryAction] [cscfhcalloam002] scroll returned [1] documents with a scroll id of [DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAA4RFmFkSHo1VVhjVG1HZjZBYkp6ZEF4Z3cAAAAAAAAOEBZhZEh6NVVYY1RtR2Y2QWJKemRBeGd3AAAAAAAAEuYWeEJ5aXV3NWtRcW1TWlc5RGpiQUFwdwAAAAAAABLoFnhCeWl1dzVrUXFtU1pXOURqYkFBcHcAAAAAAAAS5xZ4QnlpdXc1a1FxbVNaVzlEamJBQXB3]
[2018-07-06T14:51:47,073][DEBUG][o.e.i.r.TransportDeleteByQueryAction] [cscfhcalloam002] sending [1] entry, [50b] bulk request
[2018-07-06T14:51:47,293][DEBUG][o.e.i.r.TransportDeleteByQueryAction] [cscfhcalloam002] Freed [5] contexts

Problematic index ,
root@cscfhcalloam002> curl -XGET 'localhost:9200/noderegister/_mapping?pretty'

{
"noderegister" : {
"mappings" : {
"register" : {
"properties" : {
"Address" : {
"type" : "ip"
},
"NodeName" : {
"type" : "keyword"
},
"NodeType" : {
"type" : "keyword"
},
"Uuid" : {
"type" : "keyword"
}
}
},
"noderegister1" : {
"properties" : {
"Address" : {
"type" : "ip"
},
"NodeName" : {
"type" : "keyword"
},
"NodeType" : {
"type" : "keyword"
},
"Uuid" : {
"type" : "keyword"
}
}
}
}
}
}

I am not sure whether this the problem but it might.
First created index is 'register' with type 'register'. later there was some issues so we created other index with name 'noderegister' with type 'noderegister1' and moved all data from 'register' to noderegister and deleted the ole 'register' index and gave alias name 'noderegister' as 'register'.
Above part is perfectly working. And when try to add data it is working but it is getting deleted immediately.

kindly check as huge data is getting deleted.

dadoonet · July 6, 2018, 9:55am

There is nothing in elasticsearch which does that automatically.

So you have something or someone who is sending those requests to your cluster.

I hope it's not exposed publicly without any protection.

kish · July 6, 2018, 10:09am

No, there is no such request coming. We are 100% sure.
And also in indices output also we can see that docs are getting deleted less than 7-10sec.

is there any logs you need or where we can check?

dadoonet · July 6, 2018, 10:36am

Try to remove the Network connection. If this is not happening anymore, then my theory is exact.

If it's still happening, run a ps -ef to see what is running locally.

kish · July 6, 2018, 10:40am

Could you to explain it? you mean to say external connection? If then it is cluster node with no access to other network elements.
If some process is deleting it then in at least we should be able to see in logs? i dont see any such thing.

dadoonet · July 6, 2018, 11:05am

What is the output of

GET /_cat/nodes?v
GET /_cat/plugins?v
GET /_cat/indices?v

dadoonet · July 6, 2018, 11:06am

I forgot to ask. Are you running with a old version of elasticsearch and using the now removed TTL feature?

kish · July 6, 2018, 11:22am

please the details

root@cscfhcalloam002> curl -XGET 'localhost:9200'
{
  "name" : "cscfhcalloam002",
  "cluster_name" : "elastic-OAMUNIT",
  "cluster_uuid" : "hQBPen2kRNe8IlxOdbcTOg",
  "version" : {
    "number" : "5.1.1",
    "build_hash" : "5395e21",
    "build_date" : "2017-02-21T12:09:24.490Z",
    "build_snapshot" : true,
    "lucene_version" : "6.3.0"
  },
  "tagline" : "You Know, for Search"
}
root@cscfhcalloam002> curl -XGET 'localhost:9200/_cat/indices?v'
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   tdcorescaleio     6k2HYYU8SlOFbHJ1H-4lHQ   5   1          5            0     37.5kb         18.7kb
green  open   pmdatastore       bPcWADZCSPSRXzGPMw9fXQ   5   1          0            0      1.3kb           708b
green  open   tdgmscaleio       gaIu_X8fR_KLoSn7ZeNjrA   5   1          0            0      1.2kb           650b
green  open   hsscallpcounter   I5jL7Y7IQ0KQ_kSlvm4dZw   5   1          0            0      1.3kb           708b
green  open   cscfcounter       9uMfrHm9R8ClBPSGTHBedQ   5   1          0            0      1.3kb           708b
green  open   noderegister      Jddxe1pFRHSZeUZcMpvR0g   5   1          0            0      1.5kb           795b
green  open   fmdatastore       gQ-1DzQ5SLG76CvTWtxoAA   5   1          0            0      1.2kb           650b
green  open   diameterlbcounter PVLzvzyBSp2w6HPIRbpexA   5   1          0            0      1.3kb           679b
green  open   cscfscaleio       _aZ8bWFuSLuZzttodCEXKw   5   1          0            0      1.2kb           650b
green  open   tdgmcounter       60bcktJwSa6SFKyBk4kdnA   5   1          0            0      1.3kb           708b
green  open   tdcorecounter     YPVTp0X5RC6N_y7ssIAeng   5   1          6            0     79.8kb         39.9kb
green  open   lbcounter         owo9ZydeSEKMJMet_5J_dw   5   1          0            0      1.3kb           708b
green  open   queueio           Wl-EzkwSTgWEjIjy4xMv_A   5   1          0            0      1.2kb           650b
root@cscfhcalloam002> curl -XGET 'localhost:9200/_cat/plugins?v'
name component version
root@cscfhcalloam002> curl -XGET 'localhost:9200/_cat/nodes?v'
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
2a00::108a            5          30  11    0.61    0.35     0.32 mdi       -      cscfhcalloam003
2a00::1089            8          33  40    2.19    2.22     2.15 mdi       *      cscfhcalloam002
2a00::1088            6          27   2    0.22    0.20     0.16 mdi       -      cscfhcalloam001
root@cscfhcalloam002>

kish · July 6, 2018, 11:28am

few more details
root@cscfhcalloam002> curl localhost:9200/_cat/health?v

epoch      timestamp cluster         status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1530876279 16:54:39  elastic-OAMUNIT green           3         3    130  65    0    0        0             0                  -                100.0%

dadoonet · July 6, 2018, 11:47am

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

kish · July 8, 2018, 2:52pm

Any clue/help ?

dadoonet · July 8, 2018, 6:14pm

Can you trace what are the network connections on each on your 3 nodes on port 9200?

kish · July 11, 2018, 9:49am

At present i don't have the setup to share the details, will update once i get the setup.
sorry for Inconvenience

system · August 8, 2018, 9:49am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.