Doc's are getting deleted automatically

[kish]

Doc in the index are getting deleted automatically from one particular index alone i.e immediately . I am able to see in indices output and very next second it is getting deleted.

One the ES logs i am to see as below when deletion happened.

[2018-07-06T14:51:47,067][DEBUG][o.e.i.r.TransportDeleteByQueryAction] [cscfhcalloam002] executing initial scroll against [register][register]
[2018-07-06T14:51:47,072][DEBUG][o.e.i.r.TransportDeleteByQueryAction] [cscfhcalloam002] scroll returned [1] documents with a scroll id of [DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAA4RFmFkSHo1VVhjVG1HZjZBYkp6ZEF4Z3cAAAAAAAAOEBZhZEh6NVVYY1RtR2Y2QWJKemRBeGd3AAAAAAAAEuYWeEJ5aXV3NWtRcW1TWlc5RGpiQUFwdwAAAAAAABLoFnhCeWl1dzVrUXFtU1pXOURqYkFBcHcAAAAAAAAS5xZ4QnlpdXc1a1FxbVNaVzlEamJBQXB3]
[2018-07-06T14:51:47,073][DEBUG][o.e.i.r.TransportDeleteByQueryAction] [cscfhcalloam002] sending [1] entry, [50b] bulk request
[2018-07-06T14:51:47,293][DEBUG][o.e.i.r.TransportDeleteByQueryAction] [cscfhcalloam002] Freed [5] contexts

Problematic index ,
root@cscfhcalloam002> curl -XGET 'localhost:9200/noderegister/_mapping?pretty'

{
"noderegister" : {
"mappings" : {
"register" : {
"properties" : {
"Address" : {
"type" : "ip"
},
"NodeName" : {
"type" : "keyword"
},
"NodeType" : {
"type" : "keyword"
},
"Uuid" : {
"type" : "keyword"
}
}
},
"noderegister1" : {
"properties" : {
"Address" : {
"type" : "ip"
},
"NodeName" : {
"type" : "keyword"
},
"NodeType" : {
"type" : "keyword"
},
"Uuid" : {
"type" : "keyword"
}
}
}
}
}
}

I am not sure whether this the problem but it might.
First created index is 'register' with type 'register'. later there was some issues so we created other index with name 'noderegister' with type 'noderegister1' and moved all data from 'register' to noderegister and deleted the ole 'register' index and gave alias name 'noderegister' as 'register'.
Above part is perfectly working. And when try to add data it is working but it is getting deleted immediately.

kindly check as huge data is getting deleted.

[dadoonet (David Pilato)]

There is nothing in elasticsearch which does that automatically.

So you have something or someone who is sending those requests to your cluster.

I hope it's not exposed publicly without any protection.

[kish]

No, there is no such request coming. We are 100% sure.
And also in indices output also we can see that docs are getting deleted less than 7-10sec.

is there any logs you need or where we can check?

[dadoonet (David Pilato)]

Try to remove the Network connection. If this is not happening anymore, then my theory is exact.

If it's still happening, run a ps -ef to see what is running locally.

[kish]

@dadoonet

Try to remove the Network connection

Could you to explain it? you mean to say external connection? If then it is cluster node with no access to other network elements.
If some process is deleting it then in at least we should be able to see in logs? i dont see any such thing.

[dadoonet (David Pilato)]

What is the output of

GET /_cat/nodes?v
GET /_cat/plugins?v
GET /_cat/indices?v

[dadoonet (David Pilato)]

I forgot to ask. Are you running with a old version of elasticsearch and using the now removed TTL feature?

[kish]

please the details

root@cscfhcalloam002> curl -XGET 'localhost:9200'
{
  "name" : "cscfhcalloam002",
  "cluster_name" : "elastic-OAMUNIT",
  "cluster_uuid" : "hQBPen2kRNe8IlxOdbcTOg",
  "version" : {
    "number" : "5.1.1",
    "build_hash" : "5395e21",
    "build_date" : "2017-02-21T12:09:24.490Z",
    "build_snapshot" : true,
    "lucene_version" : "6.3.0"
  },
  "tagline" : "You Know, for Search"
}
root@cscfhcalloam002> curl -XGET 'localhost:9200/_cat/indices?v'
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   tdcorescaleio     6k2HYYU8SlOFbHJ1H-4lHQ   5   1          5            0     37.5kb         18.7kb
green  open   pmdatastore       bPcWADZCSPSRXzGPMw9fXQ   5   1          0            0      1.3kb           708b
green  open   tdgmscaleio       gaIu_X8fR_KLoSn7ZeNjrA   5   1          0            0      1.2kb           650b
green  open   hsscallpcounter   I5jL7Y7IQ0KQ_kSlvm4dZw   5   1          0            0      1.3kb           708b
green  open   cscfcounter       9uMfrHm9R8ClBPSGTHBedQ   5   1          0            0      1.3kb           708b
green  open   noderegister      Jddxe1pFRHSZeUZcMpvR0g   5   1          0            0      1.5kb           795b
green  open   fmdatastore       gQ-1DzQ5SLG76CvTWtxoAA   5   1          0            0      1.2kb           650b
green  open   diameterlbcounter PVLzvzyBSp2w6HPIRbpexA   5   1          0            0      1.3kb           679b
green  open   cscfscaleio       _aZ8bWFuSLuZzttodCEXKw   5   1          0            0      1.2kb           650b
green  open   tdgmcounter       60bcktJwSa6SFKyBk4kdnA   5   1          0            0      1.3kb           708b
green  open   tdcorecounter     YPVTp0X5RC6N_y7ssIAeng   5   1          6            0     79.8kb         39.9kb
green  open   lbcounter         owo9ZydeSEKMJMet_5J_dw   5   1          0            0      1.3kb           708b
green  open   queueio           Wl-EzkwSTgWEjIjy4xMv_A   5   1          0            0      1.2kb           650b
root@cscfhcalloam002> curl -XGET 'localhost:9200/_cat/plugins?v'
name component version
root@cscfhcalloam002> curl -XGET 'localhost:9200/_cat/nodes?v'
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
2a00::108a            5          30  11    0.61    0.35     0.32 mdi       -      cscfhcalloam003
2a00::1089            8          33  40    2.19    2.22     2.15 mdi       *      cscfhcalloam002
2a00::1088            6          27   2    0.22    0.20     0.16 mdi       -      cscfhcalloam001
root@cscfhcalloam002>

[kish]

few more details
root@cscfhcalloam002> curl localhost:9200/_cat/health?v

epoch      timestamp cluster         status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1530876279 16:54:39  elastic-OAMUNIT green           3         3    130  65    0    0        0             0                  -                100.0%

[dadoonet (David Pilato)]

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

[kish]

Any clue/help ?

[dadoonet (David Pilato)]

Can you trace what are the network connections on each on your 3 nodes on port 9200?

[kish]

At present i don't have the setup to share the details, will update once i get the setup.
sorry for Inconvenience