Hi, we are experiencing sudden data lost recently. it is only happened on certain index but not all the indexer. we only have today's data starts from 8AM, (before 8AM gone also)
in the elasticsearch log file, I saw pattern for the index creation is very suspicious but duno why it is like that. host-* is one of the index having data lost.
[2016-05-03 00:00:05,934][INFO ][cluster.metadata ] [fslelkprod01] [host-2016.05.02] creating index, cause [auto(bulk api)], templates [hostlog], shards [3]/[0], mappings [Auto::Host::mqs, Auto::Host::err]
[2016-05-03 00:00:06,124][INFO ][cluster.routing.allocation] [fslelkprod01] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[host-2016.05.02][1], [host-2016.05.02][2], [host-2016.05.02][0]] ...]).
[2016-05-03 00:00:06,232][INFO ][cluster.metadata ] [fslelkprod01] [host-2016.05.02] update_mapping [Auto::Host::mqs]
same thing for index amhs-* (also data lost, only have data starts at 8AM today)
[2016-05-03 00:10:02,563][INFO ][cluster.metadata ] [fslelkprod01] [amhs-2016.05.02] creating index, cause [auto(bulk api)], templates [], shards [1]/[0], mappings [AMHSBusinessEventLogs]
[2016-05-03 00:10:02,837][INFO ][cluster.routing.allocation] [fslelkprod01] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[amhs-2016.05.02][0]] ...]).
I don't understand why on today (05/03) it tried to create yesterday's (05/02) data. Normally each day's data is created at 8AM on that day from what i saw in elasticsearch log file. (below highlight is the problem index)
[2016-05-03 08:00:00,075][INFO ][cluster.metadata ] [fslelkprod01] [hostmetrics-2016.05.03] creating index, cause [auto(bulk api)], templates [], shards [1]/[0], mappings [Auto::Host::Metrics]
[2016-05-03 08:00:00,259][INFO ][cluster.metadata ] [fslelkprod01] [onemetrics-2016.05.03] creating index, cause [auto(bulk api)], templates [], shards [1]/[0], mappings [onemetrics]
[2016-05-03 08:00:00,373][INFO ][cluster.metadata ] [fslelkprod01] [esda-2016.05.03] creating index, cause [auto(bulk api)], templates [], shards [1]/[0], mappings [esda]
[2016-05-03 08:00:00,484][INFO ][cluster.routing.allocation] [fslelkprod01] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[hostmetrics-2016.05.03][0], [onemetrics-2016.05.03][0], [hostmetrics-2016.05.03][0], [onemetrics-2016.05.03][0], [hostmetrics-2016.05.03][0], [esda-2016.05.03][0]] ...]).
[2016-05-03 08:00:00,569][INFO ][cluster.metadata ] [fslelkprod01] [e3v1-2016.05.03] creating index, cause [auto(bulk api)], templates [template_e3], shards [1]/[0], mappings [default, R2R_DB_ClientConnection, R2R_DB_ParamTable_CMP_W2W_WaferTracking]
[2016-05-03 08:00:00,692][INFO ][cluster.routing.allocation] [fslelkprod01] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[e3v1-2016.05.03][0]] ...]).
[2016-05-03 08:00:00,746][INFO ][cluster.metadata ] [fslelkprod01] [actgateway-2016.05.03] creating index, cause [auto(bulk api)], templates [gatewaylog], shards [3]/[0], mappings [Auto::ActGateway]
[2016-05-03 08:00:00,950][INFO ][cluster.routing.allocation] [fslelkprod01] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[actgateway-2016.05.03][1], [actgateway-2016.05.03][2], [actgateway-2016.05.03][0]] ...]).
[2016-05-03 08:00:01,035][INFO ][cluster.metadata ] [fslelkprod01] [host-2016.05.03] creating index, cause [auto(bulk api)], templates [hostlog], shards [3]/[0], mappings [Auto::Host::mqs]
[2016-05-03 08:00:01,295][INFO ][cluster.routing.allocation] [fslelkprod01] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[host-2016.05.03][2], [host-2016.05.03][0], [host-2016.05.03][1]] ...]).
[2016-05-03 08:00:01,418][INFO ][cluster.metadata ] [fslelkprod01] [scheduling-2016.05.03] creating index, cause [auto(bulk api)], templates [], shards [1]/[0], mappings [mtsjob]
[2016-05-03 08:00:01,541][INFO ][cluster.routing.allocation] [fslelkprod01] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[scheduling-2016.05.03][0]] ...]).
[2016-05-03 08:00:01,625][INFO ][cluster.metadata ] [fslelkprod01] [rda-2016.05.03] creating index, cause [auto(bulk api)], templates [], shards [1]/[0], mappings [logs]
[2016-05-03 08:00:01,790][INFO ][cluster.routing.allocation] [fslelkprod01] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[rda-2016.05.03][0]] ...]).
[2016-05-03 08:00:01,826][INFO ][cluster.metadata ] [fslelkprod01] [amhs-2016.05.03] creating index, cause [auto(bulk api)], templates [], shards [1]/[0], mappings [AMHSBusinessEventLogs]
[2016-05-03 08:00:01,964][INFO ][cluster.routing.allocation] [fslelkprod01] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[amhs-2016.05.03][0]] ...]).