Doc.deleted keeps increasing in indices

Elastic Stack Elasticsearch
1

Hello guys,

Could you help me why "docs.count" below is not increasing? it's stack at
2307764. while the "docs.deleted" keeps increasing.

i'm just running a php script to ingest the csv source data to ES.

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 412834 1.2gb
638mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 421090 1.2gb
637.8mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 434017 1.2gb
638.5mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 437557 1.2gb
632.6mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 451098 1.2gb
642.4mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 514957 1.3gb
670.1mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 882639 1.4gb
734.7mb

It's running 4 nodes of c3.large. ES-1.2.3 Is there anything I need to add
in the elasticsearch.yml below ?

cluster.name: clustername

node.name: “machine00"

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: [“machine01","machine02","machine03”]

discovery.zen.minimum_master_nodes: 3

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9b056741-69f8-4288-9b4a-e77755c66249%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

May be you are doing a lot of updates (using same IDs)?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 7 août 2014 à 05:52, vjbangis jessviray0708@gmail.com a écrit :

Hello guys,

Could you help me why "docs.count" below is not increasing? it's stack at 2307764. while the "docs.deleted" keeps increasing.

i'm just running a php script to ingest the csv source data to ES.

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 412834 1.2gb 638mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 421090 1.2gb 637.8mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 434017 1.2gb 638.5mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 437557 1.2gb 632.6mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 451098 1.2gb 642.4mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 514957 1.3gb 670.1mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size pri.store.size

green kibana-int 5 1 1 0 25.6kb 12.8kb

green basic-info 5 1 2307764 882639 1.4gb 734.7mb

It's running 4 nodes of c3.large. ES-1.2.3 Is there anything I need to add in the elasticsearch.yml below ?

cluster.name: clustername

node.name: “machine00"

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: [“machine01","machine02","machine03”]

discovery.zen.minimum_master_nodes: 3

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9b056741-69f8-4288-9b4a-e77755c66249%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8F2028BA-BB33-4AC7-9490-AFF77940D056%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

3

I see, so it's normal.

It took 9 1/2 hours to update almost 19 million documents. Is that normal
or okay? The documents contains 12 fields.

BTW, may you help me how could I address below? the nodes are c3.large EC2
instance. Do I have to change the instance type to m3.large or m3.xlarge,
to increase the RAM? Could you advise?

TIA!

https://lh4.googleusercontent.com/-clguHcIwV1w/U-QkFe1eGcI/AAAAAAAAABE/hr67kRrMybk/s1600/es-high-search-query.png

On Thursday, August 7, 2014 2:07:32 PM UTC+8, David Pilato wrote:

May be you are doing a lot of updates (using same IDs)?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 7 août 2014 à 05:52, vjbangis <jessvi...@gmail.com <javascript:>> a
écrit :

Hello guys,

Could you help me why "docs.count" below is not increasing? it's stack at
2307764. while the "docs.deleted" keeps increasing.

i'm just running a php script to ingest the csv source data to ES.

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size
pri.store.size

green kibana-int 5 1 1 0 25.6kb
12.8kb

green basic-info 5 1 2307764 412834 1.2gb
638mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size
pri.store.size

green kibana-int 5 1 1 0 25.6kb
12.8kb

green basic-info 5 1 2307764 421090 1.2gb
637.8mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size
pri.store.size

green kibana-int 5 1 1 0 25.6kb
12.8kb

green basic-info 5 1 2307764 434017 1.2gb
638.5mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size
pri.store.size

green kibana-int 5 1 1 0 25.6kb
12.8kb

green basic-info 5 1 2307764 437557 1.2gb
632.6mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size
pri.store.size

green kibana-int 5 1 1 0 25.6kb
12.8kb

green basic-info 5 1 2307764 451098 1.2gb
642.4mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size
pri.store.size

green kibana-int 5 1 1 0 25.6kb
12.8kb

green basic-info 5 1 2307764 514957 1.3gb
670.1mb

[login@machine elasticsearch]$ curl 'localhost:9200/_cat/indices?v'

health index pri rep docs.count docs.deleted store.size
pri.store.size

green kibana-int 5 1 1 0 25.6kb
12.8kb

green basic-info 5 1 2307764 882639 1.4gb
734.7mb

It's running 4 nodes of c3.large. ES-1.2.3 Is there anything I need to add
in the elasticsearch.yml below ?

cluster.name: clustername

node.name: “machine00"

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: [“machine01","machine02","machine03”]

discovery.zen.minimum_master_nodes: 3

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9b056741-69f8-4288-9b4a-e77755c66249%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9b056741-69f8-4288-9b4a-e77755c66249%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0ec73b88-7a05-433a-9ad5-43059fef2532%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.