# Inconsistent sum\_doc\_freq and sum\_ttf numbers in \_mtermvectors
**URL:** https://discuss.elastic.co/t/inconsistent-sum-doc-freq-and-sum-ttf-numbers-in-mtermvectors/128753
**Category:** Elasticsearch
**Created:** [April 19, 2018, 5:25pm UTC](https://discuss.elastic.co/t/inconsistent-sum-doc-freq-and-sum-ttf-numbers-in-mtermvectors/128753 "2018-04-19T17:25:36Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@yluo](https://discuss.elastic.co/u/yluo)
#### Post date: [April 19, 2018, 5:25pm UTC](https://discuss.elastic.co/t/inconsistent-sum-doc-freq-and-sum-ttf-numbers-in-mtermvectors/128753/1 "2018-04-19T17:25:37Z")
Hi there,
We have been using Elasticsearch and ingest-attachment to index program files for full-text searching and keyword extraction (filtered by TFxIDF) using metadata from \_mtermvectors. However, we recently found an inconsistency about \_mtermvectors api.
Suppose we indexed 46 program documents, and use \_mtermvectors then we can get field\_statistics like this:
"field\_statistics": {
"sum\_doc\_freq": 8715,
"doc\_count": 46,
"sum\_ttf": 27672
}
Then I modified only one document, suppose it is called a.java, with only adding a space character and indexed this updated document, and used \_mtermvectors, I got an updated field\_statistics:
"field\_statistics": {
"sum\_doc\_freq": 8995,
"doc\_count": 47,
"sum\_ttf": 30346
}
From above result, we found the doc\_count became to 47 (we only had 46 documents), and sum\_doc\_freq and sum\_ttf have duplicated term counts and seemed that terms in a.java got cloned somehow, with only one more space added.
I tried refreshing my index but I still see the cloned terms field statistics. But if I force index all 46 documents in my index again, then the field\_statistics comes back to the correct numbers again:
"field\_statistics": {
"sum\_doc\_freq": 8715,
"doc\_count": 46,
"sum\_ttf": 27672
}
I am using Elasticsearch 5.6.1 and ingest-attachment 5.6.1 plugin, not sure if this issue is related to this plugin.
Thanks for the helps in advance.
---
### Author:  [@mayya](https://discuss.elastic.co/u/mayya)
#### Post date: [April 27, 2018, 8:11pm UTC](https://discuss.elastic.co/t/inconsistent-sum-doc-freq-and-sum-ttf-numbers-in-mtermvectors/128753/2 "2018-04-27T20:11:37Z")
This is expected behaviour. When you modify a document, you actually delete your previous document and create a new document instead of it. For some time deleted documents are still in memory and counted towards field\_statistics (but searches are smart enough to exclude - this is what refresh is for) . Then when segments are merged into a new segment, deleted documents will be completely wiped out, and a new created Lucene document will not contain them anymore and this is where you will get your correct statistics.
You can [forcefully merge Lucene segments](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-forcemerge.html) in rare cases (not recommended to do often)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 25, 2018, 8:11pm UTC](https://discuss.elastic.co/t/inconsistent-sum-doc-freq-and-sum-ttf-numbers-in-mtermvectors/128753/3 "2018-05-25T20:11:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.