# Problem finding duplicate documents via kibana? Issue with unique count!

**URL:** <https://discuss.elastic.co/t/problem-finding-duplicate-documents-via-kibana-issue-with-unique-count/128132>\
**Category:** Elasticsearch\
**Created:** [April 16, 2018, 8:34am UTC](https://discuss.elastic.co/t/problem-finding-duplicate-documents-via-kibana-issue-with-unique-count/128132 "2018-04-16T08:34:30Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [April 16, 2018, 8:34am UTC](https://discuss.elastic.co/t/problem-finding-duplicate-documents-via-kibana-issue-with-unique-count/128132/1 "2018-04-16T08:34:30Z")

</div>

Hi,

I am a bit confused:

All our documents are shipped via filebeat. Filebeat allways adds the meta information field `offset` (position in logfile) - so far my understanding.

If I filter for a single logfile, then the unique count of offset field should be identical to the count, as long I have no duplicates. Am i correct?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/e/2eb55ed8945ab759bdc1a0129e29165161132716.png)

But kibana shows different values. Lower number on unique counts means that there must be duplicates - my understanding.

Now I want to search the documents, where an offset appears more often, so I would like to run a term query on offset field and filter / sort for the number of occurrence:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a9e9eabed75e2ebd1b1146e2f8b4a053e8bed2b5.png)

When adding additional json options to the term aggregation

```
{
  "min_doc_count" : 2
}

```

I get no results.

So the result is not what I expected. Can anyone shed some light on it?

Thanks, Andreas

---

<div class="post-metadata">

**Author:** ![Bill\_McConaghy](https://avatars.discourse-cdn.com/v4/letter/b/ed655f/32.png) [@Bill\_McConaghy](https://discuss.elastic.co/u/Bill_McConaghy)\
**Post date:** [April 16, 2018, 1:09pm UTC](https://discuss.elastic.co/t/problem-finding-duplicate-documents-via-kibana-issue-with-unique-count/128132/2 "2018-04-16T13:09:55Z")

</div>

Maybe the first entry from a given logfile has no offset ? That's the only thing I can think of that would account for this.

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [April 17, 2018, 7:58am UTC](https://discuss.elastic.co/t/problem-finding-duplicate-documents-via-kibana-issue-with-unique-count/128132/3 "2018-04-17T07:58:40Z")

</div>

but we have a difference of 19, not of 1.

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [April 17, 2018, 8:11am UTC](https://discuss.elastic.co/t/problem-finding-duplicate-documents-via-kibana-issue-with-unique-count/128132/4 "2018-04-17T08:11:03Z")

</div>

I checked against the input logfile. So the count seems to be correct. So do we have an issue with unique count aggregation in ES / Kibana?

---

<div class="post-metadata">

**Author:** ![Bill\_McConaghy](https://avatars.discourse-cdn.com/v4/letter/b/ed655f/32.png) [@Bill\_McConaghy](https://discuss.elastic.co/u/Bill_McConaghy)\
**Post date:** [April 17, 2018, 11:25am UTC](https://discuss.elastic.co/t/problem-finding-duplicate-documents-via-kibana-issue-with-unique-count/128132/5 "2018-04-17T11:25:39Z")

</div>

The aggregation is done in ES, so if it is an agg issue it would be there. This is puzzling.

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [April 17, 2018, 12:41pm UTC](https://discuss.elastic.co/t/problem-finding-duplicate-documents-via-kibana-issue-with-unique-count/128132/6 "2018-04-17T12:41:17Z")

</div>

moved to elasticsearch  
stack version is 6.2.3

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 15, 2018, 12:41pm UTC](https://discuss.elastic.co/t/problem-finding-duplicate-documents-via-kibana-issue-with-unique-count/128132/7 "2018-05-15T12:41:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.