# Duplicates when paging
**URL:** https://discuss.elastic.co/t/duplicates-when-paging/14101
**Category:** Elasticsearch
**Created:** [October 24, 2013, 9:20pm UTC](https://discuss.elastic.co/t/duplicates-when-paging/14101 "2013-10-24T21:20:58Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Juan](https://discuss.elastic.co/u/Juan)
#### Post date: [October 24, 2013, 9:20pm UTC](https://discuss.elastic.co/t/duplicates-when-paging/14101/1 "2013-10-24T21:20:58Z")
Hi,
I am using ElasticSearch version 0.90.2 and when using the paging
functionality I manage to get duplicates returned. The issue is that on our
application we use the total records found, but when we start paging the
duplicates documents are removed from the U.I. for obvious reasons. Thus
not matching the total. I had a look at the scroll search type but is not
really an option because it does not support as far as I understand,
sorting scoring and faceting. We have a 2 node cluster with 5 shards and 5
replicas each.
Here is an example of my mapping.
{
"OrganizationProfile" : {
"properties" : {
"Campaigns" : {
"properties" : {
"AddedAt" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"Name" : {
"type" : "string"
}
}
},
"CandidateType" : {
"type" : "long"
},
"Fired" : {
"type" : "boolean"
},
"FirstJoined" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"Flags" : {
"type" : "string"
},
"JoinCampaign" : {
"type" : "string"
},
"LastJoined" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"LastUpdated" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"NoCall" : {
"type" : "boolean"
},
"Notes" : {
"properties" : {
"Author" : {
"type" : "string"
},
"AuthorId" : {
"type" : "string"
},
"Content" : {
"type" : "string"
},
"Created" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"NoteType" : {
"type" : "long"
},
"\_id" : {
"type" : "string"
}
}
},
"OrganizationId" : {
"type" : "long"
},
"Profile" : {
"properties" : {
"ExistingProfileId" : {
"type" : "long"
},
"PersonalDetail" : {
"properties" : {
"Email" : {
"type" : "string"
},
"FirstName" : {
"type" : "string"
},
"HighestEducationLevel" : {
"type" : "long"
},
"LastName" : {
"type" : "string"
},
"Location" : {
"properties" : {
"Address" : {
"type" : "string"
},
"City" : {
"type" : "string"
},
"Coordinates" : {
"type" : "geo\_point"
},
"Country" : {
"type" : "string"
},
"PostCode" : {
"type" : "string"
},
"State" : {
"type" : "string"
}
}
},
"Phone" : {
"type" : "string"
}
}
},
"PhotoUrl140" : {
"type" : "string"
},
"PhotoUrl50" : {
"type" : "string"
},
"PhotoUrl80" : {
"type" : "string"
},
"ProfessionalDetail" : {
"properties" : {
"Education" : {
"properties" : {
"CourseMajor" : {
"type" : "string"
},
"EndDate" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"Level" : {
"type" : "long"
},
"OrganizationName" : {
"type" : "string"
},
"StartDate" : {
"type" : "date",
"format" : "dateOptionalTime"
}
}
},
"Interests" : {
"properties" : {
"Title" : {
"type" : "string"
},
"\_id" : {
"type" : "long"
}
}
},
"WorkHistories" : {
"properties" : {
"Employer" : {
"type" : "string"
},
"EndDate" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"JobTitle" : {
"type" : "string"
},
"StartDate" : {
"type" : "date",
"format" : "dateOptionalTime"
}
}
}
}
},
"ShortCode" : {
"type" : "string"
},
"Version" : {
"type" : "long"
},
"\_id" : {
"type" : "string"
}
}
},
"RecruiterCampaigns" : {
"properties" : {
"AddedAt" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"Name" : {
"type" : "string"
}
}
},
"ReferringUrl" : {
"type" : "string"
},
"Source" : {
"type" : "string"
},
"Student" : {
"type" : "boolean"
},
"Tags" : {
"type" : "string"
},
"Unsubscribed" : {
"type" : "boolean"
},
"Version" : {
"type" : "long"
},
"Veteran" : {
"type" : "boolean"
},
"id" : {
"type" : "string"
},
"organizationId" : {
"type" : "long"
}
}
}
}
And here is an example of the query.
{
"from": 0,
"size": 20,
"query": {
"filtered": {
"query": {
"query\_string": {
"query": "mcghee\*",
"fields": [
"JoinCampaign^1",
"Campaigns.Name^2",
"RecruiterCampaigns.Name^2",
"Notes.Content^2",
"Tags^2",
"Profile.ProfessionalDetail.WorkHistories.Employer^3",
"Profile.ProfessionalDetail.WorkHistories.JobTitle^3",
"Profile.PersonalDetail.Location.City^4",
"Profile.PersonalDetail.Location.Country^4",
"Profile.PersonalDetail.Location.PostCode^3",
"Profile.PersonalDetail.Location.State^4",
"Profile.PersonalDetail.Phone^1",
"Profile.PersonalDetail.FirstName^5",
"Profile.PersonalDetail.LastName^5",
"Profile.PersonalDetail.Email^7"
],
"default\_operator": "and"
}
},
"filter": {
"term": {
"OrganizationId": "23"
}
}
}
},
"fields": [
"Profile",
"Tags",
"Campaigns",
"RecruiterCampaigns",
"CandidateType",
"Unsubscribed",
"Student",
"Fired",
"NoCall",
"Veteran",
"Notes",
"\_id",
"OrganizationId"
]
}
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [October 25, 2013, 7:44am UTC](https://discuss.elastic.co/t/duplicates-when-paging/14101/2 "2013-10-25T07:44:58Z")
Hey,
not sure, if I got your question right? Do you have this problem while
indexing and searching (which means new data is added, changing the search
results), or while search only? If the latter, you should take a look at
the preference setting, which could be your users sessionid from the web
application. See
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
--Alex
On Thu, Oct 24, 2013 at 11:20 PM, Juan Herbst [jjherbst@gmail.com](mailto:jjherbst@gmail.com) wrote:
> Hi,
>
> I am using Elasticsearch version 0.90.2 and when using the paging
> functionality I manage to get duplicates returned. The issue is that on our
> application we use the total records found, but when we start paging the
> duplicates documents are removed from the U.I. for obvious reasons. Thus
> not matching the total. I had a look at the scroll search type but is not
> really an option because it does not support as far as I understand,
> sorting scoring and faceting. We have a 2 node cluster with 5 shards and 5
> replicas each.
>
> Here is an example of my mapping.
>
> {
> "OrganizationProfile" : {
> "properties" : {
> "Campaigns" : {
> "properties" : {
> "AddedAt" : {
> "type" : "date",
> "format" : "dateOptionalTime"
> },
> "Name" : {
> "type" : "string"
> }
> }
> },
> "CandidateType" : {
> "type" : "long"
> },
> "Fired" : {
> "type" : "boolean"
> },
> "FirstJoined" : {
> "type" : "date",
> "format" : "dateOptionalTime"
> },
> "Flags" : {
> "type" : "string"
> },
> "JoinCampaign" : {
> "type" : "string"
> },
> "LastJoined" : {
> "type" : "date",
> "format" : "dateOptionalTime"
> },
> "LastUpdated" : {
> "type" : "date",
> "format" : "dateOptionalTime"
> },
> "NoCall" : {
> "type" : "boolean"
> },
> "Notes" : {
> "properties" : {
> "Author" : {
> "type" : "string"
> },
> "AuthorId" : {
> "type" : "string"
> },
> "Content" : {
> "type" : "string"
> },
> "Created" : {
> "type" : "date",
> "format" : "dateOptionalTime"
> },
> "NoteType" : {
> "type" : "long"
> },
> "\_id" : {
> "type" : "string"
> }
> }
> },
> "OrganizationId" : {
> "type" : "long"
> },
> "Profile" : {
> "properties" : {
> "ExistingProfileId" : {
> "type" : "long"
> },
> "PersonalDetail" : {
> "properties" : {
> "Email" : {
> "type" : "string"
> },
> "FirstName" : {
> "type" : "string"
> },
> "HighestEducationLevel" : {
> "type" : "long"
> },
> "LastName" : {
> "type" : "string"
> },
> "Location" : {
> "properties" : {
> "Address" : {
> "type" : "string"
> },
> "City" : {
> "type" : "string"
> },
> "Coordinates" : {
> "type" : "geo\_point"
> },
> "Country" : {
> "type" : "string"
> },
> "PostCode" : {
> "type" : "string"
> },
> "State" : {
> "type" : "string"
> }
> }
> },
> "Phone" : {
> "type" : "string"
> }
> }
> },
> "PhotoUrl140" : {
> "type" : "string"
> },
> "PhotoUrl50" : {
> "type" : "string"
> },
> "PhotoUrl80" : {
> "type" : "string"
> },
> "ProfessionalDetail" : {
> "properties" : {
> "Education" : {
> "properties" : {
> "CourseMajor" : {
> "type" : "string"
> },
> "EndDate" : {
> "type" : "date",
> "format" : "dateOptionalTime"
> },
> "Level" : {
> "type" : "long"
> },
> "OrganizationName" : {
> "type" : "string"
> },
> "StartDate" : {
> "type" : "date",
> "format" : "dateOptionalTime"
> }
> }
> },
> "Interests" : {
> "properties" : {
> "Title" : {
> "type" : "string"
> },
> "\_id" : {
> "type" : "long"
> }
> }
> },
> "WorkHistories" : {
> "properties" : {
> "Employer" : {
> "type" : "string"
> },
> "EndDate" : {
> "type" : "date",
> "format" : "dateOptionalTime"
> },
> "JobTitle" : {
> "type" : "string"
> },
> "StartDate" : {
> "type" : "date",
> "format" : "dateOptionalTime"
> }
> }
> }
> }
> },
> "ShortCode" : {
> "type" : "string"
> },
> "Version" : {
> "type" : "long"
> },
> "\_id" : {
> "type" : "string"
> }
> }
> },
> "RecruiterCampaigns" : {
> "properties" : {
> "AddedAt" : {
> "type" : "date",
> "format" : "dateOptionalTime"
> },
> "Name" : {
> "type" : "string"
> }
> }
> },
> "ReferringUrl" : {
> "type" : "string"
> },
> "Source" : {
> "type" : "string"
> },
> "Student" : {
> "type" : "boolean"
> },
> "Tags" : {
> "type" : "string"
> },
> "Unsubscribed" : {
> "type" : "boolean"
> },
> "Version" : {
> "type" : "long"
> },
> "Veteran" : {
> "type" : "boolean"
> },
> "id" : {
> "type" : "string"
> },
> "organizationId" : {
> "type" : "long"
> }
> }
> }
> }
>
> And here is an example of the query.
>
> {
> "from": 0,
> "size": 20,
> "query": {
> "filtered": {
> "query": {
> "query\_string": {
> "query": "mcghee\*",
> "fields": [
> "JoinCampaign^1",
> "Campaigns.Name^2",
> "RecruiterCampaigns.Name^2",
> "Notes.Content^2",
> "Tags^2",
> "Profile.ProfessionalDetail.WorkHistories.Employer^3",
> "Profile.ProfessionalDetail.WorkHistories.JobTitle^3",
> "Profile.PersonalDetail.Location.City^4",
> "Profile.PersonalDetail.Location.Country^4",
> "Profile.PersonalDetail.Location.PostCode^3",
> "Profile.PersonalDetail.Location.State^4",
> "Profile.PersonalDetail.Phone^1",
> "Profile.PersonalDetail.FirstName^5",
> "Profile.PersonalDetail.LastName^5",
> "Profile.PersonalDetail.Email^7"
> ],
> "default\_operator": "and"
> }
> },
> "filter": {
> "term": {
> "OrganizationId": "23"
> }
> }
> }
> },
> "fields": [
> "Profile",
> "Tags",
> "Campaigns",
> "RecruiterCampaigns",
> "CandidateType",
> "Unsubscribed",
> "Student",
> "Fired",
> "NoCall",
> "Veteran",
> "Notes",
> "\_id",
> "OrganizationId"
> ]
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Juan](https://discuss.elastic.co/u/Juan)
#### Post date: [October 28, 2013, 7:46pm UTC](https://discuss.elastic.co/t/duplicates-when-paging/14101/3 "2013-10-28T19:46:34Z")
Thank you Alex, I will give it a try. FYI it even happens while not
indexing.
Juan
On Friday, 25 October 2013 20:44:58 UTC+13, Alexander Reelsen wrote:
> Hey,
>
> not sure, if I got your question right? Do you have this problem while
> indexing and searching (which means new data is added, changing the search
> results), or while search only? If the latter, you should take a look at
> the preference setting, which could be your users sessionid from the web
> application. See
>
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-preference.html)
>
> --Alex
>
> On Thu, Oct 24, 2013 at 11:20 PM, Juan Herbst \<[jjhe...@gmail.com](mailto:jjhe...@gmail.com)\
>
> > wrote:
>
> > Hi,
> >
> > I am using Elasticsearch version 0.90.2 and when using the paging
> > functionality I manage to get duplicates returned. The issue is that on our
> > application we use the total records found, but when we start paging the
> > duplicates documents are removed from the U.I. for obvious reasons. Thus
> > not matching the total. I had a look at the scroll search type but is not
> > really an option because it does not support as far as I understand,
> > sorting scoring and faceting. We have a 2 node cluster with 5 shards and 5
> > replicas each.
> >
> > Here is an example of my mapping.
> >
> > {
> > "OrganizationProfile" : {
> > "properties" : {
> > "Campaigns" : {
> > "properties" : {
> > "AddedAt" : {
> > "type" : "date",
> > "format" : "dateOptionalTime"
> > },
> > "Name" : {
> > "type" : "string"
> > }
> > }
> > },
> > "CandidateType" : {
> > "type" : "long"
> > },
> > "Fired" : {
> > "type" : "boolean"
> > },
> > "FirstJoined" : {
> > "type" : "date",
> > "format" : "dateOptionalTime"
> > },
> > "Flags" : {
> > "type" : "string"
> > },
> > "JoinCampaign" : {
> > "type" : "string"
> > },
> > "LastJoined" : {
> > "type" : "date",
> > "format" : "dateOptionalTime"
> > },
> > "LastUpdated" : {
> > "type" : "date",
> > "format" : "dateOptionalTime"
> > },
> > "NoCall" : {
> > "type" : "boolean"
> > },
> > "Notes" : {
> > "properties" : {
> > "Author" : {
> > "type" : "string"
> > },
> > "AuthorId" : {
> > "type" : "string"
> > },
> > "Content" : {
> > "type" : "string"
> > },
> > "Created" : {
> > "type" : "date",
> > "format" : "dateOptionalTime"
> > },
> > "NoteType" : {
> > "type" : "long"
> > },
> > "\_id" : {
> > "type" : "string"
> > }
> > }
> > },
> > "OrganizationId" : {
> > "type" : "long"
> > },
> > "Profile" : {
> > "properties" : {
> > "ExistingProfileId" : {
> > "type" : "long"
> > },
> > "PersonalDetail" : {
> > "properties" : {
> > "Email" : {
> > "type" : "string"
> > },
> > "FirstName" : {
> > "type" : "string"
> > },
> > "HighestEducationLevel" : {
> > "type" : "long"
> > },
> > "LastName" : {
> > "type" : "string"
> > },
> > "Location" : {
> > "properties" : {
> > "Address" : {
> > "type" : "string"
> > },
> > "City" : {
> > "type" : "string"
> > },
> > "Coordinates" : {
> > "type" : "geo\_point"
> > },
> > "Country" : {
> > "type" : "string"
> > },
> > "PostCode" : {
> > "type" : "string"
> > },
> > "State" : {
> > "type" : "string"
> > }
> > }
> > },
> > "Phone" : {
> > "type" : "string"
> > }
> > }
> > },
> > "PhotoUrl140" : {
> > "type" : "string"
> > },
> > "PhotoUrl50" : {
> > "type" : "string"
> > },
> > "PhotoUrl80" : {
> > "type" : "string"
> > },
> > "ProfessionalDetail" : {
> > "properties" : {
> > "Education" : {
> > "properties" : {
> > "CourseMajor" : {
> > "type" : "string"
> > },
> > "EndDate" : {
> > "type" : "date",
> > "format" : "dateOptionalTime"
> > },
> > "Level" : {
> > "type" : "long"
> > },
> > "OrganizationName" : {
> > "type" : "string"
> > },
> > "StartDate" : {
> > "type" : "date",
> > "format" : "dateOptionalTime"
> > }
> > }
> > },
> > "Interests" : {
> > "properties" : {
> > "Title" : {
> > "type" : "string"
> > },
> > "\_id" : {
> > "type" : "long"
> > }
> > }
> > },
> > "WorkHistories" : {
> > "properties" : {
> > "Employer" : {
> > "type" : "string"
> > },
> > "EndDate" : {
> > "type" : "date",
> > "format" : "dateOptionalTime"
> > },
> > "JobTitle" : {
> > "type" : "string"
> > },
> > "StartDate" : {
> > "type" : "date",
> > "format" : "dateOptionalTime"
> > }
> > }
> > }
> > }
> > },
> > "ShortCode" : {
> > "type" : "string"
> > },
> > "Version" : {
> > "type" : "long"
> > },
> > "\_id" : {
> > "type" : "string"
> > }
> > }
> > },
> > "RecruiterCampaigns" : {
> > "properties" : {
> > "AddedAt" : {
> > "type" : "date",
> > "format" : "dateOptionalTime"
> > },
> > "Name" : {
> > "type" : "string"
> > }
> > }
> > },
> > "ReferringUrl" : {
> > "type" : "string"
> > },
> > "Source" : {
> > "type" : "string"
> > },
> > "Student" : {
> > "type" : "boolean"
> > },
> > "Tags" : {
> > "type" : "string"
> > },
> > "Unsubscribed" : {
> > "type" : "boolean"
> > },
> > "Version" : {
> > "type" : "long"
> > },
> > "Veteran" : {
> > "type" : "boolean"
> > },
> > "id" : {
> > "type" : "string"
> > },
> > "organizationId" : {
> > "type" : "long"
> > }
> > }
> > }
> > }
> >
> > And here is an example of the query.
> >
> > {
> > "from": 0,
> > "size": 20,
> > "query": {
> > "filtered": {
> > "query": {
> > "query\_string": {
> > "query": "mcghee\*",
> > "fields": [
> > "JoinCampaign^1",
> > "Campaigns.Name^2",
> > "RecruiterCampaigns.Name^2",
> > "Notes.Content^2",
> > "Tags^2",
> > "Profile.ProfessionalDetail.WorkHistories.Employer^3",
> > "Profile.ProfessionalDetail.WorkHistories.JobTitle^3",
> > "Profile.PersonalDetail.Location.City^4",
> > "Profile.PersonalDetail.Location.Country^4",
> > "Profile.PersonalDetail.Location.PostCode^3",
> > "Profile.PersonalDetail.Location.State^4",
> > "Profile.PersonalDetail.Phone^1",
> > "Profile.PersonalDetail.FirstName^5",
> > "Profile.PersonalDetail.LastName^5",
> > "Profile.PersonalDetail.Email^7"
> > ],
> > "default\_operator": "and"
> > }
> > },
> > "filter": {
> > "term": {
> > "OrganizationId": "23"
> > }
> > }
> > }
> > },
> > "fields": [
> > "Profile",
> > "Tags",
> > "Campaigns",
> > "RecruiterCampaigns",
> > "CandidateType",
> > "Unsubscribed",
> > "Student",
> > "Fired",
> > "NoCall",
> > "Veteran",
> > "Notes",
> > "\_id",
> > "OrganizationId"
> > ]
> > }
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:10am UTC](https://discuss.elastic.co/t/duplicates-when-paging/14101/4 "2017-07-06T02:10:08Z")