# How to get unique docment from elastic db from start time and end time?

**URL:** <https://discuss.elastic.co/t/how-to-get-unique-docment-from-elastic-db-from-start-time-and-end-time/141076>\
**Category:** Elasticsearch\
**Created:** [July 23, 2018, 5:15am UTC](https://discuss.elastic.co/t/how-to-get-unique-docment-from-elastic-db-from-start-time-and-end-time/141076 "2018-07-23T05:15:06Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![RameshNagargoje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshnagargoje/32/30212_2.png) [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\
**Post date:** [July 23, 2018, 5:15am UTC](https://discuss.elastic.co/t/how-to-get-unique-docment-from-elastic-db-from-start-time-and-end-time/141076/1 "2018-07-23T05:15:06Z")

</div>

Here is my python snippet query to get documents from ES

```
    res = es.search(index="test-data", timeout=500000, body={"query":
                    {"bool":
                        {"must": [
                                  {"range": {"timestamp": {
                                      "gte": startTime,
                                      "lte": 'now'
                    }}}
            ]}},
        "sort": [{"timestamp": {"order": "asc"}}]
    })

```

I want to get all unique documents from start time and end time please help me.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 23, 2018, 7:14am UTC](https://discuss.elastic.co/t/how-to-get-unique-docment-from-elastic-db-from-start-time-and-end-time/141076/2 "2018-07-23T07:14:58Z")

</div>

Not sure what **unique documents** means here.

Anyway to extract all the documents that match a query, you should use the scroll API if you will get more than 10000 results.

---

<div class="post-metadata">

**Author:** ![RameshNagargoje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshnagargoje/32/30212_2.png) [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\
**Post date:** [July 23, 2018, 9:08am UTC](https://discuss.elastic.co/t/how-to-get-unique-docment-from-elastic-db-from-start-time-and-end-time/141076/3 "2018-07-23T09:08:06Z")

</div>

```
{  
  "highlight":{  
    "pre_tags":[  
      "@kibana-highlighted-field@"
    ],
    "post_tags":[  
      "@/kibana-highlighted-field@"
    ],
    "fields":{  
      "*":{  

      }
    },
    "fragment_size":2147483647
  },
  "query":{  
    "filtered":{  
      "query":{  
        "query_string":{  
          "query":"*",
          "analyze_wildcard":"true",
          "allow_leading_wildcard":"false"
        }
      },
      "filter":{  
        "bool":{  
          "must":[  
            {  
              "query":{  
                "match":{  
                  "boardName":{  
                    "query":"boardName",
                    "type":"phrase"
                  }
                }
              }
            },
            {  
              "query":{  
                "match":{  
                  "tag":{  
                    "query":"tagName",
                    "type":"phrase",
                    "analyzer":"english"
                  }
                }
              }
            },
            {  
              "query":{  
                "match":{  
                  "module":{  
                    "query":"module",
                    "type":"phrase"
                  }
                }
              }
            },
            {  
              "query":{  
                "match":{  
                  "os":{  
                    "query":"os",
                    "type":"phrase"
                  }
                }
              }
            },
            {  
              "range":{  
                "timestamp":{  
                  "gte":"startTime",
                  "lte":"now"
                }
              }
            }
          ],
          "must_not":[  

          ]
        }
      }
    }
  },
  "fields":[  
    "*",
    "_source"
  ],
  "script_fields":{  
    "origin":{  
      "script":"0",
      "lang":"expression"
    },
    "_count":{  
      "script":"1",
      "lang":"expression"
    }
  },
  "fielddata_fields":[  
    "timestamp"`Preformatted text`
  ]
}

@dadoonet i want to get unique records of boardName, tagName, module, os for each combination of these.`Preformatted text`
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 23, 2018, 9:18am UTC](https://discuss.elastic.co/t/how-to-get-unique-docment-from-elastic-db-from-start-time-and-end-time/141076/4 "2018-07-23T09:18:19Z")

</div>

Please format your code, logs or configuration files using `</>` icon as explained in [this guide](https://discuss.elastic.co/t/about-the-elasticsearch-category/21) and not the citation button. It will make your post more readable.

Or use markdown style like:

````
```
CODE
```

````

This is the icon to use if you are not using markdown format:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7e6e239431ec2d71cbf1beef741f2e93e7cc762c.jpg)

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.  
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.  
Please update your post.

---

<div class="post-metadata">

**Author:** ![RameshNagargoje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshnagargoje/32/30212_2.png) [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\
**Post date:** [July 23, 2018, 10:11am UTC](https://discuss.elastic.co/t/how-to-get-unique-docment-from-elastic-db-from-start-time-and-end-time/141076/5 "2018-07-23T10:11:48Z")

</div>

@dadoonet edited post please check.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 23, 2018, 11:00am UTC](https://discuss.elastic.co/t/how-to-get-unique-docment-from-elastic-db-from-start-time-and-end-time/141076/6 "2018-07-23T11:00:28Z")

</div>

I'd probably use some [terms aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html) on fields `boardName`, `tagName`, `module` and `os`.

---

<div class="post-metadata">

**Author:** ![RameshNagargoje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshnagargoje/32/30212_2.png) [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\
**Post date:** [July 24, 2018, 5:33am UTC](https://discuss.elastic.co/t/how-to-get-unique-docment-from-elastic-db-from-start-time-and-end-time/141076/7 "2018-07-24T05:33:57Z")

</div>

@dadoonet

```
Tried to fetch it worked 
        GET test_index/_search
        {
            "aggs" : {
                "genres" : {
                    "terms" : { "field" : "boardName.keyword" }
                }
            }
        } 

But when i used other fields it shows duplicate key error

GET bat/_search
{
"aggs" : {
    "genres" : {
        "terms" : { "field" : "boardName.keyword" },
        "terms" : { "field" : "os.keyword" },
        "terms" : { "field" : "tagName.keyword" },
        "terms" : { "field" : "module.keyword" }
    }
}
}
```

---

<div class="post-metadata">

**Author:** ![RameshNagargoje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshnagargoje/32/30212_2.png) [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\
**Post date:** [July 24, 2018, 5:48am UTC](https://discuss.elastic.co/t/how-to-get-unique-docment-from-elastic-db-from-start-time-and-end-time/141076/8 "2018-07-24T05:48:03Z")

</div>

```
 Sorry, I was wrong,
it Solves the duplicate key problem.
    "aggs" : {
        "board" : {
            "terms" : { "field" : "boardName.keyword" }
            
        },
        "tag" : {
            "terms" : { "field" : "tagName.keyword" }
            
        },
        "os" : {
            "terms" : { "field" : "OS.keyword" }
            
        },
        
        "module" : {
            "terms" : { "field" : "module.keyword" }
            
        }
    }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 21, 2018, 5:48am UTC](https://discuss.elastic.co/t/how-to-get-unique-docment-from-elastic-db-from-start-time-and-end-time/141076/9 "2018-08-21T05:48:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.