Retrieving all the unique events

[23141]

Hey,

I have an field in my documents called eventType and I want to get all the unique event types over a given time range. How can I achieve this?

curl -X GET "http://server:9200/_all/_search" -H 'Content-Type: application/json' -d'
{
  "size": 10000,
  "_source": ["level", "eventType"],  
  "aggs" : {
    "uniq_events" : {
        "terms" : { "field" : "eventType.keyword" }
    }
  },
  "query": {
    "match": { 
      "level": "WARN"
    }
  }
}
'

The above is what I have written but I can't filter by timestamp and size 10000 doesn't bring back all the unqiue records. Please help

[gbrown (Gordon Brown)]

As for the filter by timestamp, you should be able to do that with a range query.

Now, the answer to the other part of your question, it depends on exactly what you're trying to retrieve. As written, you query will return 10,000 documents with level: WARN, as well as the 10 keywords which occur most often across all documents which have level: WARN, which I suspect is not what you want.

Without knowing more exactly what you're trying to do, it's hard to make a specific recommendation, but if you want to retrieve all unique values, the Composite aggregation may be useful if eventType.keyword has a large number of unique values.

[23141]

Hi @gbrown - thanks so much for trying to help.

I'm trying to get all the events ("eventType") for the past week. I played around with the sizes and realized the below is what I want.

curl -X GET "http://server:9200/_all/_search" -H 'Content-Type: application/json' -d'
{
  "size": 0,
  "_source": ["level", "eventType"],  
  "aggs" : {
    "uniq_events" : {
        "terms" : { "field" : "eventType.keyword" },
        "size": 1000
    }
  },
  "query": {
    "match": { 
      "level": "WARN"
    }
  }
}
'

As you mentioned

As for the filter by timestamp, you should be able to do that with a range query.

I'm not sure how to do a filtered search while still want to match on warn

"query": {
    "match": { 
      "level": "WARN"
    }
  }

[gbrown (Gordon Brown)]

Good to hear you figured out the aggregation you need! To combine the two queries, you'll need to use a Bool Query. In your case, I think your query will end up looking something like this:

"query": {
  "bool": {
    "filter": [
      {"match": {"level": "WARN"}},
      {"range": {"timestamp": {"gte": "2018/1/1", "lte": "2018/1/31"}}}
    ]
  }
}

Although you may need to adjust the date format to match the format in your mapping.

[23141]

me <3 @gbrown
Thanks for your help