Kibana dashboard DSL aggregations problem

[harryb]

I am trying to filter an Elastic index to only return one record per doc_id and which I want to base the visualization on. If I use this in a Kibana dashboard visualization filter I get an error: '[size] query malformed, no start_object after query name'.

The filter is:

{
  "size": 500,
  "aggs": {
    "Unique_docs": {
      "terms": {
        "field": "DOC_ID"
      }
    }
  }
}

If I remove the size, the error changes to 'unknown field [aggs]'. The DSL runs in the console but not as a dashboard filter. Can anyone tell me if you can use aggregations in a dashboard filter please?

[flash1293 (Joe Reuter)]

You can't use aggregations in a filter - filter only operate on a per-document basis.

Can you explain which kind of visualization you want to build? Maybe there's another way to achieve your goal.

[harryb]

Thanks Joe - the data has multiple documents with the same doc_id but I want to sum the values for another field in the record called citationCount but summing it only once for each doc_id, and showing it in a bar graph as total citations by year. The citationCount values are the same across documents with the same doc_id.

So I was trying to filter down the index records to get just one per doc_id, then sum the citationCount by year. Is there another way to do that?

[flash1293 (Joe Reuter)]

Not directly in the chart configuration, but it should be possible with transforms. Look at Transform examples | Elasticsearch Guide [7.13] | Elastic , this seems very close to your case.

A transform is basically a job you let run in the background which will pre-aggregate your data in some way (e.g. only keeping the latest document).

Then you can do a regular visualization on top of this pre-aggregated index