Metric count mismatch in production setup 7.9.0

[abhishek30 (Abhishek)]

Hi,

I have a standalone setup of ELK stack version 7.12.1. in windows machine. I have ingested examination data from csv file to kibana using logstash. Using this data I had to show total students count. I did this using metric visualization as shown below.

This gives me the correct count of students. Below is the request for this visualization.

{
  "aggs": {
    "1": {
      "cardinality": {
        "field": "EEID.keyword"
      }
    }
  },
  "size": 0,
  "fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "exam_date",
      "format": "date_time"
    }
  ],
  "script_fields": {
    "Negative Mark": {
      "script": {
        "source": "0.25",
        "lang": "painless"
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "runtime_mappings": {},
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2021-03-11T07:13:49.217Z",
              "lte": "2021-06-09T07:13:49.217Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

Now, I also have same standalone production environment with ELK stack version 7.9.0. in centOS machine. I ingested data to production using logstash 7.12.1. I want to create same use case as in my local setup - total count of students. But I am getting count mismatch when using metric visualization.

The request for above visualization

{
  "aggs": {
    "1": {
      "cardinality": {
        "field": "EEID.keyword"
      }
    }
  },
  "size": 0,
  "stored_fields": [
    "*"
  ],
  "script_fields": {
    "value_field": {
      "script": {
        "source": "1",
        "lang": "painless"
      }
    }
  },
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "exam_date",
      "format": "date_time"
    }
  ],
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2021-03-11T07:30:34.150Z",
              "lte": "2021-06-09T07:30:34.150Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

And when checking the count using data table visualization it gives correct count.

Kindly help.

Thank you,
Regards Abhishek

[flash1293 (Joe Reuter)]

Cardinality isn't 100% precise for performance reasons: Cardinality aggregation | Elasticsearch Guide [7.13] | Elastic

You can use the advanced input to specify the precision_threshold to get better values for your use case.

[abhishek30 (Abhishek)]

Firstly, thank you for such quick response.
So I tried the precision threshold giving such result.

But still count mismatch in metric visualization. Below the snapshot and request for the visualization.

{
  "aggs": {
    "1": {
      "cardinality": {
        "field": "EEID.keyword"
      }
    }
  },
  "size": 0,
  "stored_fields": [
    "*"
  ],
  "script_fields": {
    "value_field": {
      "script": {
        "source": "1",
        "lang": "painless"
      }
    }
  },
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "exam_date",
      "format": "date_time"
    }
  ],
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2021-03-11T08:20:24.150Z",
              "lte": "2021-06-09T08:20:24.151Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

Kindly Help.
Thank you

[flash1293 (Joe Reuter)]

What I meant is the "Advanced > JSON input" in the visualization editor:

You can put JSON in there and it will be merged with the rest of the request

[abhishek30 (Abhishek)]

Thank you for the help.