Why Kibana Dashboard values are different from index query

[Vitor_Nilson]

Hello,

I have a index called kong (the main index), and a Rollup job in this index grouping every 24h which has a index calld rollup_job_kong_gateway.

If we take a look directely in the index Kong and run the following query:

GET kong/_search
{
  "size": 0, 
  "query": {
    
   "bool": {
     
     "must": [
     
    {"range": {
      "@timestamp": {
         "gte": "2023-05-28T00:00:00.000",
        "lte": "2023-05-30T00:00:00.000"
      }
    }}
    
    
     ]
   }
    
    
    
  },
  "aggs": {
    
      "daily_sum": {
      "date_histogram": {
        "field": "@timestamp",
        "calendar_interval": "1d"
      },
      
        
      "aggs": {
        "daily_count": {
          "value_count": {
            "field": "@timestamp"
          }
        }}
      
      
    },
    
    "count_total_requests": {
      "value_count": {
        "field": "@timestamp"
      }
    },
    "count_total_nome_integrador": {
      "cardinality": {
        "field": "nomeIntegrador.keyword"
      }
    }
    
  }
}

This is the result:

Day 2023-05-28 total of 1,428,413 requests
Day 2023-05-29 total of 1,735,944 requests.

And now if we execute the same query (just summing instead of counting) on the rollup index we have the same values:

GET rollup_job_kong_gateway/_search
{
   "size": 0, 
  "query": {
    
   "bool": {
     
     "must": [

    {"range": {
      "@timestamp.date_histogram.timestamp": {
        "gte": "2023-05-28T00:00:00.000",
        "lte": "2023-05-29T00:00:00.000"
      }
    }}
    
    
     ]
   }
    
    
  },
  "aggs": {
    
    "daily_sum": {
      "date_histogram": {
        "field": "@timestamp.date_histogram.timestamp",
        "calendar_interval": "1d"
      },
      
        
      "aggs": {
        "daily_count": {
          "sum": {
            "field": "@timestamp.date_histogram._count"
          }
        }}
      
      
    },
    
    
    
    "count_total_requests": {
          "sum": {
            "field": "@timestamp.date_histogram._count"
          }
    },
    "count_total_nome_integrador": {
          "cardinality": {
            "field": "nomeIntegrador.keyword.terms.value"
          }
        }
        
        
    
  }
}

What I can't understand is: Why Kibana dashboard is showing different values?

This is a dashboard on Kong (main index): (Only day 28)
Why does it now only shows 1,436,142 instead of 1,428,413 that is in its index?

If we take a look, it's just showing the total of records:

And this is the Rollup Job Dashboard: (Only day 28)
It shows the right value

If we take a look, it's just showing the total of records (by summing the count of timestamp):

I'm litte bit confused, if someone could help me I'd feel happy.

[jsanz (Jorge Sanz)]

You can check what exactly queries your dashboard render with the Inspect tool on the top right bar. Have you checked it to compare queries requests and responses with your Console work?

[Vitor_Nilson]

Wow, I found out why it never matchs.

On the dashboard I set the following range:

And this is the request:

GET .....
{
  "aggs": {
    "0": {
      "sum": {
        "field": "@timestamp.date_histogram._count"
      }
    }
  },
  "size": 0,
  "fields": [
    {
      "field": "@timestamp.date_histogram.timestamp",
      "format": "date_time"
    }
  ],
  "script_fields": {},
  "stored_fields": [
    "*"
  ],
  "runtime_mappings": {},
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "range": {
            "@timestamp.date_histogram.timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2023-05-28T03:00:00.000Z",
              "lte": "2023-05-29T03:00:00.000Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

Why is it adding 3 hours to my selected range? Looks like it's getting my current timezone and formatting the date.

[jsanz (Jorge Sanz)]

@Vitor_Nilson

Looks like it's getting my current timezone and formatting the date.

That is standard Kibana behavior, yes. You can change i thoug through the Advanced Settings page on the General section to use a fixed time zone.

Advanced Settings | Kibana Guide [8.8] | Elastic