Composite date aggregation not consistent

[morlokdk (Dusan Kmet)]

Hi, i using elastic 6.3.2 and have issue with date_aggregation and timezones

This is me aggregation

"aggs": {
  "histograms2": {
    "date_histogram": {
      "field": "createdAt",
      "interval": "hour",
      "format": "YYYY_MM_dd:HH",
      "time_zone": "+03:00"
    }
  },
  "histograms": {
    "composite": {
      "size": 1000,
      "sources": [
        {
          "agg_0": {
            "date_histogram": {
              "field": "createdAt",
              "interval": "hour",
              "format": "YYYY_MM_dd:HH",
              "time_zone": "+03:00"
            }
          }
        }
      ]
    }
  }
}

and result is

"aggregations": {
  "histograms": {
    "after_key": {
      "agg_0": "2019_03_13:11"
    },
    "buckets": [
      {
        "key": {
          "agg_0": "2019_03_13:10"
        },
        "doc_count": 7
      },
      {
        "key": {
          "agg_0": "2019_03_13:11"
        },
        "doc_count": 1
      }
    ]
  },
  "histograms2": {
    "buckets": [
      {
        "key_as_string": "2019_03_13:13",
        "key": 1552471200000,
        "doc_count": 7
      },
      {
        "key_as_string": "2019_03_13:14",
        "key": 1552474800000,
        "doc_count": 1
      }
    ]
  }
}

As you can see "string" keys are different. Me objects have dates "2019-03-13T10:23:30.992Z" until "2019-03-13T11:23:30.992Z".

Why keys are not same?

UPDATE: if i change aggs to daily aggregation result is also different

"aggregations": {
  "histograms": {
    "after_key": {
      "agg_0": "2019_03_12"
    },
    "buckets": [
      {
        "key": {
          "agg_0": "2019_03_12"
        },
        "doc_count": 8
      }
    ]
  },
  "histograms2": {
    "buckets": [
      {
        "key_as_string": "2019_03_13",
        "key": 1552424400000,
        "doc_count": 8
      }
    ]
  }
}

Date key 2019_03_13 seems to be correct, but not 2019_03_12

[clement_tourriere]

Hello,

it seems indeed to be a bug with the format parameter in date_histogram from composite aggregation that performs a wrong rounding of the date.

The full reproduction:

# Create index
PUT test_date_histo
{
  "settings": {
    "number_of_shards": 5
  },
  "mappings": {
    "_doc": {
      "properties": {
        "date": {
          "type": "date"
        }
      }
    }
  }
}

# Index some data
POST test_date_histo/_doc
{
  "date": "2019-03-01T00:01:00+00:00"
}

# Perform a comparative query
GET test_date_histo/_search
{
  "size": 0,
  "aggs": {
    "date_histo": {
      "date_histogram": {
        "field": "date",
        "interval": "day",
        "time_zone": "+02:00",
        "format": "yyyy-MM-dd"
      }
    },
    "composite_date_histo": {
      "composite": {
        "sources": [
          {
            "date_histo": {
              "date_histogram": {
                "field": "date",
                "interval": "day",
                "time_zone": "+02:00",
                "format": "yyyy-MM-dd"
              }
            }
          }
        ]
      }
    }
  }
}

# Without format parameter in composite, epoch date is the same between date_histogram and composite date_histogram
GET test_date_histo/_search
{
  "size": 0,
  "aggs": {
    "date_histo": {
      "date_histogram": {
        "field": "date",
        "interval": "day",
        "time_zone": "+02:00",
        "format": "yyyy-MM-dd"
      }
    },
    "composite_date_histo": {
      "composite": {
        "sources": [
          {
            "date_histo": {
              "date_histogram": {
                "field": "date",
                "interval": "day",
                "time_zone": "+02:00"
              }
            }
          }
        ]
      }
    }
  }
}