Elasticsearch 2.1.2 after update , unexpected result

Elastic Stack Elasticsearch
1 
GET cross_dms_receive_send_diff/cross_dms_receive_send_diff/_search
{
    "query": {
      "bool": {
        "must": [
          {
            "term": {
              "waybill_code": "VD52794532626"
            }
          },
          {
            "range": {
              "inspect_time": {
                "gt": "1556294400000",  //  2019/4/27 0:0:0
                "lt": "1556467199000"   //  2019/4/28 23:59:59
              }
            }
          }
        ]
      }
    },
    "aggs": {
      "inspect_time": {
        "terms": {
          "field": "inspect_time"
        }
      }
    }
  }

 {
  "_shards": {
    "total": 32,
    "failed": 0,
    "successful": 32
  },
  "hits": {
    "hits": [
      {
        "_index": "cross_dms_receive_send_diff",
        "_type": "cross_dms_receive_send_diff",
        "_source": {
          "waybill_code": "VD52794532626",
          "inspect_time": 1553966138000 // 1. 2019/3/31 1:15:38
        },
        "_id": "VD52794532626-1-1-|1086",
        "_score": 17.218697
      }
    ],
    "total": 1,
    "max_score": 17.218697
  },
  "took": 12,
  "timed_out": false,
  "aggregations": {
    "inspect_time": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key_as_string": "2019-03-30T17:15:38.000Z",// 2.
          "doc_count": 1,
          "key": 1553966138000 
        },
        {
          "key_as_string": "2019-04-27T02:51:50.000Z",//3.
          "doc_count": 1,
          "key": 1556333510000
        }
      ]
    }
  }
}

This is the production environment, the test environment cannot be reproduced。
one document after update operation , looks like there are 2 indices .
one for old ,another for new ,but the _source is old .

"_id": "VD52794532626-1-1-|1086"
"key_as_string": "2019-03-30T17:15:38.000Z"// 2. first index  the doc  
"key_as_string": "2019-04-27T02:51:50.000Z",//3.  update...

Many of the data updated during this time are like this(wrong), and after one day , the subsequent update operations are normal. No operation during this period

General speeking ,the result of the aggregation should be a bucket, not the two here.

How to explain this? How to avoid the next occurrence？

The doc field inspect_time mapping:

"inspect_time": {
    "format": "strict_date_optional_time||epoch_millis",
    "type": "date"
}

Thank you for your help。

2

It's not really clear what you are asking or what you problem is sorry. Are you able to provide more information?

3

@warkolm

I added some descriptions, please help me find it again.

GET cross_dms_receive_send_diff/cross_dms_receive_send_diff/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "waybill_code": "VD52794532626"
          }
        },
        {
          "range": {
            "inspect_time": {
              "gt": "1556294400000",    // 2019/4/27 0:0:0
              "lt": "1556467199000"   //  2019/4/28 23:59:59
            }
          }
        }
      ]
    }
  }
}

{
  "_shards": {
    "total": 32,
    "failed": 0,
    "successful": 32
  },
  "hits": {
    "hits": [
      {
        "_index": "cross_dms_receive_send_diff",
        "_type": "cross_dms_receive_send_diff",
        "_source": {
          "waybill_code": "VD52794532626",
          "inspect_time": 1553966138000    //2019/3/31 1:15:38
        },
        "_id": "VD52794532626-1-1-|1086",
        "_score": 17.219076
      }
    ],
    "total": 1,
    "max_score": 17.219076
  },
  "took": 117,
  "timed_out": false
}

Initially found the problem,the result does not match the query criteria。

4

Some comments:

  • Use _doc instead of cross_dms_receive_send_diff type name. Type are going to be removed in the future.
  • Depending on the waybill_code mapping, your term query with VD52794532626 may not match.
  • in the future please provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.

5
  • Use _doc , i know ,but it's old cluster ,so next upgrade will change
  • the problem not about waybill_code , It appears here, just to facilitate filtering out specific data from massive data. the problem is inspect_time , another example maybe help you understand
  • This is an accident in the production environment, it can't be reproduced. If it can be reproduced, I will track the code debugging. I want to know under what circumstances this possibility problem will occur, or after Elasticsearch version 2.1.2 has fixed related bugs.
   .... the query same as previous , omitted here 
    "aggs": {
      "inspect_time": {
        "terms": {
          "field": "inspect_time"
        },
        "aggs": {
          "xx": {
            "top_hits": {
              "size": 2
            }
          }
        }
      }
    }

result:
    "aggregations": {
      "inspect_time": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key_as_string": "2019-04-10T02:40:00.000Z", //key 1  
            "xx": {
              "hits": {
                "hits": [
                  {
                    "_index": "cross_dms_receive_send_diff",
                    "_type": "cross_dms_receive_send_diff",
                    "_source": {
                      "waybill_code": "85640578962",
                      "inspect_time": 1554864000000 //equal the key
                    },
                    "_id": "85640578962-1-1-002|1086",// doc id 1
                    "_score": 17.2106
                  }
                ],
                "total": 1,
                "max_score": 17.2106
              }
            },
            "doc_count": 1,
            "key": 1554864000000  //key 1
          },
          {
            "key_as_string": "2019-04-27T02:14:08.000Z", //key 2 
            "xx": {
              "hits": {
                "hits": [
                  {
                    "_source": {
                      "waybill_code": "85640578962",
                      "inspect_time": 1554864000000 // Note here: not equal the key
                    },
                    "_id": "85640578962-1-1-002|1086", //doc id 2 ,  the same id as last bucket .
                  }
                ],
                "total": 1,
                "max_score": 17.2106
              }
            },
            "doc_count": 1,
            "key": 1556331248000 //key 2
          }
        ]
      }
    }

es
es.jpg1618×1695 250 KB

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.