Strings that can be interpreted as Date cause trouble even if mapped as text

Elastic Stack Elasticsearch
1

Hello there,

we have an application which allows our customer to upload and index almost any data. We seem to have an issue, if we have a property whose type is "Text", but the input data can be interpreted as a datetime.

First of all, let me show you a part of our mapping, which was received via Kibana Get index/_mappings:

 "fi@Date": {
   "type": "text",
   "fields": {
     "keyword": {
         "type": "keyword"
    }}}

fi@Date is our property and its type is text. Therefore any string can be used! Our index currently has 2 entries, which look like this (once again partially reduced):

    "_index": "cxcrgygrgemj9cjaofexlq_v1",
    "_source": {
      "filename": "DemoFile01",
      "fi@Date": "2018-10-31T08:25:52",

Second entry:

       "_index": "cxcrgygrgemj9cjaofexlq_v1",
        "_source": {
          "filename": "DemoFile02",
          "fi@Date": "Abc8-10-31T08:25:52"

Note that the first entry has "fi@Date": "2018-10-31T08:25:52",. The right hand side is a datetime representation, but the property fi@Date is defined as a text.

This causes currently two problems:

  1. Using the NEST high level client, we create a request of type

    SearchRequest<Dictionary<string, object>>
    The key of the result dictionary is the name of the property (e.g. "fi@Date") and the value its according value. The NEST client converts this value to DateTime. So we lose some information of the original string (the T is lost). However, if we perform a text search on this property, the correct string is vital. E.g. a TermsQuery does only hit, if we provide the exact string "2018-10-31T08:25:52".
    The second entry "Abc8-10-31T08:25:52" is returned as a normal string and does not yield any problems. The NEST client seems to convert entries based on the result, not on the defined mapping. Is there a way to circumvent this behaviour?

  2. Using the following query in Kibana causes an exception. The same happens with a QueryStringQuery in the NEST Client.

    GET cxcrgygrgemj9cjaofexlq/_search
{
"query": {
 "query_string" : {
     "query": "2018-10-31T08:25:52",
     "default_field": "fi@Date",
     "default_operator": "or"
    }
  }
}

It gives
"reason": "Failed to parse query [2018-10-31T08:25:52]",
"reason": "Cannot parse '2018-10-31T08:25:52': Encountered " ":" ": "" at line 1, column 16.\r\nWas expecting one of:\r\n \r\n ...\r\n ...\r\n ...\r\n "+" ...\r\n "-" ...\r\n ...\r\n "(" ...\r\n "" ...\r\n "^" ...\r\n ...\r\n ...\r\n <FUZZY_SLOP> ...\r\n ...\r\n ...\r\n ...\r\n "[" ...\r\n "{" ...\r\n ...\r\n ",
"caused_by": {
"type": "parse_exception",
"reason": "Encountered " ":" ": "" at line 1, column 16.\r\nWas expecting one of:\r\n \r\n ...\r\n ...\r\n ...\r\n "+" ...\r\n "-" ...\r\n ...\r\n "(" ...\r\n "" ...\r\n "^" ...\r\n ...\r\n ...\r\n <FUZZY_SLOP> ...\r\n ...\r\n ...\r\n ...\r\n "[" ...\r\n "{" ...\r\n ...\r\n "
}
Edit: There is no error, if we change slightly change the query, so it does no longer match a datetime e.g. "2018-10-31T08:2552" (removed colon). /Edit

Any ideas? Is this an ElasticSearch Bug? Can I do something to prevent both behaviours?

Version Info
NEST 5.6.5
ElasticSearch.Net 5.6.5
Elastic Search on Server 5.5

Kind regards,

Jörg

2

I was able to narrow the first part down (automapping to DateTime object):
It is default Json Behaviour to map any DateTime-ish string to a DateTime object.
(Discussion around this can be found here: https://github.com/JamesNK/Newtonsoft.Json/issues/862).

In Json this can be fixed via Serializersettings. In Nest this would look like:

 mEsClientConnectionSettings = new ConnectionSettings(connectionPool, connection, new SerializerFactory((settings, values) =>
{
  settings.NullValueHandling = NullValueHandling.Ignore;
  settings.TypeNameHandling = TypeNameHandling.None;
  settings.DateParseHandling = DateParseHandling.None;
}));

However this does not help me. The Client still deserializes my string into a datetime object. I checked the specific client via Debugging and its serializersettings does have the DateParseHandling = DateParseHandling.None; set.

It does work in a small testprogram with NewtonSoft.Json:

JsonSerializerSettings settings = new JsonSerializerSettings
  {
    DateParseHandling = DateParseHandling.None
  };
  var obj = JsonConvert.DeserializeObject<Dictionary<string, Object>>("{\"MyKey\" : \"2018-10-31T08:25:52\"}", settings);

Any ideas how to fix this in Nest?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.