Date field mapping error with type [date] to [long]

Lorenzo_Zuluaga · April 20, 2021, 9:39pm

Hi, I'm currently having a strange issue and I can't find anything related to it. I have an elasticsearch cluster for my production environment with 3 nodes. Recently I created a single-node cluster for my staging environment. My issue is that I created a brand new index to test things on staging, and the data was getting persisted successfully, the issue is that the dynamic mapping its recognizing my "time" field as a long which is a UNIX date on milliseconds format and is working ok on production.
So I tried to add the mapping manually to be recognized as a date:

but in those cases when I try to send data to be persisted I get the following error on the http response:

I tried also using the exact same mappings I was using on my production env and I get the same results.

My production ELK stack is version 7.6 and the staging one I built is 7.12. Also tried with a 7.10 version with the same results. I read about the breaking changes and I didn't found anything that could be causing this.

Thanks in advance!!

warkolm · April 20, 2021, 11:11pm

Welcome to our community!

You cannot change a mapping once an index has been created. You will need to delete the index, create it with the mapping (or use an index template) and then index the data.

Lorenzo_Zuluaga · April 20, 2021, 11:49pm

Thanks for your reply,
Yes sir, that's exactly what I've been doing and that's why is very strange for me, I created a new index on the STAGING cluster which is the single-node one. There I tried to persist a document equal to the ones I'm dealing with on the PRODUCTION enviroment, and that works, but when I try to execute a range query, it gives me the following error:

This is the actual mapping of my data on the PRODUCTION Env:

Even putting on the STAGING Env the exact same mappings I got issues on the range queries or when trying to persist them I got this http response:

result: %{
    "error" => %{
      "reason" => "mapper [time] cannot be changed from type [date] to [long]",
      "root_cause" => [
        %{
          "reason" => "mapper [time] cannot be changed from type [date] to [long]",
          "type" => "illegal_argument_exception"
        }
      ],
      "type" => "illegal_argument_exception" 
    },
    "status" => 400
  }

If I try to manually add the mappings for the new index like this:
1.

PUT /my-new-index/_mappings
{
  "properties" : {
  "time" : {
    "type" : "date"
    }
  }
}

POST /my-new-index/_doc
{
   "time": 1615205294745,
   "name": "Test Name"
}

Doing this I can work with range in queries without issue:

GET /my-new-index*/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "bool": {
            "must": [
              {
                "range": {
                  "time": {
                    "format": "yyyy/MM/dd||yyyy-MM-dd",
                    "gte": "2021-02-19",
                    "lte": "2021-03-21"

                  }
                }
              }
            ]
          }
        }
      ]
    }
  }
}

I have been searching the whole day but I can't find anything that help me, I don't think this is related with any breaking change since I'm able to deal with date fields, but not with my actual mapping structure.

Thanks in advance

warkolm · April 20, 2021, 11:54pm

This all comes down to mappings.

If you are not providing the mapping before ingesting the document, then Elasticsearch will try it's best to match what it things the mapping should be (which is dynamic mapping). That is not always right though, which is why you need to explicitly map things if you want them that way.

Lorenzo_Zuluaga · April 21, 2021, 12:02am

Thank for your help,
I apologize if my english isn't very good and I don't make my self understand as I wish.
I'm providing the mapping before the first documents get ingested, and still getting the same issue.

"reason" => "mapper [time] cannot be changed from type [date] to [long]"

Seems like its trying to handle the field as a long even when the mappings specify it as a date field.

warkolm · April 21, 2021, 12:09am

I ran your examples;

PUT my-new-index
{
  "mappings": {
    "properties": {
      "time": {
        "type": "date"
      }
    }
  }
}

And then;

POST /my-new-index/_doc
{
   "time": 1615205294745,
   "name": "Test Name"
}

And didn't get any mapping errors.

If that's not what you are seeing then it'd be helpful to post an exact replication of what you are doing so we can test it ourselves

Lorenzo_Zuluaga · April 21, 2021, 12:39am

Sure, It's very strange that it works perfect with my production enviroment, but when try to integrate my ingest app with the staging cluster I run into this issue, because if I let the dynamic mapping to work the data ingest works as expected but the range queries fails, and if I manually specify the time field as a date, the data ingest stops working.

since I'm doing the requests from an http client built in Elixir which is my ingest app, I will try to explain you:
This is how to data looks on production:

%{
  event: "View Property",
  properties: %{
    seller_id: ["\"pxs:1\""],
    actor_status: "Active",
    seller_name: "administrator",
    language: "en",
    listing_title: "MULTIGROUP PROPERTY",
    seller_pxs_id: "1",
    currency: "USD",
    listing_types: ["\"Residential High-rise\", \"Industrial Parks\", \"Tourist Destinations\""],
    seller_pxc_id: nil,
    listing_mls_number: "",
    initial_referring_domain: "showcase-dev.com:3000",
    referrer: "http://showcase-dev.com:3000/developments",
    area_unit: "ft2",
    mp_country_code: nil,
    listing_id: ["\"pxs:5\""],
    screen_height: 1080,
    actor_id: ["\"pxs:1\""],
    actor_pxs_id: "1",
    initial_referrer: "http://showcase-dev.com:3000/developments",
    listing_postcode: "20004",
    browser_version: "88",
    actor_email: "admin@pxs.com", 
    country: nil,
    listing_customer_id: "9",
    listing_pxp_id: nil,
    listing_customer: "Folio - Knight Frank", 
    city: nil,
    mp_processing_time_ms: nil,
    actor_name: "administrator",
    page_type: "Property Details",
    listing_country: "US",
    time: 1618965120519,
    distinct_id: nil,
    referring_domain: "showcase-dev.com:3000",
    listing_state: "District of Columbia",
    event: "View Property",
    current_url: "http://showcase-dev.com:3000/developments/multigroup-property",
    seller_pxp_id: nil,
    listing_owner: "Folio - Knight Frank",
    seller_status: "Active",
    os: "Linux",
    mp_lib: nil,
    listing_price_usd: 0.0,
    listing_pxs_id: "5",
    app: "Showcase",
    lib_version: nil,
    seller_email: "admin@pxs.com",
    screen_width: 1920,
    ...
  },
  time: 1618965120519
}

This is how it looks like when its encoded for the http request:

"{\"time\":1618965120519,\"properties\":{\"region\":null,\"device\":null,\"listing_owner_id\":\"9\",\"listing_pxc_id\":null,\"listing_address\":\"555 11th Street Northwest\",\"listing_city\":\"Washington, D.C.\",\"listing_status\":\"active\",\"search_engine\":null,\"browser\":\"Chrome\",\"screen_width\":1920,\"seller_email\":\"admin@pxs.com\",\"lib_version\":null,\"app\":\"Showcase\",\"listing_pxs_id\":\"5\",\"listing_price_usd\":0.0,\"mp_lib\":null,\"os\":\"Linux\",\"seller_status\":\"Active\",\"listing_owner\":\"Folio - Knight Frank\",\"seller_pxp_id\":null,\"current_url\":\"http://showcase-dev.com:3000/developments/multigroup-property\",\"event\":\"View Property\",\"listing_state\":\"District of Columbia\",\"referring_domain\":\"showcase-dev.com:3000\",\"distinct_id\":null,\"time\":1618965120519,\"listing_country\":\"US\",\"page_type\":\"Property Details\",\"actor_name\":\"administrator\",\"mp_processing_time_ms\":null,\"city\":null,\"listing_customer\":\"Folio - Knight Frank\",\"listing_pxp_id\":null,\"listing_customer_id\":\"9\",\"country\":null,\"actor_email\":\"admin@pxs.com\",\"browser_version\":\"88\",\"listing_postcode\":\"20004\",\"initial_referrer\":\"http://showcase-dev.com:3000/developments\",\"actor_pxs_id\":\"1\",\"actor_id\":[\"\\\"pxs:1\\\"\"],\"screen_height\":1080,\"listing_id\":[\"\\\"pxs:5\\\"\"],\"mp_country_code\":null,\"area_unit\":\"ft2\",\"referrer\":\"http://showcase-dev.com:3000/developments\",\"initial_referring_domain\":\"showcase-dev.com:3000\",\"listing_mls_number\":\"\",\"seller_pxc_id\":null,\"listing_types\":[\"\\\"Residential High-rise\\\", \\\"Industrial Parks\\\", \\\"Tourist Destinations\\\"\"],\"currency\":\"USD\",\"seller_pxs_id\":\"1\",\"listing_title\":\"MULTIGROUP PROPERTY\",\"language\":\"en\",\"seller_name\":\"administrator\",\"actor_status\":\"Active\",\"seller_id\":[\"\\\"pxs:1\\\"\"]},\"event\":\"View Property\"}"

If I allow the dynamic mappings to do it, and I just ingest the first doc I get the following:

%{
  result: %{
    "_id" => "TlzX8XgBBO59mGZ_Shdz",
    "_index" => "pxdata-2021-4",
    "_primary_term" => 1,
    "_seq_no" => 0,
    "_shards" => %{"failed" => 0, "successful" => 1, "total" => 2},
    "_type" => "event",
    "_version" => 1,
    "result" => "created"
  }
}

and If I fetch the index docs I can get the event data of the ingest but, if the query contains range as I showed above I get:

If you try with the following allowing the dynamic mapping work, you will run on the exact issue I'm facing.

1. PUT /index-name
2. POST /index-name/_doc
{
  "time" : 1615145429958,
  "properties" : {
    "actor_status" : null,
    "seller_name" : null,
    "listing_mls_number" : null,
    "listing_customer" : "CB Canada",
    "seller_status" : null,
    "utm_source" : null,
    "media_url" : "https://res.asdas,
    "lib_version" : null,
    "browser_version" : "82",
    "actor_pxs_id" : null,
    "seller_id" : null,
    "current_url" : "https://www.coldwellbanker.ca/property/409-2240-2nd-avenue-whitehorse-yt-y1a1c8-mls-13092/?currency=CAD",
    "listing_pxs_id" : null,
    "initial_referrer" : "$direct",
    "currency" : null,
    "listing_types" : null,
    "media_language" : null,
    "listing_title" : null,
    "listing_country" : null,
    "referrer" : "https://www.coldwellbanker.ca/properties/ca/whitehorse-yukon/?zoom=7&center=60.737419%2C-135.083771&sort=price-&contractTypes=sale&paged=2&language=en&contractTypes=sale&currency=CAD&minBedrooms=0&minBathrooms=0&currency=CAD",
    "utm_term" : null,
    "search_engine" : null,
    "listing_state" : null,
    "seller_pxc_id" : null,
    "listing_postcode" : null,
    "listing_id" : [
      "pxc:5fb6b57f23d1bc2c25d9cca0"
    ],
    "os" : "Windows",
    "screen_height" : 900,
    "device" : null,
    "listing_pxc_id" : "5fb6b57f23d1bc2c25d9cca0",
    "seller_pxp_id" : null,
    "time" : 1615145429958,
    "mp_keyword" : null,
    "city" : "Offenbach",
    "actor_email" : null,
    "listing_price_usd" : null,
    "initial_referring_domain" : "$direct",
    "country" : "DE",
    "actor_id" : null,
    "app" : "B2C-Portals",
    "page_type" : "Property Details",
    "screen_width" : 1600,
    "mp_country_code" : null,
    "utm_medium" : null,
    "browser" : "Firefox",
    "actor_pxc_id" : null,
    "listing_status" : null,
    "actor_pxp_id" : null,
    "referring_domain" : "www.coldwellbanker.ca",
    "listing_address" : null,
    "area_unit" : null,
    "utm_campaign" : null,
    "seller_pxs_id" : null,
    "actor_name" : null,
    "insert_id" : null,
    "listing_pxp_id" : null,
    "mp_processing_time_ms" : null,
    "distinct_id" : null,
    "region" : "Hesse",
    "seller_email" : null,
    "mp_lib" : null,
    "listing_city" : null,
    "media_type" : "Photo",
    "language" : null
  },
  "event" : "View Property Media"
}
3. GET /index-name/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "time": {
              "format": "yyyy/MM/dd||yyyy-MM-dd",
              "gte": "2021-02-19",
              "lte": "2021-04-21"

            }
          }
        }
      ]
    }
  }
}

really appreciate your help
Thanks in advance.

warkolm · April 21, 2021, 1:09am

Ok but what is the actual mapping when that happens?

Yes because it's mapped as this;

"time" : {
    "type" : "long"
}

Lorenzo_Zuluaga · April 21, 2021, 1:14am

This one:

"mappings" : {
      "properties" : {
        "event" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "properties" : {
          "properties" : {
            "app" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "browser" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "browser_version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "city" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "country" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "current_url" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "initial_referrer" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "initial_referring_domain" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "listing_customer" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "listing_id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "listing_pxc_id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "media_type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "media_url" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "os" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "page_type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "referrer" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "referring_domain" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "region" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "screen_height" : {
              "type" : "long"
            },
            "screen_width" : {
              "type" : "long"
            },
            "time" : {
              "type" : "long"
            }
          }
        },
        "time" : {
          "type" : "long"
        }
      }
    }

It's the exact mapping I have on production but the outer time field is date type on production, if I manually set that field to be type: "date", then I'm not able to ingest data because of this error:
"reason" => "mapper [time] cannot be changed from type [date] to [long]"
So apparently one way or another it always seeks to read it as a long type

warkolm · April 21, 2021, 1:16am

Yes. To be clear you need to take these steps;

Create the index
Add the mapping so that your time field is a date`
Index the document

You cannot change the time field mapping once the index has been created and the mapping with the long has been added. It's just not something that Elasticsearch supports. You need to delete the index and follow the above steps.

Lorenzo_Zuluaga · April 21, 2021, 1:22am

Yes sir, I deleted the index,
DELETE /previous-index
added the new one with:

PUT /index-name/
{
  "mappings": {
   "properties" : {
      "time" : {
       "type" : "date"
      }
    }
  }
}

But when doing that the data ingest stop working because of that error:
"reason" => "mapper [time] cannot be changed from type [date] to [long]",

warkolm · April 21, 2021, 1:24am

Ok if it's the ingest process then something else is happening. Is there debug logging you can enable with your client?

Lorenzo_Zuluaga · April 21, 2021, 1:45am

Thanks @warkolm for all of your help, I found the issue was caused by the index type, I was using a custom one called event for staging on my ingest app , while for production I was using _doc . I'm not exactly sure why this was causing the issue with the mappings but I hope this can be useful at some point for someone.

system · May 19, 2021, 1:45am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch dynamic date field mapping Elasticsearch	2	169	June 23, 2023
Inconsistent field mapping -- sometimes string, sometimes long Elasticsearch	4	1433	July 5, 2017
'date' field changes to 'long' after rollover Elasticsearch	3	446	March 7, 2017
Elasticsearch - JSON date payload being converted to type long, cannot use for range queries Elasticsearch	7	2994	March 13, 2018
Update type of a field in Elasticsearch Elasticsearch	4	1188	July 5, 2017

Date field mapping error with type [date] to [long]

Related topics