Sure, It's very strange that it works perfect with my production enviroment, but when try to integrate my ingest app with the staging cluster I run into this issue, because if I let the dynamic mapping to work the data ingest works as expected but the range queries fails, and if I manually specify the time field as a date, the data ingest stops working.
since I'm doing the requests from an http client built in Elixir which is my ingest app, I will try to explain you:
This is how to data looks on production:
%{
event: "View Property",
properties: %{
seller_id: ["\"pxs:1\""],
actor_status: "Active",
seller_name: "administrator",
language: "en",
listing_title: "MULTIGROUP PROPERTY",
seller_pxs_id: "1",
currency: "USD",
listing_types: ["\"Residential High-rise\", \"Industrial Parks\", \"Tourist Destinations\""],
seller_pxc_id: nil,
listing_mls_number: "",
initial_referring_domain: "showcase-dev.com:3000",
referrer: "http://showcase-dev.com:3000/developments",
area_unit: "ft2",
mp_country_code: nil,
listing_id: ["\"pxs:5\""],
screen_height: 1080,
actor_id: ["\"pxs:1\""],
actor_pxs_id: "1",
initial_referrer: "http://showcase-dev.com:3000/developments",
listing_postcode: "20004",
browser_version: "88",
actor_email: "admin@pxs.com",
country: nil,
listing_customer_id: "9",
listing_pxp_id: nil,
listing_customer: "Folio - Knight Frank",
city: nil,
mp_processing_time_ms: nil,
actor_name: "administrator",
page_type: "Property Details",
listing_country: "US",
time: 1618965120519,
distinct_id: nil,
referring_domain: "showcase-dev.com:3000",
listing_state: "District of Columbia",
event: "View Property",
current_url: "http://showcase-dev.com:3000/developments/multigroup-property",
seller_pxp_id: nil,
listing_owner: "Folio - Knight Frank",
seller_status: "Active",
os: "Linux",
mp_lib: nil,
listing_price_usd: 0.0,
listing_pxs_id: "5",
app: "Showcase",
lib_version: nil,
seller_email: "admin@pxs.com",
screen_width: 1920,
...
},
time: 1618965120519
}
This is how it looks like when its encoded for the http request:
"{\"time\":1618965120519,\"properties\":{\"region\":null,\"device\":null,\"listing_owner_id\":\"9\",\"listing_pxc_id\":null,\"listing_address\":\"555 11th Street Northwest\",\"listing_city\":\"Washington, D.C.\",\"listing_status\":\"active\",\"search_engine\":null,\"browser\":\"Chrome\",\"screen_width\":1920,\"seller_email\":\"admin@pxs.com\",\"lib_version\":null,\"app\":\"Showcase\",\"listing_pxs_id\":\"5\",\"listing_price_usd\":0.0,\"mp_lib\":null,\"os\":\"Linux\",\"seller_status\":\"Active\",\"listing_owner\":\"Folio - Knight Frank\",\"seller_pxp_id\":null,\"current_url\":\"http://showcase-dev.com:3000/developments/multigroup-property\",\"event\":\"View Property\",\"listing_state\":\"District of Columbia\",\"referring_domain\":\"showcase-dev.com:3000\",\"distinct_id\":null,\"time\":1618965120519,\"listing_country\":\"US\",\"page_type\":\"Property Details\",\"actor_name\":\"administrator\",\"mp_processing_time_ms\":null,\"city\":null,\"listing_customer\":\"Folio - Knight Frank\",\"listing_pxp_id\":null,\"listing_customer_id\":\"9\",\"country\":null,\"actor_email\":\"admin@pxs.com\",\"browser_version\":\"88\",\"listing_postcode\":\"20004\",\"initial_referrer\":\"http://showcase-dev.com:3000/developments\",\"actor_pxs_id\":\"1\",\"actor_id\":[\"\\\"pxs:1\\\"\"],\"screen_height\":1080,\"listing_id\":[\"\\\"pxs:5\\\"\"],\"mp_country_code\":null,\"area_unit\":\"ft2\",\"referrer\":\"http://showcase-dev.com:3000/developments\",\"initial_referring_domain\":\"showcase-dev.com:3000\",\"listing_mls_number\":\"\",\"seller_pxc_id\":null,\"listing_types\":[\"\\\"Residential High-rise\\\", \\\"Industrial Parks\\\", \\\"Tourist Destinations\\\"\"],\"currency\":\"USD\",\"seller_pxs_id\":\"1\",\"listing_title\":\"MULTIGROUP PROPERTY\",\"language\":\"en\",\"seller_name\":\"administrator\",\"actor_status\":\"Active\",\"seller_id\":[\"\\\"pxs:1\\\"\"]},\"event\":\"View Property\"}"
If I allow the dynamic mappings to do it, and I just ingest the first doc I get the following:
%{
result: %{
"_id" => "TlzX8XgBBO59mGZ_Shdz",
"_index" => "pxdata-2021-4",
"_primary_term" => 1,
"_seq_no" => 0,
"_shards" => %{"failed" => 0, "successful" => 1, "total" => 2},
"_type" => "event",
"_version" => 1,
"result" => "created"
}
}
and If I fetch the index docs I can get the event data of the ingest but, if the query contains range as I showed above I get:
If you try with the following allowing the dynamic mapping work, you will run on the exact issue I'm facing.
1. PUT /index-name
2. POST /index-name/_doc
{
"time" : 1615145429958,
"properties" : {
"actor_status" : null,
"seller_name" : null,
"listing_mls_number" : null,
"listing_customer" : "CB Canada",
"seller_status" : null,
"utm_source" : null,
"media_url" : "https://res.asdas,
"lib_version" : null,
"browser_version" : "82",
"actor_pxs_id" : null,
"seller_id" : null,
"current_url" : "https://www.coldwellbanker.ca/property/409-2240-2nd-avenue-whitehorse-yt-y1a1c8-mls-13092/?currency=CAD",
"listing_pxs_id" : null,
"initial_referrer" : "$direct",
"currency" : null,
"listing_types" : null,
"media_language" : null,
"listing_title" : null,
"listing_country" : null,
"referrer" : "https://www.coldwellbanker.ca/properties/ca/whitehorse-yukon/?zoom=7¢er=60.737419%2C-135.083771&sort=price-&contractTypes=sale&paged=2&language=en&contractTypes=sale¤cy=CAD&minBedrooms=0&minBathrooms=0¤cy=CAD",
"utm_term" : null,
"search_engine" : null,
"listing_state" : null,
"seller_pxc_id" : null,
"listing_postcode" : null,
"listing_id" : [
"pxc:5fb6b57f23d1bc2c25d9cca0"
],
"os" : "Windows",
"screen_height" : 900,
"device" : null,
"listing_pxc_id" : "5fb6b57f23d1bc2c25d9cca0",
"seller_pxp_id" : null,
"time" : 1615145429958,
"mp_keyword" : null,
"city" : "Offenbach",
"actor_email" : null,
"listing_price_usd" : null,
"initial_referring_domain" : "$direct",
"country" : "DE",
"actor_id" : null,
"app" : "B2C-Portals",
"page_type" : "Property Details",
"screen_width" : 1600,
"mp_country_code" : null,
"utm_medium" : null,
"browser" : "Firefox",
"actor_pxc_id" : null,
"listing_status" : null,
"actor_pxp_id" : null,
"referring_domain" : "www.coldwellbanker.ca",
"listing_address" : null,
"area_unit" : null,
"utm_campaign" : null,
"seller_pxs_id" : null,
"actor_name" : null,
"insert_id" : null,
"listing_pxp_id" : null,
"mp_processing_time_ms" : null,
"distinct_id" : null,
"region" : "Hesse",
"seller_email" : null,
"mp_lib" : null,
"listing_city" : null,
"media_type" : "Photo",
"language" : null
},
"event" : "View Property Media"
}
3. GET /index-name/_search
{
"query": {
"bool": {
"must": [
{
"range": {
"time": {
"format": "yyyy/MM/dd||yyyy-MM-dd",
"gte": "2021-02-19",
"lte": "2021-04-21"
}
}
}
]
}
}
}
really appreciate your help
Thanks in advance.