Elasticsearch - JSON date payload being converted to type long, cannot use for range queries


(Levi) #1

Hi!

I create a mapping like:
> {

  "test3" : {
    "mappings" : {
      "doc" : {
        "properties" : {
          "alerttype" : {
            "type" : "text"
          },
          "devicename" : {
            "type" : "text"
          },
          "eventtime" : {
            "type" : "date"
          },
          "text" : {
            "type" : "text"
          },
          "timestamp" : {
            "type" : "date",
            "format" : "epoch_second"
          }
        }
      }
    }
  }
}

But after my payload is posted, the mapping shows a second "timestamp" entry, that looks like:

},
"timestamp" : {
"type" : "long"
}

At the bottom of the mapping, my original mapping still exists:

},
"timestamp" : {
"type" : "date",
"format" : "epoch_second"
}

So, when I try a query like:
{"query":{"range":{"timestamp":{"from":"1518468475","lte":"now"}}}}

Looking at the above query, I don't suspect this can work. Maybe I'll try using actual dates. Perhaps some conversion will happen on the backend?

But my main point here - how do I force my json epoch seconds to stay as a type 'date'? Or do I just not worry about it, and use them as long ints?


(David Pilato) #2

This can not happen. I mean the way you described it can not happen. You should see rejection or something.

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.


(Levi) #3

Thanks for helping! I'll try to follow formatting.

So I create an index with the following mapping like this:

`PUT /test3
{
  "test3" : {
    "mappings" : {
      "doc" : {
        "properties" : {
          "alerttype" : {
            "type" : "text"
          },
          "devicename" : {
            "type" : "text"
          },
          "eventtime" : {
            "type" : "date"
          },
          "text" : {
            "type" : "text"
          },
          "timestamp" : {
            "type" : "date",
            "format" : "epoch_second"
          }
        }
      }
    }
  }
}`

Then my application does the following:

POST /test3/doc/
{
  "alertinfo": {
    "alerttype": "Showoff-Soft-Trigger",
    "deviceRef": {
      "refName": "Lab-7070",
      "refObjectType": "device_vs_camera_ip",
      "refUid": "161272cc-62e7-4fc4-8b13-ab3daf0b9645",
      "refVsomUid": "1d0ac9ec-e357-47e5-b9c8-978705b9f0dc"
    },
    "devicename": "Lab-7070",
    "eventtime": "Feb 12 at  2018 3:23:36 PM ",
    "text": "Holy crap Batman!!!<br>On Feb 12 at  2018 3:23:36 PM , we received an alert from location: Main Office, with severity: INFO, triggered by device name: Lab-7070.<br>Alert data: Showoff-Soft-Trigger<br>",
    "timestamp": 1518470616
  }
}

After that POST - the return from Elasticsearch is:

{
"_index": "test3",
"_type": "doc",
"_id": "s2nmi2EBkXZxUv2r1RM8",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1
}

When I pull the mapping after the first document is posted my mapping looks like this:

{
  "test3": {
    "mappings": {
      "doc": {
        "properties": {
          "alertinfo": {
            "properties": {
              "alerttype": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "deviceRef": {
                "properties": {
                  "refName": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "refObjectType": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "refUid": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "refVsomUid": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  }
                }
              },
              "devicename": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "eventtime": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "text": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "timestamp": {
                "type": "long"
              }
            }
          },
          "alerttype": {
            "type": "text"
          },
          "devicename": {
            "type": "text"
          },
          "eventtime": {
            "type": "date"
          },
          "text": {
            "type": "text"
          },
          "timestamp": {
            "type": "date",
            "format": "epoch_second"
          }
        }
      }
    }
  }
}`

(Levi) #4

omg. Apparently I cannot figure out your formatting tools here or my browser hates this site :frowning:


(Levi) #5

Just in case - I pulled a pcap from the machine posting my json payload to the elasticsearch instance. I was thinking maybe the key was malformatted. Or maybe a spelling mistake I'm not seeing.

Here is the copy/paste:

`{"alertinfo": {"devicename": "Lab-7070", "eventtime": "Feb 12 at  2018 6:26:04 PM ", "alerttype": "Showoff-Soft-Trigger", "text": "Holy crap Batman!!!<br>On Feb 12 at  2018 6:26:04 PM , we received an alert from location: Main Office, with severity: INFO, triggered by device name: Lab-7070.<br>Alert data: Showoff-Soft-Trigger<br>", "time": 1518481565, "deviceRef": {"refUid": "161272cc-62e7-4fc4-8b13-ab3daf0b9645", "refVsomUid": "1d0ac9ec-e357-47e5-b9c8-978705b9f0dc", "refObjectType": "device_vs_camera_ip", "refName": "Lab-7070"}}}`

(David Pilato) #6

Don't use the citation icon but only the code icon.

In your example you defined a mapping for a field named timestamp but then you provided a document with a field named alertinfo.timestamp which is not known by elasticsearch and then is created as a number.


(Levi) #7

AH! So mappings need to be nested in cases of nested objects. So I can use the mapping that was dynamically generated as a formatting guide.

Assuming that's correct, thank you!!!


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.