Elasticsearch - JSON date payload being converted to type long, cannot use for range queries

Elastic Stack Elasticsearch
1

Hi!

I create a mapping like:
> {

  "test3" : {
    "mappings" : {
      "doc" : {
        "properties" : {
          "alerttype" : {
            "type" : "text"
          },
          "devicename" : {
            "type" : "text"
          },
          "eventtime" : {
            "type" : "date"
          },
          "text" : {
            "type" : "text"
          },
          "timestamp" : {
            "type" : "date",
            "format" : "epoch_second"
          }
        }
      }
    }
  }
}

But after my payload is posted, the mapping shows a second "timestamp" entry, that looks like:

},
"timestamp" : {
"type" : "long"
}

At the bottom of the mapping, my original mapping still exists:

},
"timestamp" : {
"type" : "date",
"format" : "epoch_second"
}

So, when I try a query like:
{"query":{"range":{"timestamp":{"from":"1518468475","lte":"now"}}}}

Looking at the above query, I don't suspect this can work. Maybe I'll try using actual dates. Perhaps some conversion will happen on the backend?

But my main point here - how do I force my json epoch seconds to stay as a type 'date'? Or do I just not worry about it, and use them as long ints?

2

This can not happen. I mean the way you described it can not happen. You should see rejection or something.

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

3

Thanks for helping! I'll try to follow formatting.

So I create an index with the following mapping like this:

`PUT /test3
{
  "test3" : {
    "mappings" : {
      "doc" : {
        "properties" : {
          "alerttype" : {
            "type" : "text"
          },
          "devicename" : {
            "type" : "text"
          },
          "eventtime" : {
            "type" : "date"
          },
          "text" : {
            "type" : "text"
          },
          "timestamp" : {
            "type" : "date",
            "format" : "epoch_second"
          }
        }
      }
    }
  }
}`

Then my application does the following:

POST /test3/doc/
{
  "alertinfo": {
    "alerttype": "Showoff-Soft-Trigger",
    "deviceRef": {
      "refName": "Lab-7070",
      "refObjectType": "device_vs_camera_ip",
      "refUid": "161272cc-62e7-4fc4-8b13-ab3daf0b9645",
      "refVsomUid": "1d0ac9ec-e357-47e5-b9c8-978705b9f0dc"
    },
    "devicename": "Lab-7070",
    "eventtime": "Feb 12 at  2018 3:23:36 PM ",
    "text": "Holy crap Batman!!!<br>On Feb 12 at  2018 3:23:36 PM , we received an alert from location: Main Office, with severity: INFO, triggered by device name: Lab-7070.<br>Alert data: Showoff-Soft-Trigger<br>",
    "timestamp": 1518470616
  }
}

After that POST - the return from Elasticsearch is:

{
"_index": "test3",
"_type": "doc",
"_id": "s2nmi2EBkXZxUv2r1RM8",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1
}

When I pull the mapping after the first document is posted my mapping looks like this:

{
  "test3": {
    "mappings": {
      "doc": {
        "properties": {
          "alertinfo": {
            "properties": {
              "alerttype": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "deviceRef": {
                "properties": {
                  "refName": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "refObjectType": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "refUid": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "refVsomUid": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  }
                }
              },
              "devicename": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "eventtime": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "text": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "timestamp": {
                "type": "long"
              }
            }
          },
          "alerttype": {
            "type": "text"
          },
          "devicename": {
            "type": "text"
          },
          "eventtime": {
            "type": "date"
          },
          "text": {
            "type": "text"
          },
          "timestamp": {
            "type": "date",
            "format": "epoch_second"
          }
        }
      }
    }
  }
}`
4

omg. Apparently I cannot figure out your formatting tools here or my browser hates this site :frowning:

5

Just in case - I pulled a pcap from the machine posting my json payload to the elasticsearch instance. I was thinking maybe the key was malformatted. Or maybe a spelling mistake I'm not seeing.

Here is the copy/paste:

`{"alertinfo": {"devicename": "Lab-7070", "eventtime": "Feb 12 at  2018 6:26:04 PM ", "alerttype": "Showoff-Soft-Trigger", "text": "Holy crap Batman!!!<br>On Feb 12 at  2018 6:26:04 PM , we received an alert from location: Main Office, with severity: INFO, triggered by device name: Lab-7070.<br>Alert data: Showoff-Soft-Trigger<br>", "time": 1518481565, "deviceRef": {"refUid": "161272cc-62e7-4fc4-8b13-ab3daf0b9645", "refVsomUid": "1d0ac9ec-e357-47e5-b9c8-978705b9f0dc", "refObjectType": "device_vs_camera_ip", "refName": "Lab-7070"}}}`
6

Don't use the citation icon but only the code icon.

In your example you defined a mapping for a field named timestamp but then you provided a document with a field named alertinfo.timestamp which is not known by elasticsearch and then is created as a number.

7

AH! So mappings need to be nested in cases of nested objects. So I can use the mapping that was dynamically generated as a formatting guide.

Assuming that's correct, thank you!!!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.