EKK stack and Timestamps

[luciddr34m3r (Curt Barnard)]

Hey there,

I'm sending syslog data to Elasticsearch through an AWS Kinesis Firehose. I send a json object of parsed syslog information. I convert the timestamp to epoch and add it as an additional field named "date". I set the index mapping for date to be interpreted as a date type with format epoch_second, but once a record is sent through the firehose, it creates a mapping conflict and Kibana says the field is a number, not a date.

Any ideas on what I'm doing wrong?

[jbudz (Jon Budzenski)]

Are there any old indices with the field mapped as a number that still match the index pattern?

Kibana's index pattern page will use the field capalities api behind the scenes, and then stores a copy of the response in the .kibana index. Sometimes this needs to be refreshed by clicking the refresh button in Kibana. Trying that may help too.

[luciddr34m3r (Curt Barnard)]

No. I run the following in the console:

DELETE _all

PUT iptables

PUT iptables/_mapping/iptables
{
  "iptables": {
    "properties": {
      "date": {
        "type": "date"
      }
    }
  }
}

And then Firehose is set to put objects of type 'iptables' into the 'iptables' index.

Date is an epoch timestamp in seconds, but it doesn't seem to matter if I set it or not. I've tried it every way I can think of.

If I don't create an index first, Kibana says the documents have no time data. The json objects look like this:

{
  "_index": "iptables-2018-04-02",
  "_type": "iptables",
  "_id": "49581617944237641073795881024580975589499732442392559618.0",
  "_score": 1,
  "_source": {
    "processid": null,
    "hostname": "REDACTED",
    "ipt_src": "REDACTED",
    "ipt_dpt": "22",
    "ipt_spt": "60690",
    "timestamp": "Apr  2 17:20:36",
    "ipt_proto": "TCP",
    "ipt_dst": "REDACTED",
    "program": "kernel",
    "date": 1522689636,
    "message": "REDACTED ",
    "type": "ipt",
    "ipt_ttl": "33"
  }
}