Continuous Transform Timestamp isn't a timestamp

Elastic Stack Elasticsearch
1

Set up a continuous transform, grouping by a time historgram driven from the @timestamp field. The transform contains an @timestamp value, but it's classified as a number rather than as a date (according to Kibana when I'm trying to define an index pattern so I can look at the data).

I've not got a field mapping for the transformed index - I was hoping the automatics would get it right. Guess I need to manually create a field mapping for it :frowning:

2

Can you post the relevant parts of your transform configuration?
What is the created mapping of the destination index?

Do you mean kibana index pattern? There is an open issue about improving index pattern generation.

3

I'm using a continuous transform that I used Kibana to construct, then exported to a file.

From the transform:

  "pivot": {
    "group_by": {
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1h"
        }
      },

...and...

  "sync": {
    "time": {
      "field": "ingest_epoch",
      "delay": "60s"
    }
  }

The @timestamp field ended up with a type of long in the index the Continuous transform wrote to, which meant I couldn't set it as the time field in the index pattern for the output index.

The point with the sync clip is that I'm not using @timestamp as the sync field for the transform.

After finding this problem, I exported the mapping for the index the transform was writing to and change the type to date. Then trashed my dev stack and reloaded the whole thing from scratch, using my edited mapping for the transformed index (rather than letting elasticsearch compose one) which meant the it ended up with the date type for @timestamp that Id specified.

This is with 7.8.

Mik

4

I tried to reproduce your issue and checked the code and can not explain it. The mapping type for the group_by is taken from the mapping of the source. If @timestamp ends up as long, it is a long in the source index, too. If it's date, it will be date in the transform dest index. Can you verify the mappings of your source index?

For a dry-run you can use the preview API, it shows which mappings transform would create. As you already found out, you can create the destination index yourself. This is for customization and special requirements. However you have to ensure that the mapping is compatible with the transform, otherwise the transform might fail.

Using a different timestamp for sync than for group_by is possible, but has performance implications. If both fields are the same a performance optimization kicks in, which can make a significant difference. In future we might optimize this, but at least till 7.8 its not advised to use different fields for sync and group_by for larger datasets.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.