# Fields and Row template in Annotations \[TSVB\]

**URL:** <https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433>\
**Category:** Kibana\
**Tags:** elastic-stack-machine-learning\
**Created:** [May 7, 2021, 3:08pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433 "2021-05-07T15:08:23Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Oskr](https://avatars.discourse-cdn.com/v4/letter/o/e495f1/32.png) [@Oskr](https://discuss.elastic.co/u/Oskr)\
**Post date:** [May 7, 2021, 3:08pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/1 "2021-05-07T15:08:23Z")

</div>

Hi.  
I am using TSVB to visualize the data stored in the machine learning index, I have seen that I can use annotations to visualize the details of the anomaly but when I try to add the causes.typical field it does not generate the annotations in the visualization.  
Data:

```auto
"_source": {
    "job_id": "customers_fraud",
    "result_type": "record",
    "probability": 0.0017884393731519299,
    "record_score": 0.4412245161339784,
    "initial_record_score": 0.4412245161339784,
    "bucket_span": 3600,
    "detector_index": 0,
    "is_interim": true,
    "timestamp": 1620396000000,
    "function": "count",
    "function_description": "count",
    "over_field_name": "customer_id.keyword",
    "over_field_value": "123456789",
    "causes": [
      {
        "probability": 0.0017884393731519297,
        "function": "count",
        "function_description": "count",
        "typical": [
          4.239593819083309
        ],
        "actual": [
          19
        ],
        "over_field_name": "customer_id.keyword",
        "over_field_value": "123456789"
      }
    ],
    "influencers": [
      {
        "influencer_field_name": "customer_id.keyword",
        "influencer_field_values": [
          "123456789"
        ]
      },
      {
        "influencer_field_name": "payment_method.keyword",
        "influencer_field_values": [
          "CREDIT"
        ]
      },
      {
        "influencer_field_name": "currency.keyword",
        "influencer_field_values": [
          "USD"
        ]
      }
    ],
    "customer_id.keyword": [
      "123456789"
    ],
    "currency.keyword": [
      "USD"
    ],
    "payment_method.keyword": [
      "CREDIT"
    ]
  },
  "fields": {
    "timestamp": [
      "2021-05-07T14:00:00.000Z"
    ]
  },
  "sort": [
    1620396000000
  ]
}

```

 ![Screen Shot 2021-05-07 at 10.05.30 AM](https://us1.discourse-cdn.com/elastic/original/3X/8/f/8fb7074cab3783d87aef1b7dba101835a4fe77cd.png)

---

<div class="post-metadata">

**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\
**Post date:** [May 10, 2021, 11:49am UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/2 "2021-05-10T11:49:37Z")

</div>

Hi Oscar,

These is an array of causes so I think that using `causes.typical` won't address any field from there. It would have to be something like `causes[0].typical[0]` You can try with an without the [0] for typical. I do have doubts that this is a supported scenario, so if you come back and say that it doesn't work like that for you, then I'll open an issue for it in the Kibana repo.

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [May 10, 2021, 12:01pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/3 "2021-05-10T12:01:52Z")

</div>

> [@Marius\_Dragomir](#):
>
> causes[0].typical[0]

I have reproduced the situation for Oscar and I've tried all kinds of combinations of `causes.actual` , `causes.actual._value` , `causes.0.actual`, and now `causes[0].typical[0]` but none seem to work.

I'm wondering if this is related to this bug/enhancement: [TSVB Make mustache template field accessors consistent · Issue #59435 · elastic/kibana · GitHub](https://github.com/elastic/kibana/issues/59435)

---

<div class="post-metadata">

**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\
**Post date:** [May 10, 2021, 12:05pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/4 "2021-05-10T12:05:41Z")

</div>

Yeah, I think we need to create an issue for it then, just specifically and then it can be linked to a meta issue for more improvements.

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [May 10, 2021, 2:52pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/5 "2021-05-10T14:52:03Z")

</div>

I think the issue is primarily because `causes` is a `nested` object....

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [May 10, 2021, 2:56pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/6 "2021-05-10T14:56:10Z")

</div>

@Oskr - A possible workaround could be the usage of [Transforms](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html). In particular, you could use Transforms to re-format the `.ml-anomalies-*` index into a new, very small index for reporting purposes for TSVB. For example:

```auto
PUT _transform/my_ml_annotations
{
  "source": {
    "index": [
      ".ml-anomalies-*"
    ],
    "query": {
      "bool": {
        "filter": [
          {
            "term": {
              "result_type": "record"
            }
          },
          {
            "term": {
              "job_id": "url_scanning"
            }
          },
          {
            "range": {
              "record_score": {
                "gte": "99"
              }
            }
          }
        ]
      }
    }
  },
  "dest": {
    "index": "my_ml_annotations"
  },
  "pivot": {
    "group_by": {
      "timestamp": {
        "date_histogram": {
          "field": "timestamp",
          "fixed_interval": "15m"
        }
      },
      "clientip": {
        "terms": {
          "field": "clientip"
        }
      }
    },
    "aggregations": {
      "record_score": {
        "max": {
          "field": "record_score"
        }
      },
      "typical": {
        "scripted_metric": {
          "init_script": "state.typical = null",
          "map_script": "state.typical = params._source.causes.0.typical.0",
          "combine_script": "return state.typical",
          "reduce_script": "for (d in states) if (d != null) return d"
        }
      },
      "actual": {
        "scripted_metric": {
          "init_script": "state.actual = null",
          "map_script": "state.actual = params._source.causes.0.actual.0",
          "combine_script": "return state.actual",
          "reduce_script": "for (d in states) if (d != null) return d"
        }
      }
    }
  }
}

```

The above will create a new index called `my_ml_annotations` that is "flattened" and looks like the following:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/f/0facc58540f4b295c5ed944734c84bdeef674095.png)

Then, I can use it in TSVB:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/8/f8dd9ea44a4069386496d65c6956a59b54fedac0.png)

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [May 10, 2021, 2:59pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/7 "2021-05-10T14:59:10Z")

</div>

Of course, you'd need to run the transform "continuously" by defining the `frequency` and `sync`:

> **[Create transform API | Elasticsearch Guide \[7.12\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html)**

---

<div class="post-metadata">

**Author:** ![flash1293](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flash1293/32/41227_2.png) [@flash1293](https://discuss.elastic.co/u/flash1293)\
**Post date:** [May 10, 2021, 3:00pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/8 "2021-05-10T15:00:26Z")

</div>

This is possible, but the correct syntax is indeed very hard to hit:

 ![Screenshot 2021-05-10 at 16.59.21](https://us1.discourse-cdn.com/elastic/original/3X/3/2/32fcb4e67d729d0296c42c676a33cfbbc34a79c6.png)

The `fields` list only has to mention `causes`, then in the row template you can get it using this syntax: `{{causes.[0].actual.[0]}}`

It's weird, I know, but it correctly picks the `19` value for the tooltip

---

<div class="post-metadata">

**Author:** ![Oskr](https://avatars.discourse-cdn.com/v4/letter/o/e495f1/32.png) [@Oskr](https://discuss.elastic.co/u/Oskr)\
**Post date:** [May 10, 2021, 4:52pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/9 "2021-05-10T16:52:45Z")

</div>

Hi @flash1293 thanks for your help, Can you confirm the version of Kibana you are working on?  
I tried to do the visualization in the way you describe and it generated the same problem.

---

<div class="post-metadata">

**Author:** ![Oskr](https://avatars.discourse-cdn.com/v4/letter/o/e495f1/32.png) [@Oskr](https://discuss.elastic.co/u/Oskr)\
**Post date:** [May 10, 2021, 4:56pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/10 "2021-05-10T16:56:07Z")

</div>

Thanks @richcollier I think it is a good solution although I don't know if it will generate an overload to the cluster in the future. I will propose it to our team and test it.

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [May 10, 2021, 4:59pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/11 "2021-05-10T16:59:18Z")

</div>

@Oskr - I'm not sure @flash1293 's solution works specifically with the `.ml-anomalies-*` index because the way that it is mapped., but we can wait on his clarification (I couldn't get his suggestion to work either). I suspect his test didn't actually use the true `.ml-anomalies-*` index, but rather a mock-up.

My workaround using Transforms will be incredibly lightweight. Transforms just uses elasticsearch `aggregations` under the hood and the `.ml-anomalies-*` index that it is operating on shouldn't be that big in the first place! 😉

---

<div class="post-metadata">

**Author:** ![Oskr](https://avatars.discourse-cdn.com/v4/letter/o/e495f1/32.png) [@Oskr](https://discuss.elastic.co/u/Oskr)\
**Post date:** [May 10, 2021, 4:59pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/12 "2021-05-10T16:59:29Z")

</div>

> [@Marius\_Dragomir](#):
>
> Hi Oscar,
> 
> These is an array of causes so I think that using `causes.typical` won't address any field from there. It would have to be something like `causes[0].typical[0]` You can try with an without the [0] for typical. I do have doubts that this is a supported scenario, so if you come back and say that it doesn't work like that for you, then I'll open an issue for it in the Kibana

Thanks @richcollier I tried to solve the problem with your tips but they didn't work.

---

<div class="post-metadata">

**Author:** ![Oskr](https://avatars.discourse-cdn.com/v4/letter/o/e495f1/32.png) [@Oskr](https://discuss.elastic.co/u/Oskr)\
**Post date:** [May 10, 2021, 5:01pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/13 "2021-05-10T17:01:38Z")

</div>

Hi @Marius_Dragomir yes the problem is when trying to generate annotations from arrays.  
I tried to solve the problem with your tips but they didn't work.

---

<div class="post-metadata">

**Author:** ![flash1293](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flash1293/32/41227_2.png) [@flash1293](https://discuss.elastic.co/u/flash1293)\
**Post date:** [May 11, 2021, 8:07am UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/14 "2021-05-11T08:07:20Z")

</div>

I'm no ML expert, maybe something special is going on there. TSVB is simply reading the `_source` from the document, so if the source is available, is should work. Maybe ml anomalies are not storing this part of the source?

The part I'm sure about is the mustache syntax for accessing the first value of an array is `path.[0]` (not `path.0` or `path[0]` which would make more sense)

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [May 11, 2021, 4:32pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/15 "2021-05-11T16:32:10Z")

</div>

Yes, ML stores the `causes` array in `_source` with a mapping type of `nested` so there must be something else going on here.

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [May 12, 2021, 12:47pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/16 "2021-05-12T12:47:32Z")

</div>

It turns out this is exactly the situation. The `causes` array in `_source` with a mapping type of `nested` messes up TSVB - as TSVB executes an `exists` filter on the data:

```auto
        {
          "exists": {
            "field": "causes"
          }
        }

```

which returns nothing.

For now, stick with the transforms workaround. There could also be a possibility of also using a [runtime field](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime.html) for items buried in the `causes` array but I have yet to test that (and perhaps TSVB doesn't support runtime fields until 7.13)

---

<div class="post-metadata">

**Author:** ![Oskr](https://avatars.discourse-cdn.com/v4/letter/o/e495f1/32.png) [@Oskr](https://discuss.elastic.co/u/Oskr)\
**Post date:** [May 13, 2021, 5:25pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/17 "2021-05-13T17:25:36Z")

</div>

Thank you @richcollier for the explanation.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 10, 2021, 5:25pm UTC](https://discuss.elastic.co/t/fields-and-row-template-in-annotations-tsvb/272433/18 "2021-06-10T17:25:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.