APM pipeline painless script vanishes after upgrade (runtime field instead?)

Elastic Observability APM
1

Hello,

We used to have the following painless script for two of our APM pipelines (traces-apm-8.1.2 & traces-apm.rum-8.1.2) that vanished after upgrading escloud version from 8.1.2 to 8.2.3.

{
  "script": {
    "lang": "painless",
    "source": "double apdex_t = 800.0 * 1000;\nif (ctx.labels==null) {\nctx.labels = [:];\n}\nif (ctx.transaction.duration.us <= apdex_t) {\nctx.labels.apdexscore=1.0;\n}\nelse f (ctx.transaction.duration.us <= (apdex_t * 4)) {\nctx.labels.apdexscore=0.5;\n}\nelse {\nctx.labels.apdexscore=0.0;\n}",
    "ignore_failure": true
  }
}

That script generates an apdex value an creates a new field with the value.

We don't only have the problem of removing the painless script when upgrading but we also have the problem of having new pipelines after the upgrade (traces-apm.rum-8.2.0 & traces-apm-8.2.0)

We were thinking of moving that painless script to a runtime field for the Index Templates traces-apm & traces-apm-rum , but we don't know how to do that and we also don't know if that runtimefield will disappear when we upgrade escloud in the future.

Anybody has an idea how to fix this? Thanks

Kind regards

Mario

2

Hi @mmartinez,

As of writing this comment, pipelines don't persist through upgrades. However, the feature to support optional and custom ingest pipelines is in work.

For runtime fields, you can define them on a data-view from the Stack Management/Data Views section. Documentation for the same can be found here. Runtime fields also use painless script, an example of the script you shared translated to be used with runtime field would be something like:

if (!doc.containsKey('transaction.duration.us') ||
   doc['transaction.duration.us'].empty
) { return; }

def txnDuration = doc['transaction.duration.us'].value;
double apdex_t = 800.0 * 1000;

if (txnDuration <= apdex_t) {
    emit(1.0);
} else if (txnDuration <= (apdex_t * 4)) {
    emit(0.5);
} else {
    emit(0.0);
}
3

Hi @lahsivjar,

Thanks for all your help. I created the runtime field at the Data View with the script you mentioned and it worked. I also modified some visualisations pointing to that new field and they worked too but there is a watcher failing since then.

I attach an screenshot of the runtime field set up.

Screenshot 2022-07-27 at 15.57.41
Screenshot 2022-07-27 at 15.57.41877×1141 112 KB

And this is the watcher:

  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "traces-apm*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-60m"
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "services": {
              "terms": {
                "field": "service.name",
                "size": 30
              },
              "aggs": {
                "avg_apdexscore": {
                  "avg": {
                    "field": "apdexscore"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": """
      return ctx.payload.aggregations.services.buckets.stream()       
        .filter(services -> services.avg_apdexscore.value < 0.8)               
        .count() > 0 
      """,
      "lang": "painless"
    }
  },
  "actions": {
    "log": {
      "transform": {
        "script": {
          "source": """
        return ctx.payload.aggregations.services.buckets.stream()       
        .filter(services -> services.avg_apdexscore.value < 0.8)               
        .collect(Collectors.toList());
        """,
          "lang": "painless"
        }
      },
      "logging": {
        "level": "info",
        "text": """
        {{#ctx.payload._value}} Service {{key}} is below 0.8 with apdexscore value={{avg_apdexscore.value}}
        {{/ctx.payload._value}}
        """
      }
    }
  }
}

But all services are returning null values like this

          "services": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "doc_count": 1352150,
                "avg_apdexscore": {
                  "value": null
                },
                "key": "prod-wh"
              },

And this is the error related to those null values.

      "type": "null_pointer_exception",
      "reason": "Cannot invoke \"Object.getClass()\" because \"leftObject\" is null",
      "stack_trace": "java.lang.NullPointerException: Cannot invoke \"Object.getClass()\" because \"leftObject\" is null\n\tat

Do you know why this is happening? Thanks in advance

Mario

4

Glad that the runtime fields worked for you. I am not a watcher expert but, AFAIK, dataview runtime fields are not available to watcher for a query. Dataview alerts can be used with runtime fields on dataview but I think they are available in 8.3+ versions.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.