Outputting service.name instead of / alongside context.service.name

[rozling]

My question also relates to upcoming schema / mapping changes but I didn't want to hijack the other thread from today, and I think my question might be a bit simpler for now.

I am currently creating service dashboards, and trying to pull in data from APM, filebeat, metricbeat & custom logstash-prefixed indices.

Currently the dashboards break as soon as we filter on a service name, since this field is mapped differently across agents / beats.

After reading about the Elastic Common Schema I want to start standardising things, starting with the service name.

My question is: is it possible, without upgrading the agent / server version, to add a service.name field alongsidecontext.service.name?

This would be a big win for us and would mean we could get most of our important metrics on one dashboard vs multiple dashboards.

APM Server version
6.4.3

APM Agent language and version:
NodeJS, 1.9.0

Fresh install or upgraded from other version?
Fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

No lb, single instance. we do possibly have a custom agent config

[gil (Gil Raphaelli)]

Wow @rozling, I'm impressed you are so up on the other postings here. I've replied on the other thread, I hope you find that helpful too.

without upgrading the agent / server version, to add a service.name field alongside context.service.name ?

If you're on Elasticsearch >= 6.4 you can utilize field aliases to accomplish this. For example, to add an alias to an existing index apm-6.6.0-transaction-2019.02.19:

PUT apm-6.6.0-transaction-2019.02.19/_mapping/doc
{
  "properties": {
    "service": {
      "properties": {
        "name": {
          "type": "alias",
          "path": "context.service.name"
        }
      }
    }
  }
}

You probably also want to update your index template so that any new indices are created with aliases in place. For that, add a new index template like:

PUT _template/apm-6-aliases
{
  "index_patterns": [
    "apm-6.*"
  ],
  "mappings": {
    "doc": {
      "properties": {
        "context": {
          "properties": {
            "service": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            }
          }
        },
        "service": {
          "properties": {
            "name": {
              "type": "alias",
              "path": "context.service.name"
            }
          }
        }
      }
    }
  }
}

[rozling]

Hi Gil, many thanks for that - that worked flawlessly!

Actually I think I searched the forum for context before posting and there just happened to be a related post today but yes I'm hoping to spend a bit more time on the forums here as there's some great information around.

I've never used field aliases before, but as it happens that solves another issue which was that the values for the service names were also inconsistent between APM / logstash-processed indices; using this technique I can now match those up.

The only drawback as I've just found with index aliases is that they're not supported by Canvas , which is a shame as I've been putting together some cool overview-type workpads.

I was thinking of using an ingest pipeline, and taking the opportunity to enforce certain values for our service.name field. Many of our services have been renamed since we integrated APM with Node, but it will be some time before these naming changes will be reflected in our global APM config

However it seems that approach would require pipeline conditionals to work, and these are unavailable in our version of Elasticsearch (6.4).

So from what I can tell our choices are:

• Use our old service names as the common values for a service.name alias, and stick with traditional dashboards for now
• Upgrade Elasticsearch to 6.5 in order to gain conditionals in pipelines, add service.name field with correct name
• Remove the alias, use Logstash to parse / mutate APM transactions, add service.name field with correct name

Any thoughts / input would be great; I'm sure I've missed something though so will sleep on it and see if any ideas come along...

[gil (Gil Raphaelli)]

using an ingest pipeline, and taking the opportunity to enforce certain values for our service.name field

Do you intend to just rename context.service.name? If so, would the gsub processor be suitable? That should be available in 6.4.

Upgrade Elasticsearch to 6.5

An upgrade to 6.5 or later would also give you access to distributed tracing and apm-server monitoring so I'd recommend it at some point, if possible.

will sleep on it and see if any ideas come along...

Please let us know if you came up with any other solution, I don't think this is the last time we'll come across this. I've also started a discussion internally about ideas to help support these kinds of changes within apm-server.

[rozling]

Hi gil,

Do you intend to just rename context.service.name ? If so, would the gsub processor be suitable? That should be available in 6.4.

I was planning to add it as an extra field, however since posting I have realised that for non-APM data the addition of a service.x field is a problem, because most of our indices have a service field already mapped to text datatype.

So in order to unify things I would need to figure out a way to have service.x (as an object) live alongside service (as text), if that's even possible.

Please let us know if you came up with any other solution, I don't think this is the last time we'll come across this. I've also started a discussion internally about ideas to help support these kinds of changes within apm-server.

Will do, and that would be great - we got stung at one point with a filebeat minor release that changed mappings; any existing support around that kind of thing would save a lot of hassle

[rozling]

Ok, this is weird: I tried adding a field alias at the root level for serviceName context.service.name:

What I end up with in the docs is that serviceName gets put into the context.custom object:

This is a problem because our logs already have a serviceName field on the root level which I planned use to filter / group data from disparate sources on dashboards.

Is this known / expected behaviour with APM?