Problem with Elastic APM

The indices are not writing to the default names.

It looks like perhaps the apm-server.yml has been edited in the output.elasticsearch section and is no longer using the defaults particularly with respect to the indices names, did you make other changes?

If you wanted to use the default behavior you should not have edited that section.

I would go back to the original / default apm-server.yml. Make the minimal changes for the connections and restart the apm server and see if you get the correct indices.

I am not sure how to / if fix the old data, lets try to get it working first

Should look like....

health status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   apm-7.11.1-transaction-000001 uxazSOaySeSbyUu1XU_mhQ   1   1    7531683            0      5.3gb          2.6gb
green  open   apm-7.11.1-span-000001        WG9zlj0JQ6yvYAKKJ6Owgg   1   1   28420552            0       12gb            6gb
green  open   apm-7.11.1-profile-000001     U2NgU2fVRUeiQkk6jKd9gg   1   1          0            0       416b           208b
green  open   apm-7.11.1-metric-000001      mP6PcMBCSFeehkHj4NaaTQ   1   1    1868972            0      479mb        239.7mb
green  open   apm-7.11.1-error-000001       R0Xx_1HMRzq4X4wpsUeeBA   1   1      48251            0    117.7mb         58.7mb

Default yml

output.elasticsearch:
  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (`http` and `9200`).
  # In case you specify and additional path, the scheme is required: `http://localhost:9200/path`.
  # IPv6 addresses should always be defined as: `https://[2001:db8::1]:9200`.
  hosts: ["localhost:9200"]
...
  # By using the configuration below, APM documents are stored to separate indices,
  # depending on their `processor.event`:
  # - error
  # - transaction
  # - span
  # - sourcemap
  #
  # The indices are all prefixed with `apm-%{[observer.version]}`.
  # To allow managing indices based on their age, all indices (except for sourcemaps)
  # end with the information of the day they got indexed.
  # e.g. "apm-7.3.0-transaction-2019.07.20"
  #
  # Be aware that you can only specify one Elasticsearch template.
  # If you modify the index patterns you must also update these configurations accordingly,
  # as they need to be aligned:
  # * `setup.template.name`
  # * `setup.template.pattern`
  #index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
  #indices:
  #  - index: "apm-%{[observer.version]}-sourcemap"
  #    when.contains:
  #      processor.event: "sourcemap"
  #
  #  - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
  #    when.contains:
  #      processor.event: "error"
  #
  #  - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
  #    when.contains:
  #      processor.event: "transaction"
  #
  #  - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
  #    when.contains:
  #      processor.event: "span"
  #
  #  - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
  #    when.contains:
  #      processor.event: "metric"
  #
  #  - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
  #    when.contains:
  #      processor.event: "onboarding"

  # A pipeline is a definition of processors applied to documents when ingesting them to Elasticsearch.
  # APM Server comes with a default pipeline definition, located at `ingest/pipeline/definition.json`, which is
  # loaded to Elasticsearch by default (see `apm-server.register.ingest.pipeline`).
  # APM pipeline is enabled by default. To disable it, set `pipeline: _none`.
  #pipeline: "apm"
...

Also I would clear this out back to the defaults / empty.

@stephenb, not success bro :frowning:

Look this...

Created index default...

But not showing in the APM

I do not know what to tell you at this point. Did you expand the time frame?
Perhaps the template / mapping is messed up.

I just set up a full cluster, apm-server, and agents from scratch all works OOTB with no changes except endpoints / creds. If there was a serious bug / issue we would be hearing about it from a lot of different directions, that is not happening

I don't see span. apm-7.11.1-span-000001 is it there?

If you run

GET _template/apm-7.11.1

Is it there and fully populated.

If you run to look at the mapping

GET apm-7.11.0-transaction-000001/

Is it several thousand lines long?

Can you go to discover and get a transaction and make sure it is from the very latest index and show what it looks like...

Not sure where to go from here, you have a glitch in your config or settings somewhere. I will see if I can poke someone else to take a look but no promises.

Ohh BTW we are not all Bros on the forum ... After all it IS International Women's Day today after all! :slight_smile:

Yes, have the apm-7.11.1-span-000001,

Yes, have thousand lines in the GET look..

But the other GET return the error..

Okay, tks for the help!

Hahaha Happy International Woman's Day!

Sorry, I got the index..

Look...

Ahhh now we are getting somewhere... that mapping is NOT correct for GET apm-7.11.0-transaction-000001/ I can tell because I see those fields like version
that have both text and keyword

This is wrong, that is a default mapping ... the template is not getting applied.

    "version" : {
      "type" : "text",
      "fields" : {
        "keyword" : {
          "type" : "keyword",
          "ignore_above" : 256
        }
      }
    }

it should be something like

        "version" : {
          "type" : "keyword",
          "ignore_above" : 1024
        }

That means the template (_template/apm-7.11.1) is not being honored... that is why this is not working, somewhere you have disable the template / ILM etc. most likely in the apm-server.yml.

I am not sure what you did but that mapping is not correct...

IF you post your whole apm-server.yml (without creds etc) in TEXT not a screen shot, not a slimmed down version I will take a look, otherwise, like I said get the reference apm-server.yml and just put in the creds and start from there...

Also what does the top of the template look like

GET _template/apm-7.11.1

    {
      "apm-7.11.1" : {
        "order" : 1,
        "index_patterns" : [
          "apm-7.11.1*"
        ],
        "settings" : {
          "index" : {
            "codec" : "best_compression",
            "mapping" : {
              "total_fields" : {
                "limit" : "2000"
              }
            },
            "refresh_interval" : "5s",
            "number_of_shards" : "1",
            "auto_expand_replicas" : "0-1",
            "max_docvalue_fields_search" : "200",
            "query" : {

Notice Text ^^^

Oh I now that I look yeah this is not right... are you using our distribution or some other distribution.

I can see right there you are trying to set up the template instead of using the predefined one.

There is not even a setting called frontend :

In fact I am not clear what all those settings are?

Assuming you are using our distribution here is the reference apm-server.yml I would use this and then only set you connection / information and creds.

https://raw.githubusercontent.com/elastic/apm-server/7.11/apm-server.yml

Possibly useful: [APM] Check whether index template exists · Issue #41331 · elastic/kibana · GitHub and Troubleshooting | Kibana Guide [master] | Elastic

1 Like

Hi Guys,

I'm put the default configuration, but not success :(, I'm removed this me configs and put the apm-server.yml default (changing only the connections and credentials).

I don't know what I do :frowning: very bad!

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.