No services were found

Elastic Observability APM
1

Hi there,

Im trying to set up a full APM e2e scenario with the following stack:
ELK 6.5.1
APM 6.5.1
APM Agent for Java: 1.1.0
Microservices: Spring Boot 2.1

My set up for each of the microservices is:
java -javaagent:elastic-apm-agent-1.1.0.jar
-Delastic.apm.service_name=myservice
-Delastic.apm.server_url=http://apm-server:8200
-Delastic.apm.application_packages=com.mypackage
-jar myservice.jar

Everything is working fine so far as I am to able to see the traces in the timeline but the issue is that in the services tab I see "No services were found" when I should see it.

Selecci%C3%B3n_282
Selección_282.png1623×570 28.3 KB

Selecci%C3%B3n_283
Selección_283.png1583×840 53.9 KB

Any tips about what I am doing wrong?

Thanks a lot.
David

2

Hi David,

That's very interesting.
Just so I understand correctly: On the "Services" tab you see "No services were found", but if you click "Traces" you see a list of traces that you can click on, and thereby get to the second screenshot?

Couple of things to try:

  • The search query context.service.name :* should not make any difference, but try to clear it - just in case.
  • Try expanding the time range. Right now it is 24 hours. What if you change it to 30 days?

If the above steps fail, try running the following query in Kibana Dev Tools:

GET apm*/_search
{
  "bool": {
    "filter": [
      {
        "bool": {
          "should": [
            { "term": { "processor.event": "transaction" } },
            { "term": { "processor.event": "error" } }
          ]
        }
      }
    ]
  }
}

It should return all the transaction/error documents that are used to generate the service list. I'm curious to hear if it comes back empty.

3

Hi Søren,

Thanks for the reply.

I work alongside David and I've been trying what you suggested and have the following outcome:

  • I have removed the search query and still no see services
  • I have expanded the range and the same problem
  • When I try out your query I get:
    Untitled
    Untitled.png2648×930 252 KB

My apm-server setup is:

apm
apm.png964×932 77.2 KB

Thanks for your help again.
Carlos M.

4

Hi Carlos,

My bad. The query should have been:

GET apm*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": [
              {
                "term": {
                  "processor.event": "transaction"
                }
              },
              {
                "term": {
                  "processor.event": "error"
                }
              }
            ]
          }
        }
      ]
    }
  }
}
5

Hi Søren,

Attaching the output in a gist

Hope it helps.

Regards,
Carlos M.

6

Hi Carlos,

Can you please help me to understand how did you get Traces tab beside Services on APM UI.

Thanks in advance.

7

Great, so we've established that you have transaction documents in Elasticsearch.
The following is the endpoint that returns the list of services to the UI. By accessing it directly do you get an error or a 200 response? And is the response empty or not?
<kibana host>/api/apm/services?start=2018-01-29T10%3A36%3A11.547Z&end=2018-11-30T10%3A36%3A11.547Z

Please also look in the Kibana logs for errors, after accessing the above url.

Lastly, the following aggregation is exactly what is used to build the list of services.

{
    "size": 0,
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "should": [
                {
                  "term": {
                    "processor.event": "transaction"
                  }
                },
                {
                  "term": {
                    "processor.event": "error"
                  }
                }
              ]
            }
          }
        ]
      }
    },
    "aggs": {
      "services": {
        "terms": {
          "field": "context.service.name",
          "size": 500
        },
        "aggs": {
          "avg": {
            "avg": {
              "field": "transaction.duration.us"
            }
          },
          "agents": {
            "terms": {
              "field": "context.service.agent.name",
              "size": 1
            }
          },
          "events": {
            "terms": {
              "field": "processor.event",
              "size": 2
            }
          }
        }
      }
    }
  }

I would be very surprised if the api/apm/services endpoint something very differently than the above ES query.

8

Hi Sam21,

The traces view is a new screen related to Distributed Tracing, it comes with 6.5

9

Hi Alvaro,

Thanks for responding. I am using APM-Server version 6.5.0. Do we require to make any specific configuration to enable distributed tracing?

10

Hi Søren,

I get a 400 when accessing the endpoint that returns the list of services to the UI. It complains about fielddata=true on processor.event and context.service.name (which I guess fixing it will solve my problem :grinning:)

[illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [processor.event] in ...... Set fielddata=true on [processor.event] .... Set fielddata=true on [context.service.name] i... Set fielddata=true on [context.service.name] ... Set fielddata=true on [context.service.name] .... Set fielddata=true on [context.service.name]...

I couldn't figure out the query to set fielddata=true on these fields, any tips?

Last bit is that none of these fields are aggregatable, when they should???

image
image.png1900×62 9.14 KB

image
image.png1940×82 6.05 KB

image
image.png1940×84 7.04 KB

Regards,
Carlos M.

11

Hi Carlos,

It sounds like the index template for the APM indices is missing. Running the following command should show you a list of the APM index templates:

GET _template/apm*

Normally the template is created automatically when starting apm-server. Can you restart apm-server and look for template/load.go:129, and what message it says. For instance:

INFO    template/load.go:129    Template already exists and will not be overwritten.

Please view apm-server.yml and ensure the following options are not disabled - they are enabled by default:

setup.template.enabled: true
setup.template.overwrite: true

How did you install apm-server? And did you upgrade from a previous version?

12

Hi Søren,

I was about to reply to you about what you just said. The problem was indeed due to having disabled (don't know why I had that disabled) the kibana template.

Everything is working fine after enabling it back.

Thank you for the time you spent helping me out.

Kind Regards,
Carlos M.

13

Glad to hear you got it working :+1: