APM Quick Start Produces "No data found"

Elastic Observability APM
1

I tried to follow the "Just Four Lines of Code Will Get You Started" install directions here:

https://www.elastic.co/solutions/apm

But I've hit a roadblock in Kibana when I try to see the APM data. I get a "No data found" message.

Here's my install steps-

  • Ubuntu 16.04
  • Java 8 installed with apt repo: ppa:webupd8team/java
  • Elasticsearch and kibana apt-get installed with repo: https://artifacts.elastic.co/packages/6.x/apt stable main
  • X-pack installed in elasticsearch: bin/elasticsearch-plugin install x-pack
  • X-pack installed in kibana: bin/kibana-plugin install x-pack
  • Passwords generated with: elasticsearch/bin/x-pack/setup-passwords auto
  • In /etc/kibana/kibana.yml I updated elasticsearch.username and elasticsearch.password with the elastic credentials from setup-password.
  • I installed and ran apm-server setup
  • And I have apm-server -e running

All other configurations are defaults. I have nginx configured as reverse proxy to Kibana and APM server. Kibana loads and I can login with the kibana credentials generated from setup-passwords. However, when clicking the "Check APM Server Status" button on the APM screen I get a "No data found" message. The same applies to "Check APM agent status" even though I can successfully send a test message from the installed python client in a django project.

I'm brand new to elastic stack, but have experience with opbeat and django development. So forgive me if I'm missing something basic here. I'm not sure where to look for the problem since neither the elasticsearch or apm logs show errors.

Thanks!

Edit 12/30:

When I try to load the dashboard in Kibana I have an error at the top of the screen with a stack trace:

Error: No matching indices found: No indices match pattern "apm-*"

Under Management > Index Patterns I see the apm-* index.

But this leads me to believe there's a problem when running apm-server setup, rather than a connection issue.

2

Hi there,

Thanks for your interest in Elastic APM.
To determine whether APM-server and agent(s) are working correctly we query the APM index for transaction documents.

Specifically, for the agent we make the following query. If hits.total > 0 the check succeeds.

GET apm*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "must": {
        "exists": {
          "field": "processor.name"
        }
      }
    }
  }
}

For the server we make this query (again hits.total > 0):

GET apm*/_search
{
    "size": 0,
    "query": {
       "bool": {
          "must": {
             "exists": {
                "field": "listening"
             }
          }
       }
    }
}

It sounds like the above queries are returning with hits.total = 0. If you go to Kibana Dev tools (http://localhost:5601/app/kibana#/dev_tools or http://YOUR_DOMAIN:5601/app/kibana#/dev_tools) and execute them yourself I would expect them to return zero results.

If you query the apm index for all documents, does that yield any results?

GET apm*/_search
{
  "query": {"match_all": {}}
}

Let me know how it goes.

Happy new year!

3

Søren, thank you for the excellent response. You are correct that I am not getting any results (including the match_all query). Now that I know how to run queries I listed the indicies:

GET _cat/indices

And I see the following:

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
...
yellow open   apm-6.1.1-2017.12.29            PZk3gwSwSpSKEwqds-aC6g   1   1                                                  
...
yellow open   apm-6.1.1-2017.12.30            L8bX8i06Rhm1VcLKTSUTTw   1   1

I'm assuming I can ignore the yellow health status since I'm not running any replicas. So the data is not making it to elasticsearch. I'm going to check the apm-server logs again.

If you have any other suggestions or things I can check, it would be appreciated.

Edit:

I checked the apm-server logs and I don't see any issues:

tail -n 100 /var/log/apm-server/apm-server
2017-12-30T17:31:36Z INFO Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2017-12-30T17:31:36Z INFO Beat UUID: f5c121b6-7212-4499-85a5-8035b6068365
2017-12-30T17:31:36Z INFO Metrics logging every 30s
2017-12-30T17:31:36Z INFO Elasticsearch url: http://localhost:9200
2017-12-30T17:31:36Z INFO Connected to Elasticsearch version 6.1.1

I also ran the following test commands and received the expected output:

apm-server test config
Config OK

apm-server test output -E output.elasticsearch.username=<elastic_un> -E output.elasticsearch.password=<elastic_pw>
elasticsearch: http://localhost:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 6.1.1

I see that nginx is logging the request that is being sent from python manage.py elasticapm test but I don't see anything in the apm-server logs. I added the -v -d flags to apm-server hoping to see some more details.

4

Hi Dan

do you see any log output from your Django app? You might need to set up logging similar to our example in the docs first, but then our agent should print any issues it has when trying to send data to the APM server.

In version 6.1 of the APM server, it doesn't log received HTTP requests, so it isn't surprising that you don't see anything in the logs (we fixed that for the upcoming 6.2 release). However, if you see the requests in the nginx access logs with a response code of 202, it would indicate that the data makes it from the agent to the APM server.

5

Benjamin thanks for the help. I did have the django app configured per your docs. It's essentially the same configuration I use for opbeat. And I was receiving a 202 when running the elasticapm test management command.

I was running apm-server in the background and tailing the logs, but running it from the console produces the following additional output:

2017/12/30 20:43:17.732745 handlers.go:110: DBG [handler] Request: URI=/v1/errors, method=POST, content-length=1056
2017/12/30 20:43:17.734007 payload.go:24: DBG [error] Transform error events: events=1, app=forge, agent=python:1.0.0
2017/12/30 20:43:17.734103 handlers.go:336: DBG [request] request successful, code=202
2017/12/30 20:43:17.734261 event.go:55: DBG [event] Dropped nil value from event where key=error.exception.stacktrace.0.vars.options.settings
2017/12/30 20:43:17.734281 event.go:55: DBG [event] Dropped nil value from event where key=error.exception.stacktrace.0.vars.options.pythonpath
2017/12/30 20:43:17.734292 event.go:55: DBG [event] Dropped nil value from event where key=error.exception.stacktrace.0.vars.options.app_name
2017/12/30 20:43:17.734302 event.go:55: DBG [event] Dropped nil value from event where key=error.exception.stacktrace.0.vars.options.secret_token
2017/12/30 20:43:17.734627 processor.go:275: DBG [publish] Publish event: {
  "@timestamp": "2017-12-30T20:43:17.607Z",
  "@metadata": {
    "beat": "apm-server",
    "type": "doc",
    "version": "6.1.1"
  },
  "processor": {
    "event": "error",
    "name": "error"
  },
  "error": {
    "id": "c3b327d9-3045-4a3d-9d0c-b05b2bcd3dc9",
    "culprit": "elasticapm.contrib.django.management.commands.elasticapm.handle_test",
    "exception": {
      "type": "TestException",
      "stacktrace": [
        {...}
      ],
      "message": "TestException: Hi there!",
      "module": "elasticapm.contrib.django.management.commands.elasticapm"
    },
    "grouping_key": "3ae038f00043c5260b39efad4343f264"
  },
  "context": {
    "system": {
      "hostname": "comp.local",
      "architecture": "x86_64",
      "platform": "darwin"
    },
    "custom": {},
    "app": {
      "agent": {
        "name": "python",
        "version": "1.0.0"
      },
      "pid": 86051,
      "argv": [
        "manage.py",
        "elasticapm",
        "test"
      ],
      "language": {
        "name": "python",
        "version": "2.7.10"
      },
      "runtime": {
        "version": "2.7.10",
        "name": "CPython"
      },
      "framework": {
        "name": "django",
        "version": "1.10.7"
      },
      "name": "forge"
    }
  },
  "beat": {
    "name": "elastic-apm",
    "hostname": "elastic-apm",
    "version": "6.1.1"
  }
}
2017/12/30 20:43:18.741316 client.go:280: DBG [elasticsearch] PublishEvents: 1 events have been  published to elasticsearch in 6.404324ms.
2017/12/30 20:43:18.741392 logger.go:29: DBG [memqueue] ackloop: receive ack [1: 0, 1]
2017/12/30 20:43:18.741437 logger.go:29: DBG [memqueue] broker ACK events: count=1, start-seq=2, end-seq=2
2017/12/30 20:43:18.741464 logger.go:18: DBG [memqueue] ackloop: return ack to broker loop:1
2017/12/30 20:43:18.741480 logger.go:18: DBG [memqueue] ackloop:  done send ack
2017/12/30 20:43:30.533868 metrics.go:39: INFO Non-zero metrics in the last 30s: apm-server.processor.error.counter=1 apm-server.processor.error.transformations=1 apm-server.processor.error.validation.count=1 apm-server.server.requests.counter=1 apm-server.server.response.valid=1 beat.info.uptime.ms=30000 beat.memstats.gc_next=4194304 beat.memstats.memory_alloc=2290200 beat.memstats.memory_total=4292040 libbeat.config.module.running=0 libbeat.output.read.bytes=328 libbeat.output.write.bytes=2978 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=0 libbeat.pipeline.events.published=1 libbeat.pipeline.events.total=1 libbeat.pipeline.queue.acked=1

I'm stumped because this is exactly what I would expect. If I am reading this correctly, it is showing the event publish to elasticsearch. But querying from kibana shows no data.

Something is lying about what's happening.

6

Hi Dan,
I am sorry to hear you are having troubles seeing any data in Kibana.

You mentioned you get a notification that no indices match the pattern in Kibana. This usually only happens if no single entry is indexed with a matching index. However, you confirmed that querying for GET _cat/indices resulted in two apm indices, which indicates that data have been stored.

Usually when starting the APM Server a so called onboarding document is sent to Elasticsearch. Could you try to stop and restart the server and check in the logs if this happens?

Could you please also let us know which versions of Elasticsearch, Kibana and APM Server you are using?

7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.