No data for breakdown graphs

Elastic Observability APM
1

Hello

Kibana version: 7.5.2

Elasticsearch version: 7.5.2

APM Server version: 7.5.2

APM Agent language and version: 3.4.0

Browser version: Crome Version 80.0.3987.87 (Official Build) beta (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Kubernetes

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? No

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

A problem which I've been dealing with is missing data for " Time spent by span type" graph in my staging environment. However, it works for development environment (the app sends events to the same APM server). I checked raw events in APM logs and they looks similar for both environments. I also checked that data is in the APM indexes for both environments.

This request that loads data from breakdown metrics returns empty objects
http://0.0.0.0/api/apm/services/Buyer/transaction_groups/breakdown?start=2020-01-31T10%3A05%3A09.815Z&end=2020-01-31T10%3A20%3A09.815Z&transactionName=GET%20%2F%3Aversion%2Flots%2F%3Alot_id%2Fdetail&transactionType=request&uiFilters=%7B%22environment%22%3A%22staging%22%7D

{"kpis":[],"timeseries":[]}

I have a single APM instance running in a Kubernetes cluster along with Elastic and Kibana instances. I instrumented my Rails/Rack application using elastic_apm gem with following configuration.

config/elastic_apm.yml
server_url: 'http://0.0.0.0.0:8200' # fake ip
secret_token: ''
service_name: 'MyApp'

config.ru
require ::File.expand_path('../config/environment', FILE)

# unicorn self-process killer
require 'unicorn/worker_killer'

# worker will be killed between 400 to 420 Mb
use Unicorn::WorkerKiller::Oom, 419430400, 450887680

builder = Rack::Builder.new do
  map '/private/health-check' do
    run lambda{ |_| [200, { 'Content-Type' => 'text/plain' }, ['imalive']] }
  end

  map '/track' do
    use ElasticAPM::Middleware
    # TODO: use Honeybadger::Rack::ErrorNotifier
    use ActionDispatch::RemoteIp, false

    # add addition request information
    use CW::SDK::Middleware::LoggerContext

    run MyApp::Rack::Track.new
  end

  map '/' do
    run Rails.application
  end

Also, my staging environment is running on a Kubernetes cluster and there is no significant difference in app environments. So I got stuck with it. Any ideas what else I can check?

Thanks in advance.

2

There is a screenshot bellow that shows there traces/spans but not data for breakdown metrics.

I also checked that breakdown_metrics is enabled.

image
image3114×1262 170 KB

3

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.