Not seeing APM data on elasticsearch/Kibana

Elastic Observability APM
1

Kibana version: 6.6.1

Elasticsearch version: 6.6.1

APM Server version: 6.6.1

APM Agent language and version: Node v2.11.6

Browser version: Chrome v75.0.3770.100

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Cloud installation

Steps to reproduce:

  1. Go to APM setup page
  2. Copy snippets & set everything up with the client
  3. Load APM config data
  4. Launch APM

We have a cloud installation v6.6.1.
We have an APM instance enabled, we have followed the steps on the setup page.
We have set up the APM agent on two projects:

  1. One legacy node project,
  2. One Typescript, webpack compiled project.

We start both processes setting the config parameters as environment variables, and injecting the code by using node -r elastic-apm-node/start app.js method.

We have set the trace log level to see if anything looks ok.
We get agent configured correctly with all the settings, there are a lot of no active transaction found - cannot build new span, but some sending transaction and ending transaction messages.

I have manually tampered the Secret Key, and there were authentication errors like {"error":"invalid token"}, so I can assume we are hitting a server.

The APM section won't show anything as also querying the index with this:

GET apm-*-error-*,apm-*-transaction-*/_search
{
  "size": 0, 
  "query": {
    "match_all": {}
  }
}

Any clues?

2

Hi Juan

Welcome to the forum :slight_smile:

There can be several reasons why you don't see any data. First of all, unless you've configured the agent with metricsInterval: 0, you should be seeing data in the metrics index. Try to expand your search to include apm-*-metric.

For why you're not seeing any transactions, there can be several reasons. Before going into more details, I'd like to rule out the most common reason first: Transaction will only automatically be created if your application is an HTTP(S) server. If your application is exposing, for instance, a raw TCP API or some other protocol, or is pulling jobs off a message queue, you need to help the APM agent along by telling it when to start and end new transactions (see agent.startTransaction()).

Could this be the reason you don't see any transactions?

3

No luck with apm-*-metric and metricsInterval is set to 30.
While our app may pull data from queues, it is primarily a simple Express server.
I'll create a bare bones express server and test that to rule out software issues and let you know what I find.

4

Ok, here are my tests results, code is the next one:

const express = require('express');

const app = express();

app.get('/', (req, res, next) => {
  return res.send('Hello!');
});

app.listen(3000, () => console.log('Server up in port 3000'));

Installed libs:
express@4.17.1
elastic-apm-node@2.11.6

App output:

agent configured correctly { pid: 26687,
  ppid: 22454,
  arch: 'x64',
  platform: 'darwin',
  node: 'v8.12.0',
  agent: '2.11.6',
  ancestors:
   [ '/Users/juanstiza/dev/hunt/apm-test/node_modules/elastic-apm-node/index.js',
     '/Users/juanstiza/dev/hunt/apm-test/node_modules/elastic-apm-node/start.js',
     null,
     [length]: 3 ],
  startTrace:
   [ 'at Agent.start (/Users/juanstiza/dev/hunt/apm-test/node_modules/elastic-apm-node/lib/agent.js:149:11)',
     'at Object.<anonymous> (/Users/juanstiza/dev/hunt/apm-test/node_modules/elastic-apm-node/start.js:3:32)',
     'at Module._compile (module.js:653:30)',
     'at Object.Module._extensions..js (module.js:664:10)',
     'at Module.load (module.js:566:32)',
     'at tryModuleLoad (module.js:506:12)',
     'at Function.Module._load (module.js:498:3)',
     'at Module.require (module.js:597:17)',
     'at Function.Module._preloadModules (module.js:754:12)',
     'at preloadModules (bootstrap_node.js:488:38)',
     [length]: 10 ],
  main: undefined,
  dependencies: undefined,
  conf:
   { verifyServerCert: true,
     active: true,
     logLevel: 'trace',
     apiRequestSize: 786432,
     apiRequestTime: 10,
     stackTraceLimit: 50,
     captureExceptions: true,
     filterHttpHeaders: true,
     captureErrorLogStackTraces: 'messages',
     captureSpanStackTraces: true,
     captureBody: 'off',
     errorOnAbortedRequests: false,
     abortedErrorThreshold: 25,
     instrument: true,
     asyncHooks: true,
     sourceLinesErrorAppFrames: 5,
     sourceLinesErrorLibraryFrames: 5,
     sourceLinesSpanAppFrames: 0,
     sourceLinesSpanLibraryFrames: 0,
     errorMessageMaxLength: 2048,
     transactionMaxSpans: 500,
     transactionSampleRate: 1,
     serverTimeout: 30,
     disableInstrumentations: [ [length]: 0 ],
     containerId: undefined,
     kubernetesNodeName: undefined,
     kubernetesNamespace: undefined,
     kubernetesPodName: undefined,
     kubernetesPodUID: undefined,
     captureHeaders: true,
     metricsInterval: 30,
     usePathAsTransactionName: false,
     addPatch: undefined,
     globalLabels: undefined,
     serviceName: 'hunt-mock-service',
     secretToken: '{{SecretToken}}',
     serverUrl: '{{ServerUrl}}',
     logger:
      { trace:
         { [Function: log]
           [length]: 0,
           [name]: 'log',
           [prototype]: log { [constructor]: [Circular] } },
        debug:
         { [Function: log]
           [length]: 0,
           [name]: 'log',
           [prototype]: log { [constructor]: [Circular] } },
        info:
         { [Function: log]
           [length]: 0,
           [name]: 'log',
           [prototype]: log { [constructor]: [Circular] } },
        warn:
         { [Function: log]
           [length]: 0,
           [name]: 'log',
           [prototype]: log { [constructor]: [Circular] } },
        error:
         { [Function: log]
           [length]: 0,
           [name]: 'log',
           [prototype]: log { [constructor]: [Circular] } },
        fatal:
         { [Function: log]
           [length]: 0,
           [name]: 'log',
           [prototype]: log { [constructor]: [Circular] } } },
     ignoreUrlStr: [ [length]: 0 ],
     ignoreUrlRegExp: [ [length]: 0 ],
     ignoreUserAgentStr: [ [length]: 0 ],
     ignoreUserAgentRegExp: [ [length]: 0 ],
     transport:
      { [Function: httpTransport]
        [length]: 2,
        [name]: 'httpTransport',
        [prototype]: httpTransport { [constructor]: [Circular] } },
     serverHost: '{{ServerUrl}}',
     serverPort: 443 } }
adding hook to Node.js module loader
shimming finalhandler@1.1.2 module
shimming http@8.12.0 module
shimming http.Server.prototype.emit function
shimming http.request function
shimming http.get function
shimming http.ServerResponse.prototype.writeHead function
shimming express@4.17.1 module
shimming express.Router.use function
shimming express.static function
copying property mime from express.static
shimming express.Router.Layer.handle function: query
shimming express.Router.Layer.handle function: expressInit
shimming express.Router.Layer.handle function: bound dispatch
Server up in port 3000
no active transaction found - cannot build new span
intercepted call to http.request { id: null }
intercepted request event call to http.Server.prototype.emit
start trace { trans: '58d24bca9fa7679b',
  parent: undefined,
  trace: 'c3f7fbba0d7a07a17cde4e56c1f2d4b1',
  name: null,
  type: null }
no original function off to wrap
no original function off to wrap
setting transaction result { trans: '58d24bca9fa7679b',
  parent: undefined,
  trace: 'c3f7fbba0d7a07a17cde4e56c1f2d4b1',
  result: 'HTTP 2xx' }
setting default transaction name: GET / { trans: '58d24bca9fa7679b',
  parent: undefined,
  trace: 'c3f7fbba0d7a07a17cde4e56c1f2d4b1' }
sending transaction { trans: '58d24bca9fa7679b',
  trace: 'c3f7fbba0d7a07a17cde4e56c1f2d4b1' }
ended transaction { trans: '58d24bca9fa7679b',
  parent: undefined,
  trace: 'c3f7fbba0d7a07a17cde4e56c1f2d4b1',
  type: 'request',
  result: 'HTTP 2xx',
  name: 'GET /' }
no active transaction found - cannot build new span
intercepted call to http.request { id: null }
no active transaction found - cannot build new span
intercepted call to http.request { id: null }
no active transaction found - cannot build new span
intercepted call to http.request { id: null }

Start script: ELASTIC_APM_LOG_LEVEL=trace ELASTIC_APM_SERVICE_NAME=hunt-mock-service ELASTIC_APM_SECRET_TOKEN={{SecretToken}} ELASTIC_APM_SERVER_URL={{ServerUrl}} node -r elastic-apm-node/start index.js

Still no dice.
I tried setting the agent directly on the code:

require('elastic-apm-node').start({
  secretToken: '{{SecretToken}}',
  serverUrl: '{{ServerUrl}}',
  serviceName: 'hunt-mock-service'
});
const express = require('express');

const app = express();

app.get('/', (req, res, next) => {
  return res.send('Hello!');
});

app.listen(3000, () => console.log('Server up in port 3000'));

Same results.

5

I also tested restarting the APM server and reseting the Secret Token. Don't know what else to try.

6

Another update, we tested this same scenario on a new 7.2 install (we are testing prior migration) and it works:

image

Any clues why it wouldn't work with 6.6.1?

7

Interesting... Thanks for the updates and the test app. The agent should be just as compatible with 6.6.1 as it is with 7.2. But you might have found a compatibility bug we're not aware of. I'll have to dig a little deeper to see what's going on and get back to you.

Are you running both the APM Server, Elasticsearch, and Kibana all as v6.6.1?

8

We have a v6.6.1 deployment:

image

I am assuming all 3 services are the same version. Is there a place I can check that?

9

Ah yes - they are. I forgot you were hosted on Elastic Cloud.

I'll look more into it...

10

Thanks!

11

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.