Elastic APM Node doing hits again and again

Elastic Observability APM
,
1

I created an simple nodejs application and trying to monitoring the API using elastic-node-apm package and visualize it using kibana but when I check the discover tab of my kibana dashboard I found that my elastic-node-apm hitting again and again a request after every 30 seconds. I don't have any idea why this happening if anyone has any idea Why this is happening please tell here.

Here is my code in my nodejs application :

const apm = require("elastic-apm-node").start({
  serviceName: "nodeAPMSerivce",
  serverUrl: "http://localhost:8200",
});

const express = require("express");
const connectDB = require("./config/db");
const dotenv = require("dotenv");
const User = require("./models/userModel");

dotenv.config();

const app = express();

connectDB();

app.use(express.json());

app.get("/", (req, res) => {
  res.send("Hey, I'm here!!");
});

app.get("/users", async (req, res) => {
  try {
    const users = await User.find({});

    res.status(200).json({
      users,
    });
  } catch (err) {
    console.log(err);
  }
});

app.post("/user", async (req, res) => {
  try {
    const savedUser = await User.create({ ...req.body });

    res.status(201).json({
      savedUser,
    });
  } catch (err) {
    console.log(err);
  }
});

app.listen(8080, () => {
  console.log("Hello I'm listening on PORT 8080");
});

**I am also adding screen shot of my trace logs tab, where you can see that I hit the API ones but getting this messages again and again : **

image
image1187×364 39 KB

I am also adding my discover tab screenshot here and you can see that I made only one get request and it showing 29 hits there.

image
image1035×425 41.1 KB

I just want to know that why are there are so many hits and Is it necessary to stop them ? And if Yes, so how I can stop them ? Please explain me this, I am searching for this from a long time.

Thanks

2

@Piyush_Mittal What is your query and index for the "Discover" tab screenshot? Some of those "hits" might be things other than "requests to your application". For example the data for the one hit shown in your screenshot is a "metric" data object sent by the APM agent. In addition to sending trace data ("transactions" and "spans"), then APM agent also periodically gathers system and application metrics and sends those. Those are sent every metricsInterval which defaults to 30s.

3

@trentm My query is related to that hits:-

  1. Is these hits are necessary or Not ?
  2. Is these hits increasing the load on our system or server ?
  3. Can we stop these hits ?
  4. Can we increase these hits timeperiod so these does not make much load on server?
  5. Any better solution to this and How I can improve my elastic-apm-node file so, I can
    acheive best results.

Thanks.

4

@trentm What is this failed to find message meaning in logs came from these metrices ?

Thanks

5
  1. They are not necessary, no. These collected metrics are a feature of the Elastic APM system. You can read more about how metrics differ from "transactions", "spans", and "errors" that are collected here: Data Model | APM User Guide [8.11] | Elastic
  2. Yes, there is some load from gathering and sending metrics, but very little. Typically it is on the order of a handful of milliseconds after 30s.
  3. Yes, you can stop them by setting metricsInterval="0s" in the APM agent config (Configuration options | APM Node.js Agent Reference [4.x] | Elastic). Note that disabling metrics will mean that some charts in the APM UI in Kibana will not have data to show. Tracing will still be fine.
  4. Yes, you can set metricsInterval to a larger time span. The resolution of metrics data will be affected, but that may be fine for your case.

From your limited screenshots, I do not know what data that "failed to find message" is showing. That string does not appear in the Node.js APM agent code, nor its dependencies, so I don't know where it is coming from.

6

"failed to find message" comes from the Logs app in Kibana, when a log entry is missing the message field. We would need to see more details of the log entry to further diagnose.

7

@axw and @trentm. Ok, I am attaching an screen shot vo detailed view of failed to find message logs.

image
image676×536 36.2 KB

image

image

image

image

please check above images and please tell what is going over here, I just don't understand anything

8

Did you customise the indices for the Logs app? This is a metrics document, in the metrics index. It is not a log record, so that's why the Logs app is having trouble finding the log message.

9

@axw I think, I don't but I am not sure about it because I am new to it so may be I did it. Can you please tell how I can solve it ??

10

Take a look at the settings (top right corner) in the Logs app, and then check the "Log indices":

image
image2197×1247 168 KB

11

Hey @axw I am just try to configure this on my ec2 instance, elasticsearch, kibana is configured and now I am looking for apm-server. I installed the package which is mentioned in document but I am not able to find apm-server.yml file. Can u please tell me how I could filnd it ? It's an ubuntu ec2 instance.

Thanks

12

The directory layout depends on how exactly you installed the APM Server, the docs describing the directory layout should give you the right pointers where you can find the apm-server.yml.

13

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.