Elastic APM Node doing hits again and again

[Piyush_Mittal]

I created an simple nodejs application and trying to monitoring the API using elastic-node-apm package and visualize it using kibana but when I check the discover tab of my kibana dashboard I found that my elastic-node-apm hitting again and again a request after every 30 seconds. I don't have any idea why this happening if anyone has any idea Why this is happening please tell here.

Here is my code in my nodejs application :

const apm = require("elastic-apm-node").start({
  serviceName: "nodeAPMSerivce",
  serverUrl: "http://localhost:8200",
});

const express = require("express");
const connectDB = require("./config/db");
const dotenv = require("dotenv");
const User = require("./models/userModel");

dotenv.config();

const app = express();

connectDB();

app.use(express.json());

app.get("/", (req, res) => {
  res.send("Hey, I'm here!!");
});

app.get("/users", async (req, res) => {
  try {
    const users = await User.find({});

    res.status(200).json({
      users,
    });
  } catch (err) {
    console.log(err);
  }
});

app.post("/user", async (req, res) => {
  try {
    const savedUser = await User.create({ ...req.body });

    res.status(201).json({
      savedUser,
    });
  } catch (err) {
    console.log(err);
  }
});

app.listen(8080, () => {
  console.log("Hello I'm listening on PORT 8080");
});

**I am also adding screen shot of my trace logs tab, where you can see that I hit the API ones but getting this messages again and again : **

I am also adding my discover tab screenshot here and you can see that I made only one get request and it showing 29 hits there.

I just want to know that why are there are so many hits and Is it necessary to stop them ? And if Yes, so how I can stop them ? Please explain me this, I am searching for this from a long time.

Thanks

[trentm (Trent Mick)]

@Piyush_Mittal What is your query and index for the "Discover" tab screenshot? Some of those "hits" might be things other than "requests to your application". For example the data for the one hit shown in your screenshot is a "metric" data object sent by the APM agent. In addition to sending trace data ("transactions" and "spans"), then APM agent also periodically gathers system and application metrics and sends those. Those are sent every metricsInterval which defaults to 30s.

[Piyush_Mittal]

@trentm My query is related to that hits:-

1. Is these hits are necessary or Not ?
2. Is these hits increasing the load on our system or server ?
3. Can we stop these hits ?
4. Can we increase these hits timeperiod so these does not make much load on server?
5. Any better solution to this and How I can improve my elastic-apm-node file so, I can
   acheive best results.

Thanks.

[Piyush_Mittal]

@trentm What is this failed to find message meaning in logs came from these metrices ?

Thanks

[trentm (Trent Mick)]

@Piyush_Mittal

• Is these hits are necessary or Not ?
• Is these hits increasing the load on our system or server ?
• Can we stop these hits ?
• Can we increase these hits timeperiod so these does not make much load on server?
• Any better solution to this and How I can improve my elastic-apm-node file so, I can
  acheive best results.

1. They are not necessary, no. These collected metrics are a feature of the Elastic APM system. You can read more about how metrics differ from "transactions", "spans", and "errors" that are collected here: Data Model | APM User Guide [8.11] | Elastic
2. Yes, there is some load from gathering and sending metrics, but very little. Typically it is on the order of a handful of milliseconds after 30s.
3. Yes, you can stop them by setting metricsInterval="0s" in the APM agent config (Configuration options | APM Node.js Agent Reference [4.x] | Elastic). Note that disabling metrics will mean that some charts in the APM UI in Kibana will not have data to show. Tracing will still be fine.
4. Yes, you can set metricsInterval to a larger time span. The resolution of metrics data will be affected, but that may be fine for your case.

@Piyush_Mittal

What is this failed to find message meaning in logs came from these metrices ?

From your limited screenshots, I do not know what data that "failed to find message" is showing. That string does not appear in the Node.js APM agent code, nor its dependencies, so I don't know where it is coming from.

[axw (Andrew Wilkins)]

"failed to find message" comes from the Logs app in Kibana, when a log entry is missing the message field. We would need to see more details of the log entry to further diagnose.

[Piyush_Mittal]

@axw and @trentm. Ok, I am attaching an screen shot vo detailed view of failed to find message logs.

please check above images and please tell what is going over here, I just don't understand anything

[axw (Andrew Wilkins)]

Did you customise the indices for the Logs app? This is a metrics document, in the metrics index. It is not a log record, so that's why the Logs app is having trouble finding the log message.

[Piyush_Mittal]

@axw I think, I don't but I am not sure about it because I am new to it so may be I did it. Can you please tell how I can solve it ??

[axw (Andrew Wilkins)]

Take a look at the settings (top right corner) in the Logs app, and then check the "Log indices":

[Piyush_Mittal]

Hey @axw I am just try to configure this on my ec2 instance, elasticsearch, kibana is configured and now I am looking for apm-server. I installed the package which is mentioned in document but I am not able to find apm-server.yml file. Can u please tell me how I could filnd it ? It's an ubuntu ec2 instance.

Thanks

[simitt (Silvia Mitter)]

The directory layout depends on how exactly you installed the APM Server, the docs describing the directory layout should give you the right pointers where you can find the apm-server.yml.