Not able to see the traces of my applicaton(mule4)

[vikascateina (vikas)]

Kibana version:7.17.9

Elasticsearch version:7.17.9

APM Server version:7.17.9

APM Agent language and version:mule4 agent

Browser version:Version 114.0.5735.198 (Official Build) (64-bit)
I am using elasticsearch,kibana,apm only.
If I am adding - index: "%{[service.name]}-%{[service.environment]}-%{+yyyy.MM.dd}" in apm.yml file.I am able to create an automatic index for my application(mule 4) and I am adding cofiguration elastic.apm.environment in mule 4 but then I am not able to see the traces for that index.
If I am reverting this changes as it was before I am able to see the traces for my application.

I had followed this link How to create index for each service?. for creating the indexes automatically .

[Wolfram_Haussig]

Hi,

Did you really rename the index to %{[service.name]}-%{[service.environment]}-%{+yyyy.MM.dd}

As described in the docs here, APM requires indices starting with apm- so your index should be named:
apm-%{[service.name]}-%{[service.environment]}-%{+yyyy.MM.dd}

Otherwise, the data would be stored by the apm server but would not be read by the APM feature in Kibana.

Best regards
Wolfram

[vikascateina (vikas)]

Hi,
I tried this solution but it does'nt work.Through this solution the index is created apm-%{[service.name]}-%{[service.environment]}-%{+yyyy.MM.dd} in this format but the traces for that index is not working.

[simitt (Silvia Mitter)]

It would be helpful if you can share your apm-server.yml configuration, removing any sensitive information.

Especially the following config options are interesting:

• apm-server.ilm.setup.enabled
• output.elasticsearch.index
• output.elasticsearch.indices
• setup.template.enabled
• setup.template.overwrite

[vikascateina (vikas)]

Hi I have attached screenshot of apm.yml file. I am using default apm.yml file. I have only changed in output.elasticsearch in indices only.I have attached template.file also.I am not able to upload a file or send a text of apm.yml that's why I have attached a screenshot.

[simitt (Silvia Mitter)]

When you modify the output.elasticsearch.indices in a way that the default template pattern does not match them anymore, the resulting index templates might be problematic for the APM UI queries.
As you can see from the apm-server.yml, the default index pattern is set via setup.template.pattern: "apm-%{[observer.version]}-*". Since your indices do not contain a version, they won't be matched and the index templates won't be in the format that the APM UI expects them to be.

Removing the version from indices is generally not suggested in 7.17, as your index templates then cannot be restricted to only match certain versions.

I see two options how to move forward, please try one of them and let us know if that helped:

(1) Add the observer.version to your index - change your current indices setting and use this instead

output.elasticsearch.index: "apm-%{[observer.version]}-%{[service.name]}-%{[service.environment]}-%+yyyy.MM.dd}"

(2) If the version should not be in the index name, the template pattern needs to be changed. Try adding the following to the apm-server.yml

setup.template.name: "<choose a unique name>"
setup.template.pattern: "apm-*"
setup.template.enabled: true
setup.template.overwrite: true //if you need to reload the template

[vikascateina (vikas)]

Hi It got resolved by using 1st one .I can able to see the traces and logs for that index but I have one problem ,format of logs is not proper means in logs if there is a sucess or error response of application,In logs it shows nothing success or response of my application.It also dont show the endpoint of my application.It only shows service name and service environment and default logs but I can't see error logs or success logs.

[achyutjhunjhunwala]

Can you go to the Discover tab and search for the exact document you are searching for (you can use the traceId to find the exact document like this trace.id : "YOUR_TRACE_ID") and share it here by removing any confidential data ? We are interested in seeing the Keys mostly and if the required fields have some data.

[vikascateina (vikas)]

{
"_index": "xx",
"_type": "_doc",
"_id": "",
"_version": 1,
"_score": 1,
"_source": {
"system.process.cpu.total.norm.pct": 0.013282732447817837,
"jvm.memory.heap.used": 267462136,
"agent": {
"name": "java",
"ephemeral_id": "xx",
"version": "1.17.0"
},
"jvm.memory.non_heap.committed": 307716096,
"system.cpu.total.norm.pct": 0.03225806451612903,
"observer": {
"hostname": "xx",
"id": "xx",
"ephemeral_id": "xx",
"type": "apm-server",
"version": "7.17.9",
"version_major": 7
},
"system.process.memory.size": xxx,
"ecs": {
"version": "1.12.0"
},
"system.memory.actual.free": 87781376,
"jvm.memory.heap.committed": 368574464,
"host": {
"hostname": "xxx",
"os": {
"platform": "Linux"
},
"ip": "xxx",
"name": "xxx",
"architecture": "amd64"
},
"system.memory.total": 990019584,
"event": {
"ingested": "xxxx"
},
"jvm.thread.count": 128,
"jvm.memory.non_heap.used": 281465920,
"process": {
"pid": xxx,
"title": "/etc/jdk8/jre/bin/java"
},
"jvm.memory.heap.max": 412614656,
"jvm.gc.alloc": xxx,
"processor": {
"name": "metric",
"event": "metric"
},
"metricset.name": "app",
"@timestamp": "xxxx",
"service": {
"node": {
"name": "x"
},
"environment": "xxx",
"name": "xxxx",
"runtime": {
"name": "Java",
"version": "1.8.0_372"
},
"language": {
"name": "Java",
"version": "1.8.0_372"
},
"version": "v1.0.0"
},
"jvm.memory.non_heap.max": -1
},
"fields": {
"system.process.cpu.total.norm.pct": [
0.013
],
"jvm.memory.heap.used": [
267462140
],
"jvm.memory.non_heap.committed": [
307716100
],
"service.node.name": [
"xxxx"
],
"host.hostname": [
"xxxx"
],
"system.cpu.total.norm.pct": [
0.032
],
"process.pid": [
3059
],
"service.language.name": [
"Java"
],
"host.ip": [
"xxx"
],
"system.process.memory.size": [
3393204224
],
"processor.event": [
"metric"
],
"jvm.memory.heap.committed": [
368574460
],
"system.memory.actual.free": [
87781376
],
"agent.name": [
"java"
],
"system.memory.total": [
xx
],
"host.name": [
"xxxx"
],
"jvm.thread.count": [
128
],
"service.environment": [
"xxx"
],
"jvm.memory.non_heap.used": [
281465920
],
"jvm.memory.heap.max": [
412614660
],
"service.name": [
"xxxx"
],
"jvm.gc.alloc": [
9667630000
],
"service.runtime.name": [
"Java"
],
"processor.name": [
"metric"
],
"service.runtime.version": [
"1.8.0_372"
],
"observer.version_major": [
7
],
"observer.hostname": [
"xxx"
],
"host.architecture": [
"amd64"
],
"metricset.name": [
"app"
],
"observer.id": [
"xxx"
],
"event.ingested": [
"2023-08-17T03:42:14.993Z"
],
"@timestamp": [
"2023-08-17T03:42:04.631Z"
],
"service.version": [
"v1.0.0"
],
"observer.ephemeral_id": [
"xxxx"
],
"observer.version": [
"7.17.9"
],
"host.os.platform": [
"Linux"
],
"ecs.version": [
"1.12.0"
],
"observer.type": [
"apm-server"
],
"service.language.version": [
"1.8.0_372"
],
"jvm.memory.non_heap.max": [
-1
],
"agent.ephemeral_id": [
"xxxx"
],
"agent.version": [
"1.17.0"
],
"process.title": [
"/etc/jdk8/jre/bin/java"
]
}
}

The above json is of my logs which is coming from my mule app.In the logs I want to see the co-relation Id ,api endpoint ,success logs or error logs which is coming from mule.I am able to see all this in my mule console but I am not getting all this in elasticsearch logs.

[vikascateina (vikas)]

{
"correlationId" : "xxxx",
"message" : "xxx",
"tracePoint" : "END",
"priority" : "INFO",
"elapsed" : 4334,
"locationInfo" : {
"lineInFile" : "1773",
"component" : "json-logger:logger",
"fileName" : "xxxx",
"rootContainer" : "xxxx"
},
"timestamp" : "2023-08-17T05:23:28.831Z",
"content" : {
"payload" : {
"meta" : {
"tracingId" : "xxx**x-xxx-**xx-xx*-xxx****x",
"count" : ""
},
"data" : [ {
} ]
}
},
"applicationName" : "xxxxx",
"applicationVersion" : "1.0.3",
"environment" : "dev",
"threadName" : "xxxxxx"
}

The above json is of my logs of my application which I am getting in my mule console,but I am not getting this logs in elasticsearch.I wanted the same logs to be seen in elasticsearch.

[achyutjhunjhunwala]

Looking at the JSON, the document which you shared from Elasticsearch is a Metrics document (identified by the property processor.name: metric) and not a log document. Metrics documents are custom documents. You can find more information about them here

Logs are generally stored on a log-* index in Elasticsearch. Can you try navigating to Discover in Kibana -> Then change the Data view on the Top Left to the logs-* dataview and then search for your application log.

[vikascateina (vikas)]

There is nothing in logs as you can see in screenshot.

[achyutjhunjhunwala]

Just to be sure if i am seeing this correctly, can you try changing the time range which is currently set to today to a longer period, like 1 week or more and confirm.

[vikascateina (vikas)]

The logs is not comming here but logs from apm is coming which I shared before i.e in index traces-apm*,apm-,logs-apm,apm-,metrics-apm,apm-*. which is comming from apm I am able to see traces.but not logs

[GregKalapos]

Hi @vikascateina

I work in the APM Agents team.

The mule4 agent is not an official agent and it isn't maintained and developed by the elastic team. It's built on top of our Java Agent (which we officially do support and maintain). Having said that, we try to help you.

The latest mule4 agent is already fairly old - last release is from November 2020 if I see it correctly.

Now.. back to logs and indices: the agent itself is going to ship traces. You say:

logs from apm is coming which I shared before i.e in index traces-apm*,apm-,logs-apm ,apm-,metrics-apm ,apm-*. which is comming from apm I am able to see traces.but not logs

I assume you mean you see documents in in one of the traces indices - those are traces captured by the agent. Please note we typically don't call those "logs" - this is APM related data captured by an APM Agent (e.g. it represents an incoming HTTP call, a database call, or some other similar things).

Now.. there was a log shipping feature introduced by the Java agent in version 1.36.0 released on 2023/01/27 (see release notes). That agent ships with a feature that'll automatically send application logs. In other words the Elastic Java APM Agent will pick up your application logs and send it APM Server which will then store those in Elasticsearch. This feature is opt-in: Logging configuration options | APM Java Agent Reference [1.x] | Elastic

Now, the problem is probably that the Java agent used in the mule4 is an older one and does not contain this feature. So that agent version very likely by default won't send any application logs.

As said, the mule4 agent isn't maintained by us, so we can't really do much unfortunately.

One thing you can do is to maybe look into manually ingesting those logs. We have a documentation on it: Application logs | Elastic Observability [8.9] | Elastic

[vikascateina (vikas)]

Hi , I have shared the dependency which I am using to push the logs from mule to elasticsearch and I have updated log4j file.How do I push application logs in elasticsearch instead of apm data.

   <dependency>
		<groupId>co.elastic.apm</groupId>
		<artifactId>mule4-agent</artifactId> 
		<version>0.4.0</version>
	</dependency>
	<dependency>
		<groupId>co.elastic.apm</groupId>
		<artifactId>apm-agent-attach</artifactId>
		<version>1.17.0</version>
	</dependency>
	<dependency>
		<groupId>co.elastic.apm</groupId>
		<artifactId>apm-agent-api</artifactId>
		<version>1.21.0</version>
	</dependency>

[GregKalapos]

Yeah, seeing those versions verifies my assumptions - this is an old agent which does not have the feature to ship logs.

@vikascateina

How do I push application logs in elasticsearch instead of apm data.

That is described in the doc I shared above: Application logs | Elastic Observability [8.9] | Elastic

Primarily you'll need to use Filebeat.

[vikascateina (vikas)]

I am using elasticsearch,kibana and apm.For Application logs to push in elasticsearch what will be best logstash or filebeat ?

[GregKalapos]

If you just want to ingest your logs, then filebeat will be likely the easiest and fastest way to set this up.

[vikascateina (vikas)]

Hi,
I have implemented logstash but in my index management I am not able to see Indices logstash but Ingest pipeline is created as shown in image.

I want to push my mule application logs to logstash.How can I do it.