# Not able to see the traces of my applicaton(mule4)

**URL:** <https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421>\
**Category:** APM\
**Tags:** server\
**Created:** [August 9, 2023, 6:15am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421 "2023-08-09T06:15:53Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![vikascateina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikascateina/32/123762_2.png) [@vikascateina](https://discuss.elastic.co/u/vikascateina)\
**Post date:** [August 9, 2023, 6:15am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/1 "2023-08-09T06:15:53Z")

</div>

**Kibana version** :7.17.9

**Elasticsearch version** :7.17.9

**APM Server version** :7.17.9

**APM Agent language and version** :mule4 agent

**Browser version** :Version 114.0.5735.198 (Official Build) (64-bit)  
I am using elasticsearch,kibana,apm only.  
If I am adding - index: "%{[service.name]}-%{[service.environment]}-%{+yyyy.MM.dd}" in apm.yml file.I am able to create an automatic index for my application(mule 4) and I am adding cofiguration elastic.apm.environment in mule 4 but then I am not able to see the traces for that index.  
If I am reverting this changes as it was before I am able to see the traces for my application.

I had followed this link [How to create index for each service?](https://discuss.elastic.co/t/how-to-create-index-for-each-service/242821). for creating the indexes automatically .

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [August 9, 2023, 7:50am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/2 "2023-08-09T07:50:40Z")

</div>

Hi,

Did you really rename the index to `%{[service.name]}-%{[service.environment]}-%{+yyyy.MM.dd}`

As described in the docs [here](https://www.elastic.co/guide/en/kibana/current/apm-settings-kb.html), APM requires indices starting with `apm-` so your index should be named:  
`apm-%{[service.name]}-%{[service.environment]}-%{+yyyy.MM.dd}`

Otherwise, the data would be stored by the apm server but would not be read by the APM feature in Kibana.

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![vikascateina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikascateina/32/123762_2.png) [@vikascateina](https://discuss.elastic.co/u/vikascateina)\
**Post date:** [August 9, 2023, 10:31am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/3 "2023-08-09T10:31:32Z")

</div>

Hi,  
I tried this solution but it does'nt work.Through this solution the index is created apm-%{[service.name]}-%{[service.environment]}-%{+yyyy.MM.dd} in this format but the traces for that index is not working.

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [August 9, 2023, 11:27am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/4 "2023-08-09T11:27:00Z")

</div>

It would be helpful if you can share your `apm-server.yml` configuration, removing any sensitive information.

Especially the following config options are interesting:

- `apm-server.ilm.setup.enabled`
- `output.elasticsearch.index`
- `output.elasticsearch.indices`
- `setup.template.enabled`
- `setup.template.overwrite`

---

<div class="post-metadata">

**Author:** ![vikascateina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikascateina/32/123762_2.png) [@vikascateina](https://discuss.elastic.co/u/vikascateina)\
**Post date:** [August 9, 2023, 12:41pm UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/5 "2023-08-09T12:41:58Z")

</div>

Hi I have attached screenshot of apm.yml file. I am using default apm.yml file. I have only changed in output.elasticsearch in indices only.I have attached template.file also.I am not able to upload a file or send a text of apm.yml that's why I have attached a screenshot.

 ![Screenshot from 2023-08-09 18-00-03](https://us1.discourse-cdn.com/elastic/original/3X/c/7/c7a968b818fd490ad80d179c8785bac005441e5e.png)

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [August 9, 2023, 1:34pm UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/6 "2023-08-09T13:34:53Z")

</div>

When you modify the `output.elasticsearch.indices` in a way that the default template pattern does not match them anymore, the resulting index templates might be problematic for the APM UI queries.  
As you can see from the `apm-server.yml`, the default index pattern is set via `setup.template.pattern: "apm-%{[observer.version]}-*"`. Since your indices do not contain a `version`, they won't be matched and the index templates won't be in the format that the APM UI expects them to be.

Removing the version from indices is generally not suggested in `7.17`, as your index templates then cannot be restricted to only match certain versions.

I see two options how to move forward, please try one of them and let us know if that helped:

(1) Add the `observer.version` to your index - change your current indices setting and use this instead

```auto
output.elasticsearch.index: "apm-%{[observer.version]}-%{[service.name]}-%{[service.environment]}-%+yyyy.MM.dd}"

```

(2) If the version should not be in the index name, the template pattern needs to be changed. Try adding the following to the `apm-server.yml`

```auto
setup.template.name: "<choose a unique name>"
setup.template.pattern: "apm-*"
setup.template.enabled: true
setup.template.overwrite: true //if you need to reload the template

```

---

<div class="post-metadata">

**Author:** ![vikascateina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikascateina/32/123762_2.png) [@vikascateina](https://discuss.elastic.co/u/vikascateina)\
**Post date:** [August 11, 2023, 3:49am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/7 "2023-08-11T03:49:17Z")

</div>

Hi It got resolved by using 1st one .I can able to see the traces and logs for that index but I have one problem ,format of logs is not proper means in logs if there is a sucess or error response of application,In logs it shows nothing success or response of my application.It also dont show the endpoint of my application.It only shows service name and service environment and default logs but I can't see error logs or success logs.

---

<div class="post-metadata">

**Author:** ![achyutjhunjhunwala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/achyutjhunjhunwala/32/124654_2.png) [@achyutjhunjhunwala](https://discuss.elastic.co/u/achyutjhunjhunwala)\
**Post date:** [August 16, 2023, 9:10am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/8 "2023-08-16T09:10:58Z")

</div>

Can you go to the Discover tab and search for the exact document you are searching for (you can use the traceId to find the exact document like this `trace.id : "YOUR_TRACE_ID"`) and share it here by removing any confidential data ? We are interested in seeing the Keys mostly and if the required fields have some data.

---

<div class="post-metadata">

**Author:** ![vikascateina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikascateina/32/123762_2.png) [@vikascateina](https://discuss.elastic.co/u/vikascateina)\
**Post date:** [August 17, 2023, 5:02am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/9 "2023-08-17T05:02:48Z")

</div>

{  
"\_index": "xx",  
"\_type": "\_doc",  
"\_id": "",  
"\_version": 1,  
"\_score": 1,  
"\_source": {  
"system.process.cpu.total.norm.pct": 0.013282732447817837,  
"jvm.memory.heap.used": 267462136,  
"agent": {  
"name": "java",  
"ephemeral\_id": "xx",  
"version": "1.17.0"  
},  
"jvm.memory.non\_heap.committed": 307716096,  
"system.cpu.total.norm.pct": 0.03225806451612903,  
"observer": {  
"hostname": "xx",  
"id": "xx",  
"ephemeral\_id": "xx",  
"type": "apm-server",  
"version": "7.17.9",  
"version\_major": 7  
},  
"system.process.memory.size": xxx,  
"ecs": {  
"version": "1.12.0"  
},  
"system.memory.actual.free": 87781376,  
"jvm.memory.heap.committed": 368574464,  
"host": {  
"hostname": "xxx",  
"os": {  
"platform": "Linux"  
},  
"ip": "xxx",  
"name": "xxx",  
"architecture": "amd64"  
},  
"system.memory.total": 990019584,  
"event": {  
"ingested": "xxxx"  
},  
"jvm.thread.count": 128,  
"jvm.memory.non\_heap.used": 281465920,  
"process": {  
"pid": xxx,  
"title": "/etc/jdk8/jre/bin/java"  
},  
"jvm.memory.heap.max": 412614656,  
"jvm.gc.alloc": xxx,  
"processor": {  
"name": "metric",  
"event": "metric"  
},  
"metricset.name": "app",  
"@timestamp": "xxxx",  
"service": {  
"node": {  
"name": "x"  
},  
"environment": "xxx",  
"name": "xxxx",  
"runtime": {  
"name": "Java",  
"version": "1.8.0\_372"  
},  
"language": {  
"name": "Java",  
"version": "1.8.0\_372"  
},  
"version": "v1.0.0"  
},  
"jvm.memory.non\_heap.max": -1  
},  
"fields": {  
"system.process.cpu.total.norm.pct": [  
0.013  
],  
"jvm.memory.heap.used": [  
267462140  
],  
"jvm.memory.non\_heap.committed": [  
307716100  
],  
"service.node.name": [  
"xxxx"  
],  
"host.hostname": [  
"xxxx"  
],  
"system.cpu.total.norm.pct": [  
0.032  
],  
"process.pid": [  
3059  
],  
"service.language.name": [  
"Java"  
],  
"host.ip": [  
"xxx"  
],  
"system.process.memory.size": [  
3393204224  
],  
"processor.event": [  
"metric"  
],  
"jvm.memory.heap.committed": [  
368574460  
],  
"system.memory.actual.free": [  
87781376  
],  
"agent.name": [  
"java"  
],  
"system.memory.total": [  
xx  
],  
"host.name": [  
"xxxx"  
],  
"jvm.thread.count": [  
128  
],  
"service.environment": [  
"xxx"  
],  
"jvm.memory.non\_heap.used": [  
281465920  
],  
"jvm.memory.heap.max": [  
412614660  
],  
"service.name": [  
"xxxx"  
],  
"jvm.gc.alloc": [  
9667630000  
],  
"service.runtime.name": [  
"Java"  
],  
"processor.name": [  
"metric"  
],  
"service.runtime.version": [  
"1.8.0\_372"  
],  
"observer.version\_major": [  
7  
],  
"observer.hostname": [  
"xxx"  
],  
"host.architecture": [  
"amd64"  
],  
"metricset.name": [  
"app"  
],  
"observer.id": [  
"xxx"  
],  
"event.ingested": [  
"2023-08-17T03:42:14.993Z"  
],  
"@timestamp": [  
"2023-08-17T03:42:04.631Z"  
],  
"service.version": [  
"v1.0.0"  
],  
"observer.ephemeral\_id": [  
"xxxx"  
],  
"observer.version": [  
"7.17.9"  
],  
"host.os.platform": [  
"Linux"  
],  
"ecs.version": [  
"1.12.0"  
],  
"observer.type": [  
"apm-server"  
],  
"service.language.version": [  
"1.8.0\_372"  
],  
"jvm.memory.non\_heap.max": [  
-1  
],  
"agent.ephemeral\_id": [  
"xxxx"  
],  
"agent.version": [  
"1.17.0"  
],  
"process.title": [  
"/etc/jdk8/jre/bin/java"  
]  
}  
}

The above json is of my logs which is coming from my mule app.In the logs I want to see the co-relation Id ,api endpoint ,success logs or error logs which is coming from mule.I am able to see all this in my mule console but I am not getting all this in elasticsearch logs.

---

<div class="post-metadata">

**Author:** ![vikascateina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikascateina/32/123762_2.png) [@vikascateina](https://discuss.elastic.co/u/vikascateina)\
**Post date:** [August 17, 2023, 5:42am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/10 "2023-08-17T05:42:02Z")

</div>

{  
"correlationId" : "xxxx",  
"message" : "xxx",  
"tracePoint" : "END",  
"priority" : "INFO",  
"elapsed" : 4334,  
"locationInfo" : {  
"lineInFile" : "1773",  
"component" : "json-logger:logger",  
"fileName" : "xxxx",  
"rootContainer" : "xxxx"  
},  
"timestamp" : "2023-08-17T05:23:28.831Z",  
"content" : {  
"payload" : {  
"meta" : {  
"tracingId" : "_xxx\*\*x_-_xxx-\*\*xx-x_x\*-x_x_x\*\*\*\***x",  
"count" : "**"  
},  
"data" : [ {  
} ]  
}  
},  
"applicationName" : "xxxxx",  
"applicationVersion" : "1.0.3",  
"environment" : "dev",  
"threadName" : "xxxxxx"  
}

The above json is of my logs of my application which I am getting in my mule console,but I am not getting this logs in elasticsearch.I wanted the same logs to be seen in elasticsearch.

---

<div class="post-metadata">

**Author:** ![achyutjhunjhunwala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/achyutjhunjhunwala/32/124654_2.png) [@achyutjhunjhunwala](https://discuss.elastic.co/u/achyutjhunjhunwala)\
**Post date:** [August 17, 2023, 10:31am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/11 "2023-08-17T10:31:59Z")

</div>

Looking at the JSON, the document which you shared from Elasticsearch is a `Metrics` document (identified by the property `processor.name: metric`) and not a log document. Metrics documents are custom documents. You can find more information about them [here](https://www.elastic.co/guide/en/apm/guide/current/data-model-metrics.html#:~:text=Metrics%20measure%20the%20state%20of,visualizations%20in%20the%20APM%20app.)

Logs are generally stored on a `log-*` index in Elasticsearch. Can you try navigating to Discover in Kibana -\> Then change the Data view on the Top Left to the `logs-*` dataview and then search for your application log.

---

<div class="post-metadata">

**Author:** ![vikascateina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikascateina/32/123762_2.png) [@vikascateina](https://discuss.elastic.co/u/vikascateina)\
**Post date:** [August 17, 2023, 10:38am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/12 "2023-08-17T10:38:51Z")

</div>

![Screenshot from 2023-08-17 16-05-32](https://us1.discourse-cdn.com/elastic/original/3X/9/2/9270f2ae71435dcc0232571b2f131ed054552adf.png)  
There is nothing in logs as you can see in screenshot.

---

<div class="post-metadata">

**Author:** ![achyutjhunjhunwala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/achyutjhunjhunwala/32/124654_2.png) [@achyutjhunjhunwala](https://discuss.elastic.co/u/achyutjhunjhunwala)\
**Post date:** [August 17, 2023, 10:58am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/13 "2023-08-17T10:58:17Z")

</div>

Just to be sure if i am seeing this correctly, can you try changing the time range which is currently set to today to a longer period, like 1 week or more and confirm.

---

<div class="post-metadata">

**Author:** ![vikascateina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikascateina/32/123762_2.png) [@vikascateina](https://discuss.elastic.co/u/vikascateina)\
**Post date:** [August 17, 2023, 11:05am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/14 "2023-08-17T11:05:27Z")

</div>

![Screenshot from 2023-08-17 16-29-59](https://us1.discourse-cdn.com/elastic/original/3X/4/1/41cb8540152913036344f1f2304d480c6d86eb86.png)  
The logs is not comming here but logs from apm is coming which I shared before i.e in index traces-apm\*,apm-_,logs-apm_,apm-_,metrics-apm_,apm-\*. which is comming from apm I am able to see traces.but not logs

---

<div class="post-metadata">

**Author:** ![GregKalapos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregkalapos/32/37205_2.png) [@GregKalapos](https://discuss.elastic.co/u/GregKalapos)\
**Post date:** [August 17, 2023, 12:14pm UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/15 "2023-08-17T12:14:51Z")

</div>

Hi @vikascateina

I work in the APM Agents team.

The [mule4 agent](https://github.com/michaelhyatt/elastic-apm-mule4-agent) is not an official agent and it isn't maintained and developed by the elastic team. It's built on top of our Java Agent (which we officially do support and maintain). Having said that, we try to help you.

The latest mule4 agent is already fairly old - [last release](https://github.com/michaelhyatt/elastic-apm-mule4-agent/releases) is from November 2020 if I see it correctly.

Now.. back to logs and indices: the agent itself is going to ship traces. You say:

> logs from apm is coming which I shared before i.e in index traces-apm\*,apm-_,logs-apm_ ,apm-_,metrics-apm_ ,apm-\*. which is comming from apm I am able to see traces.but not logs

I assume you mean you see documents in in one of the `traces` indices - those are traces captured by the agent. Please note we typically don't call those "logs" - this is APM related data captured by an APM Agent (e.g. it represents an incoming HTTP call, a database call, or some other similar things).

Now.. there was a log shipping feature introduced by the Java agent in version `1.36.0` released on 2023/01/27 (see [release notes](https://www.elastic.co/guide/en/apm/agent/java/current/release-notes-1.x.html#_features_7)). That agent ships with a feature that'll automatically send application logs. In other words the Elastic Java APM Agent will pick up your application logs and send it APM Server which will then store those in Elasticsearch. This feature is opt-in: [Logging configuration options | APM Java Agent Reference [1.x] | Elastic](https://www.elastic.co/guide/en/apm/agent/java/current/config-logging.html#config-log-sending)

Now, the problem is probably that the Java agent used in the mule4 is an older one and does not contain this feature. So that agent version very likely by default won't send any application logs.

As said, the mule4 agent isn't maintained by us, so we can't really do much unfortunately.

One thing you can do is to maybe look into manually ingesting those logs. We have a documentation on it: [Application logs | Elastic Observability [8.9] | Elastic](https://www.elastic.co/guide/en/observability/8.9/application-logs.html)

---

<div class="post-metadata">

**Author:** ![vikascateina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikascateina/32/123762_2.png) [@vikascateina](https://discuss.elastic.co/u/vikascateina)\
**Post date:** [August 17, 2023, 1:17pm UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/16 "2023-08-17T13:17:20Z")

</div>

Hi , I have shared the dependency which I am using to push the logs from mule to elasticsearch and I have updated log4j file.How do I push application logs in elasticsearch instead of apm data.

```
   <dependency>
		<groupId>co.elastic.apm</groupId>
		<artifactId>mule4-agent</artifactId> 
		<version>0.4.0</version>
	</dependency>
	<dependency>
		<groupId>co.elastic.apm</groupId>
		<artifactId>apm-agent-attach</artifactId>
		<version>1.17.0</version>
	</dependency>
	<dependency>
		<groupId>co.elastic.apm</groupId>
		<artifactId>apm-agent-api</artifactId>
		<version>1.21.0</version>
	</dependency>

```

---

<div class="post-metadata">

**Author:** ![GregKalapos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregkalapos/32/37205_2.png) [@GregKalapos](https://discuss.elastic.co/u/GregKalapos)\
**Post date:** [August 17, 2023, 1:50pm UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/17 "2023-08-17T13:50:48Z")

</div>

Yeah, seeing those versions verifies my assumptions - this is an old agent which does not have the feature to ship logs.

> [@vikascateina](#):
>
> How do I push application logs in elasticsearch instead of apm data.

That is described in the doc I shared above: [Application logs | Elastic Observability [8.9] | Elastic](https://www.elastic.co/guide/en/observability/8.9/application-logs.html)

Primarily you'll need to use Filebeat.

---

<div class="post-metadata">

**Author:** ![vikascateina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikascateina/32/123762_2.png) [@vikascateina](https://discuss.elastic.co/u/vikascateina)\
**Post date:** [August 18, 2023, 3:53am UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/18 "2023-08-18T03:53:19Z")

</div>

I am using elasticsearch,kibana and apm.For Application logs to push in elasticsearch what will be best logstash or filebeat ?

---

<div class="post-metadata">

**Author:** ![GregKalapos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregkalapos/32/37205_2.png) [@GregKalapos](https://discuss.elastic.co/u/GregKalapos)\
**Post date:** [August 21, 2023, 3:06pm UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/19 "2023-08-21T15:06:08Z")

</div>

If you just want to ingest your logs, then filebeat will be likely the easiest and fastest way to set this up.

---

<div class="post-metadata">

**Author:** ![vikascateina](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikascateina/32/123762_2.png) [@vikascateina](https://discuss.elastic.co/u/vikascateina)\
**Post date:** [August 22, 2023, 1:27pm UTC](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421/20 "2023-08-22T13:27:02Z")

</div>

Hi,  
I have implemented logstash but in my index management I am not able to see Indices logstash but Ingest pipeline is created as shown in image.

 ![Screenshot from 2023-08-22 18-49-42](https://us1.discourse-cdn.com/elastic/original/3X/7/7/77b311a47b33d05f3692b3e2b9aac2e90dd9372a.png)  
I want to push my mule application logs to logstash.How can I do it.

[Next page](https://discuss.elastic.co/t/not-able-to-see-the-traces-of-my-applicaton-mule4/340421.md?page=2)