Logs Do Not Appear In APM, Although Correlation Info Is Present

[Brandon_Liles]

Using version 7.15.2 of Elasticsearch, Kibana and APM. I have APM configured and can see transactions/traces. I have Serilog configured with the Elasticsearch sink and I am using the EcsTextFormatter.

When I go to Logs > Stream in Kibana, I can see the logs being generated by my application, and I can see that the logs include the expected metadata: metadata.elastic_apm_trace_id, metadata.elastic_apm_transaction_id, trace.id and transaction.id.

I can copy a trace.id from Logs, go to APM and find the trace, but when I go to APM and click on the Logs tab I do not see any logs, ever, regardless of the time range. I am using a custom index name for my logs. I have tried to adjust the "Error indices" configuration in APM settings to include my log pattern, but it did not change anything, also I believe this configuration controls the content of the "Errors" tab, not the "Logs" tab for APM. Do I have to change my log indices to start with logs-? Why is there no configuration in APM for the logs indices like there is for the Logs feature?

[stephenb (Stephen Brown)]

What do you have set here?

Interesting I may be seeing an issue too.. I will take a look.

[Brandon_Liles]

This is the custom index pattern for my log events which is indices named system-events*

[stephenb (Stephen Brown)]

Curious can you validate you also have service.name field in those logs?
Also What Agent Language and version?
And can you check if you drill all the way down to an individual transaction / trace
Can you see the logs there (Not at the top Level)

Right now I am seeing my logs there but not here... strange....

[Brandon_Liles]

I don't see service.name, what do I need to do to populate this in the log entries?

Yes, same, I see the logs when I drill down to the individual transaction, but not at the top level logs.

One project is .NET Full Framework 4.7.2 with nuget version 1.11.1 of:
Elastic.Apm
Elastic.Apm.AspNetFullFramework
Elastic.Apm.Elasticsearch
Elastic.Apm.StackExchange.Redis
Elastic.Apm.SerilogEnricher v1.5.3
Elastic.CommonSchema.Serilog v1.5.3

The other project is .NET 5.0 with nuget packages version 1.11.1 of:
Elastic.Apm.NetCoreAll
Elastic.Apm.StackExchange.Redis
Elastic.Apm.SerilogEnricher v1.5.3
Elastic.CommonSchema.Serilog v1.5.3

[stephenb (Stephen Brown)]

For Me the service.name is automatically included with the Java ECS Logger.. looks like it is not available yet with the Serilog see here

I am not sure it is absolutely required anyways for this, I think not.

I am not .NET expert, my agent is Java but looks the same no Logs Top Service Level But Logs at the detail transaction / trace level. I will need to poke around but I am probably done for the night, and out next week.

Update I pinged some folks internally see what happens..

Update 2: I may have found something.... it may have to do with the service.name

[stephenb (Stephen Brown)]

Update it has something to do with the container.id (or perhaps host.name if you are not containerized) which is the filter it is acting on I am not sure why. I will need to ask internally. If you inject that into your logs I bet that will work.

If you go into the browser Dev Tools and dig deep enough you will see that... I am not sure of the answer but I want to find out, as my Java App is not working as well.

Here is a snapshot from a working demo .. it is filtering on container.id (s) I know that is hard to see

Blow Up

[stephenb (Stephen Brown)]

@Brandon_Liles
I got and answer..

In short it is based on host.name or container.id in the logs

See Here for a detailed explanation... sound like there might be some additional discussion.

I added container.id and it worked for me.

[APM] Create a Logs tab in the Service view to show Logs per service

[Brandon_Liles]

Thanks so much for your assistance! I can confirm making sure that host.name is set causes the logs to be populated, but this raises several questions for me:

1. Why doesn't the EcsTextFormatter/ElasticApmCorrelationInfo combination automatically add this data?
2. What about a clustered environment where there are multiple hosts?

We are working towards containerizing our application as part of a migration to .Net 6.0, but we still have part of our app running on .Net 4.7.2 and Windows VMs.

[stephenb (Stephen Brown)]

Good question.

If you use filebeat to collect the and ship the logs "most" of the time filebeat if properly configured will add the host.name or container.id

Filebeat will add container.ud automatically for many containerized / clustered env such as K8s

That is my understanding of thinking at this point.

That said, Seems like there still may be some discussion on that.

[Brandon_Liles]

Interesting. Right now I'm shipping logs with the Serilog Elasticsearch sink, not writing anything to file.

[stephenb (Stephen Brown)]

Interesting yes .. I am not familiar with the serilog (more of a java guy) but I think this is a good example of what we need to consider.. at the very least make it clear which fields are required in the document or error messages.

[Martijn_Laarman]

Thanks for highlighting this @Brandon_Liles this is an omission from the enricher!

I opened [FEATURE] Service name should by default be included to structured logs · Issue #170 · elastic/ecs-dotnet · GitHub to start work on making sure the default experience is better.

As for why, the enricher predates the log correlation feature so the .NET ecs logging integration needs to catch up again.

[Brandon_Liles]

That sounds great, appreciate what a great product you've made and continue to improve upon.