Using version 7.15.2 of Elasticsearch, Kibana and APM. I have APM configured and can see transactions/traces. I have Serilog configured with the Elasticsearch sink and I am using the EcsTextFormatter.
When I go to Logs > Stream in Kibana, I can see the logs being generated by my application, and I can see that the logs include the expected metadata: metadata.elastic_apm_trace_id, metadata.elastic_apm_transaction_id, trace.id and transaction.id.
I can copy a trace.id from Logs, go to APM and find the trace, but when I go to APM and click on the Logs tab I do not see any logs, ever, regardless of the time range. I am using a custom index name for my logs. I have tried to adjust the "Error indices" configuration in APM settings to include my log pattern, but it did not change anything, also I believe this configuration controls the content of the "Errors" tab, not the "Logs" tab for APM. Do I have to change my log indices to start with logs-? Why is there no configuration in APM for the logs indices like there is for the Logs feature?
Curious can you validate you also have service.name field in those logs?
Also What Agent Language and version?
And can you check if you drill all the way down to an individual transaction / trace
Can you see the logs there (Not at the top Level)
I don't see service.name, what do I need to do to populate this in the log entries?
Yes, same, I see the logs when I drill down to the individual transaction, but not at the top level logs.
One project is .NET Full Framework 4.7.2 with nuget version 1.11.1 of:
The other project is .NET 5.0 with nuget packages version 1.11.1 of:
For Me the service.name is automatically included with the Java ECS Logger.. looks like it is not available yet with the Serilog see here
I am not sure it is absolutely required anyways for this, I think not.
I am not .NET expert, my agent is Java but looks the same no Logs Top Service Level But Logs at the detail transaction / trace level. I will need to poke around but I am probably done for the night, and out next week.
Update I pinged some folks internally see what happens..
Update 2: I may have found something.... it may have to do with the service.name
Update it has something to do with the container.id (or perhaps host.name if you are not containerized) which is the filter it is acting on I am not sure why. I will need to ask internally. If you inject that into your logs I bet that will work.
If you go into the browser Dev Tools and dig deep enough you will see that... I am not sure of the answer but I want to find out, as my Java App is not working as well.
Here is a snapshot from a working demo .. it is filtering on container.id (s) I know that is hard to see
Interesting yes .. I am not familiar with the serilog (more of a java guy) but I think this is a good example of what we need to consider.. at the very least make it clear which fields are required in the document or error messages.