Hello everyone,
we are currently using Elastic Stack, Kibana etc in the version of 8.5.2 and are facing a weird issue with the logs View in the APM Views.
If it matters, we are sending the logs with their transaction correlation via .net Serilog Enricher.
Lets say we have the applications/apm services A and B and both are logging to Elastic.
In our scenario each of them are running on the hosts Y and Z. So in this example there are 4 instances, 2 of each.
Now when we go to the Logs-View inside of an APM Service, we have the following behaviour:
In the Logs-View of App A we can see: All Logs of A and some of B.
Same for the Logs-View of App B. All Logs of B and some of A.
The logs that appear in the wrong view are not connected to the other service by any means. They have no communication with each other (one thought would be, that the trace goes through both services). When I search for the transaction id of a wrong log entry in View of B, Kibana correctly brings me to the transaction of service A. So that cannot be it.
I was able to reproduce the problem with hosting App A with a nonrelated App C on my local machine with a completely separate Environment-Variable set. Logs of App A started to show up in App C and vice versa.
In this github discussion i read something about some magic happening with the host.name. Could that be it?
https://github.com/elastic/kibana/issues/106093#issuecomment-884288377
Either way, i feel like this behaviour makes no sense at all and doesnt allow us to host services on the same machine. The only way around this would be to insert the Service-Name into the host.name field before sending it to Elastic, but that doesnt feel right and is cumbersome.
If you need any more information, i`ll be happy to provide them.
Thanks in Advance