Hey there @Sim_Singh, thanks for bringing this up.
The code you linked is related to a new feature in the Agent where we attempt to collect metadata about the local cloud environments (Azure, GCP, AWS, etc.) via. those service's cloud metadata endpoints (ex. AWS Metadata) and then report this data to your ElasticSearch instance along with other APM metrics and data.
CallbackCoordination class where that error originates is responsible for coordinating the three network requests to each service's endpoint. The specific codepath you've hit is a fallback where, if those service's endpoints fail to respons within a certain amount of time, we give up trying to fetch that data.
The error you're seeing logged ("metadata requests did not finish, possible deadlock") should not be a fatal error. It should be logged and then the agent startup should proceed. If it's preventing your application from starting that's exactly the sort of bug we want to know about and fix.
First -- is the above the sort of context you were looking for and does it help? Or did we miss the mark?
Second, two things that may help you work around this
Change the cloudProvider value to match the cloud provider whose service you're on, or change it to
none. This may help alleviate the problem
Temporarily fallback to using an older version of the agent without this metadata fetching functionality
Finally, could you let us know a little bit about your K8s environment? Are you running this on a service provided by one of the cloud vendors we mentioned, or is it in a home grown k8s that you're running? (or, is it home grown K8s BUT running on a cloud provider's infrastructure). We didn't see the behavior you're describing when we were testing this feature, so if it IS a fatal error for you we may have missed something that's unique to your environment. The more we know about the better we can help you diagnose the issue.