I have setup a logstash service in AKS cluster. The service has a external IP address with port 443. I am sending logs from windows machine using filebeat. It works when i am running filebeat on my local machine. I can see logstash service receiving the logs( by looking at logstash service logs).
However when i am trying to send the logs from a Virtual machine, i am getting these error.
I am able to telnet the logstash service from the VM.
{"log.level":"warn","@timestamp":"2024-10-03T15:18:35.105Z","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.
(*azureMetadataFetcher).fetchMetadata","file.name":"add_cloud_metadata/provider_azure_vm.go","file.line":142},"message":"Failed to get additional AKS Cluster meta: failed to get AKS cluster name and ID: failed to advance page: GET https://management.azure.com/subscriptions/9d7acea9-fd8e-4520-9f62-c687428226ec/providers/Microsoft.ContainerService/managedClusters\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client '712b84ae-fa18-443f-8f13-922e92ef888a' with object id '712b84ae-fa18-443f-8f13-922e92ef888a' does not have authorization to perform action 'Microsoft.ContainerService/managedClusters/read' over scope '/subscriptions/9d7acea9-fd8e-4520-9f62-c687428226ec' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n","service.name":"filebeat","ecs.version":"1.6.0"}
I am also seeing these errors:
"log.level":"error","@timestamp":"2024-10-03T14:24:44.308Z","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata","file.name":"add_cloud_metadata/providers.go","file.line":190},"message":"add_cloud_metadata: received error failed fetching EC2 Identity Document: operation error ec2imds: GetInstanceIdentityDocument, failed to get API token, operation error ec2imds: getToken, http response error StatusCode: 400, request to EC2 IMDS failed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-03T14:24:44.309Z","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata","file.name":"add_cloud_metadata/providers.go","file.line":190},"message":"add_cloud_metadata: received error failed with http status code 404","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-03T14:24:44.309Z","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata","file.name":"add_cloud_metadata/providers.go","file.line":190},"message":"add_cloud_metadata: received error failed with http status code 404","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-03T14:24:44.309Z","log.logger":"add_cloud_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata.(*addCloudMetadata).fetchMetadata","file.name":"add_cloud_metadata/providers.go","file.line":190},"message":"add_cloud_metadata: received error failed with http status code 404","service.name":"filebeat","ecs.version":"1.6.0"}
What could be wrong here?
Many thanks.