Nginx ingress controller integration is not working

[Subrahmanyam_Veerank (Subramanyam v)]

Hi,

we have enabled the kubernetes and nginx ingress controller integration on the elastic agent [managed with fleet] but the access & error logs are not coming and getting the below errors.

04:19.949

elastic_agent

[elastic_agent][error] Unit state changed filestream-default-filestream-container-logs-9ddb18cb-f703-42b1-8593-e78dd99e6134-kubernetes-08d5e576-8558-4339-84a9-0e9891aba20d.speaker (HEALTHY->FAILED): Failed: pid '2393669' exited with code '-1'

14:04:19.949

elastic_agent

[elastic_agent][error] Unit state changed filestream-default-filestream-container-logs-9ddb18cb-f703-42b1-8593-e78dd99e6134-kubernetes-a3aeb38c-4238-4b97-bf58-aca3b6399574.nginx-ingress-controller (HEALTHY->FAILED): Failed: pid '2393669' exited with code '-1'

14:04:19.949

elastic_agent

[elastic_agent][error] Unit state changed filestream-default-filestream-container-logs-9ddb18cb-f703-42b1-8593-e78dd99e6134-kubernetes-0872281d-6d03-41e4-bccc-3783fe4a2c72.exporter-node (HEALTHY->FAILED): Failed: pid '2393669' exited with code '-1'

14:04:19.949

elastic_agent

[elastic_agent][error] Unit state changed filestream-default-filestream-container-logs-9ddb18cb-f703-42b1-8593-e78dd99e6134-kubernetes-2dd1b484-9b37-4971-8c2f-3bc5a3228a47.node-driver-registrar (HEALTHY->FAILED): Failed: pid '2393669' exited with code '-1'

14:04:19.949

elastic_agent

[elastic_agent][error] Unit state changed filestream-default-filestream-nginx_ingress_controller-d7276a9b-074b-4bc1-b4e8-368450bfe89f-kubernetes-4090c0b7-3f69-4daa-a4ad-11d6bd623a94.controller (HEALTHY->FAILED): Failed: pid '2393669' exited with code '-1'

14:04:19.949

elastic_agent

[elastic_agent][error] Unit state changed filestream-default (HEALTHY->FAILED): Failed: pid '2393669' exited with code '-1'

16:47:33.244

elastic_agent

[elastic_agent][error] checkin retry loop was stopped

16:47:33.244

elastic_agent

[elastic_agent][error] checkin retry loop was stopped

[Subrahmanyam_Veerank (Subramanyam v)]

11:20:05.137

elastic_agent.filebeat

[elastic_agent.filebeat][error] failed to publish events: Post "https://189.20.87.60:9200/_bulk?filter_path=errors%2Citems.%2A.error%2Citems.%2A.status": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

11:20:08.262

elastic_agent.filebeat

[elastic_agent.filebeat][error] failed to perform any bulk index operations: Post "https://189.20.87.61:9200/_bulk?filter_path=errors%2Citems.%2A.error%2Citems.%2A.status": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

11:20:10.004

elastic_agent.filebeat

[elastic_agent.filebeat][error] failed to publish events: Post "https://189.20.87.61:9200/_bulk?filter_path=errors%2Citems.%2A.error%2Citems.%2A.status": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

11:22:40.768

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting openstack metadata: Get "http://169.254.169.254/2009-04-04/meta-data/instance-id": dial tcp 169.254.169.254:80: i/o timeout

11:22:40.769

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting digitalocean metadata: Get "http://169.254.169.254/metadata/v1.json": dial tcp 169.254.169.254:80: i/o timeout

11:22:40.773

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting hetzner metadata: Get "http://169.254.169.254/hetzner/v1/metadata/region": dial tcp 169.254.169.254:80: i/o timeout

11:22:40.776

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting openstack metadata: Get "https://169.254.169.254/2009-04-04/meta-data/placement/availability-zone": dial tcp 169.254.169.254:443: i/o timeout

11:22:40.777

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting azure metadata: Get "http://169.254.169.254/metadata/instance/compute?api-version=2021-02-01": dial tcp 169.254.169.254:80: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting openstack metadata: Get "http://169.254.169.254/2009-04-04/meta-data/instance-id": dial tcp 169.254.169.254:80: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting digitalocean metadata: Get "http://169.254.169.254/metadata/v1.json": dial tcp 169.254.169.254:80: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting hetzner metadata: Get "http://169.254.169.254/hetzner/v1/metadata/region": dial tcp 169.254.169.254:80: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting azure metadata: Get "http://169.254.169.254/metadata/instance/compute?api-version=2021-02-01": dial tcp 169.254.169.254:80: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting openstack metadata: Get "https://169.254.169.254/2009-04-04/meta-data/instance-id": dial tcp 169.254.169.254:443: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting gcp metadata: Get "http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json": dial tcp 169.254.169.254:80: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting huawei metadata: Get "http://169.254.169.254/openstack/latest/meta_data.json": dial tcp 169.254.169.254:80: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting openstack metadata: Get "http://169.254.169.254/2009-04-04/meta-data/instance-id": dial tcp 169.254.169.254:80: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting digitalocean metadata: Get "http://169.254.169.254/metadata/v1.json": dial tcp 169.254.169.254:80: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting hetzner metadata: Get "http://169.254.169.254/hetzner/v1/metadata/region": dial tcp 169.254.169.254:80: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting azure metadata: Get "http://169.254.169.254/metadata/instance/compute?api-version=2021-02-01": dial tcp 169.254.169.254:80: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting openstack metadata: Get "https://169.254.169.254/2009-04-04/meta-data/instance-id": dial tcp 169.254.169.254:443: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting gcp metadata: Get "http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json": dial tcp 169.254.169.254:80: i/o timeout

11:27:13.752

elastic_agent.filebeat

[elastic_agent.filebeat][error] add_cloud_metadata: received error failed requesting huawei metadata: Get "http://169.254.169.254/openstack/latest/meta_data.json": dial tcp 169.254.169.254:80: i/o timeout

[Subrahmanyam_Veerank (Subramanyam v)]

@stephenb sir..If you have time, kindly look into it.

[stephenb (Stephen Brown)]

Please @Subrahmanyam_Veerank

I have asked that you do not @ mention people directly into your Topics unless they have already joined your topic... Especially on a brand new topic only minutes after you posted the original.. this is not proper etiquette!

Your questions are no more important than any other community member.

That said and this will be the last time I respond to an unsolicited @

Perhaps increase the resource for the agent in your manifest...

The add_cloud_metadata is not a issue... you can take out that processor if you like...

[Subrahmanyam_Veerank (Subramanyam v)]

if you mentioned that @mention people is not ok then i wouldn't have done that. Yes my questions are no more important as we are not the licensed users and using the elastic recently. whenever i have raised any issue im greeting you as "sir". So if you do not want answer my query you can just leave. The reply from you is also not a proper etiquette!!!!

[stephenb (Stephen Brown)]

@Subrahmanyam_Veerank

Apologies I confused you with another user that I had a direct conversation with and explained the case about not mentioning people not already in the topic.

I interact with many many users.

It was my mistake, I apologize

And As you know I have taken the time and effort to politely answered many of your questions, and yes you are polite as well.

We would ask your patients as there are many questions, and now you know we prefer not to @ mention people that are not already part of your topic

That said let's get back on track....And see if we can help...

It is unclear what the issue is

There is documentation on troubleshooting

Troubleshoot common problems | Fleet and Elastic Agent Guide [8.12] | Elastic

And

Frequently asked questions | Fleet and Elastic Agent Guide [8.12] | Elastic

You can the diagnostics and inspect commands

Elastic Agent command reference | Fleet and Elastic Agent Guide [8.12] | Elastic

And as I suggested you might look at increasing the resources for the agent in the manifest.

Perhaps look here

Kubernetes deployed Elastic Agent is looped in beat restart

Hi @Alphayeeeet Welcome to the community. This is often caused by an out-of-memory issue. You might want to bump up the resources for the agent. Also, I am going to close the other Topic as this is a fine / close enough category.

Those are my first suggestions