AWS logging integration error

Elastic Stack Elastic Agent
1

Hi,
I want to integrate the AWS logs into my Kibana Observability log stream.
Here is the access policy of SQS queue:

{
  "Version": "2012-10-17",
  "Id": "arn:aws:sqs:us-east-1:334811116626:ingest-ec2-logs-sqs/SQSDefaultPolicy",
  "Statement": [
    {
      "Sid": "Sid334811116626",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": [
        "sqs:SendMessage",
        "sqs:ReceiveMessage"
      ],
      "Resource": "arn:aws:sqs:us-east-1:334811116626:ingest-ec2-logs-sqs"
    }
  ]
}

I set the SQS queue as the S3 event notification destination. Here is the S3 bucket event notification setting:

image
image2853×1094 234 KB

I also configured integration in Kibana. I entered the SQS queue URL under the "Collect EC2 logs from S3" category.
image
image2060×725 62.1 KB

I installed elastic agent on my elasticsearch server. The data stream in Kibana is empty. Here is the error in the elastic agent log:

{"log.level":"error","@timestamp":"2023-09-20T13:57:02.888Z","message":"failed to perform any bulk index operations: 500 Internal Server Error: {\"error\":{\"root_cause\":[{\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"}],\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"},\"status\":500}","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"elasticsearch","log.origin":{"file.line":258,"file.name":"elasticsearch/client.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-20T13:57:02.979Z","message":"failed to publish events: 500 Internal Server Error: {\"error\":{\"root_cause\":[{\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"}],\"type\":\"illegal_state_exception\",\"reason\":\"There are no ingest nodes in this cluster, unable to forward request to an ingest node.\"},\"status\":500}","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":174,"file.name":"pipeline/client_worker.go"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

How can I fix this issue?
Thanks

2

This is your error, it says that your cluster does not have any node with the ingest role.

Can you share the response of the following request in Kibana Dev Tools?

GET _cat/nodes?s=name
3

I have two nodes in the elasticsearch cluster.
10.0.9.12 46 96 2 1.51 2.36 1.93 dm * cvelk5
10.0.9.13 6 97 2 0.81 2.25 1.98 dm - cvelk6
And this is the output configuration in elastic-agent.yml:

outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://10.0.9.12:9200'
    ssl.ca_trusted_fingerprint: f5bed68b70e82d4df44168279aa67bcf127577a76e3e7b57f53855827e2d2964
    username: 'elastic'
    password: '***'
4

Yeah, none of the nodes has the ingest role, you will need to add the ingest role as a role for at least one of the nodes in elasticsearch.yml and restart the node.

A node with the ingest role is required for the integrations to work.

You can check here for the documentation about node roles.

5

Thank you. I set the nodes as master nodes, because I thougt they already have got the full functions.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.