Missing field from my sqs queue

Elastic Stack Beats
1

We are running Elasticsearch on Kubernetes using the Elastic Cloud Operator.
We use the input sqs plugin to pull events for our SQS queues
Recently, we discovered the logs from one of our queues have the field aws.sqs.oldest_message_age.sec missing. In some of the log messages, that field is supposed to be a number by it has text like "a few seconds".
Other queues do not seem to have this problem.
This is what the sqs manifest looks like:

input:
  module: aws
  metricset: cloudwatch
  defaults:
    metrics:
      - namespace: AWS/SQS
        resource_type: sqs
        statistic: ["Average"]
        name:
          - ApproximateAgeOfOldestMessage
          - ApproximateNumberOfMessagesDelayed
          - ApproximateNumberOfMessagesNotVisible
          - ApproximateNumberOfMessagesVisible
          - NumberOfMessagesDeleted
          - NumberOfMessagesReceived
          - NumberOfMessagesSent
          - NumberOfEmptyReceives
          - SentMessageSize
processors:
  - rename:
      ignore_missing: true
      fields:
        - from: "aws.sqs.metrics.ApproximateAgeOfOldestMessage.avg"
          to: "aws.sqs.oldest_message_age.sec"
        - from: "aws.sqs.metrics.ApproximateNumberOfMessagesDelayed.avg"
          to: "aws.sqs.messages.delayed"
        - from: "aws.sqs.metrics.ApproximateNumberOfMessagesNotVisible.avg"
          to: "aws.sqs.messages.not_visible"
        - from: "aws.sqs.metrics.ApproximateNumberOfMessagesVisible.avg"
          to: "aws.sqs.messages.visible"
        - from: "aws.sqs.metrics.NumberOfMessagesDeleted.avg"
          to: "aws.sqs.messages.deleted"
        - from: "aws.sqs.metrics.NumberOfMessagesReceived.avg"
          to: "aws.sqs.messages.received"
        - from: "aws.sqs.metrics.NumberOfMessagesSent.avg"
          to: "aws.sqs.messages.sent"
        - from: "aws.sqs.metrics.NumberOfEmptyReceives.avg"
          to: "aws.sqs.empty_receives"
        - from: "aws.sqs.metrics.SentMessageSize.avg"
          to: "aws.sqs.sent_message_size.bytes"

  - drop_fields:
      ignore_missing: true
      fields:
        - "aws.sqs.metrics"

Has anyone else faced this problem and are there any thoughts on what might be failing?

2

Hey @nexus1, welcome to discuss :slight_smile:

Have you tried to use the sqs metricset instead of the cloudwatch one? Though it seems to use a similar configuration to yours.

3

Btw, where do you see this text? In the metrics document itself or somewhere in kibana?

4

Thanks, Jaime.
We are currently using the sqs metricset, actually.
This is our metricbeat deployment:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: metricbeat-deployment
  namespace: ***-sqs-metricbeat
  labels:
    app: metricbeat
spec:
  type: metricbeat
  version: 7.17.3
  elasticsearchRef:
    name: <name>
    namespace: <namespace_name>
  kibanaRef:
    name: <name>
    namespace: <namespace_name>
  config:
    setup.template.name: "metricbeat-sqs-metrics"
    setup.template.overwrite: "true"
    setup.dashboards.index: "metricbeat-sqs-metrics-*"
    setup.ilm.enabled: auto
    setup.ilm.rollover_alias: "metricbeat-sqs-metrics"
    setup.ilm.pattern: "{now/d}-000001"
    metricbeat:
      modules:
        - module: aws
          period: 60s
          regions: us-east-1
          metricsets:
          - sqs
          - kinesis
          role_arn: arn:aws:iam::***:role/***-sqs-metricbeat
    logging:
      json: true
  deployment:
    podTemplate:
      spec:
        nodeSelector:
          k8s.clio.com/node-usage: default
        securityContext:
          runAsUser: 0

Any idea what we might be doing wrong?

5

We see "a few seconds" in the in the "ApproximateAgeOfOldestMessage" field on Kibana.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.