# Missing field from my sqs queue

**URL:** <https://discuss.elastic.co/t/missing-field-from-my-sqs-queue/312746>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [August 23, 2022, 9:42pm UTC](https://discuss.elastic.co/t/missing-field-from-my-sqs-queue/312746 "2022-08-23T21:42:40Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![nexus1](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@nexus1](https://discuss.elastic.co/u/nexus1)\
**Post date:** [August 23, 2022, 9:42pm UTC](https://discuss.elastic.co/t/missing-field-from-my-sqs-queue/312746/1 "2022-08-23T21:42:40Z")

</div>

We are running Elasticsearch on Kubernetes using the Elastic Cloud Operator.  
We use the input sqs plugin to pull events for our SQS queues  
Recently, we discovered the logs from one of our queues have the field `aws.sqs.oldest_message_age.sec` missing. In some of the log messages, that field is supposed to be a number by it has text like "a few seconds".  
Other queues do not seem to have this problem.  
This is what the sqs manifest looks like:

```auto
input:
  module: aws
  metricset: cloudwatch
  defaults:
    metrics:
      - namespace: AWS/SQS
        resource_type: sqs
        statistic: ["Average"]
        name:
          - ApproximateAgeOfOldestMessage
          - ApproximateNumberOfMessagesDelayed
          - ApproximateNumberOfMessagesNotVisible
          - ApproximateNumberOfMessagesVisible
          - NumberOfMessagesDeleted
          - NumberOfMessagesReceived
          - NumberOfMessagesSent
          - NumberOfEmptyReceives
          - SentMessageSize
processors:
  - rename:
      ignore_missing: true
      fields:
        - from: "aws.sqs.metrics.ApproximateAgeOfOldestMessage.avg"
          to: "aws.sqs.oldest_message_age.sec"
        - from: "aws.sqs.metrics.ApproximateNumberOfMessagesDelayed.avg"
          to: "aws.sqs.messages.delayed"
        - from: "aws.sqs.metrics.ApproximateNumberOfMessagesNotVisible.avg"
          to: "aws.sqs.messages.not_visible"
        - from: "aws.sqs.metrics.ApproximateNumberOfMessagesVisible.avg"
          to: "aws.sqs.messages.visible"
        - from: "aws.sqs.metrics.NumberOfMessagesDeleted.avg"
          to: "aws.sqs.messages.deleted"
        - from: "aws.sqs.metrics.NumberOfMessagesReceived.avg"
          to: "aws.sqs.messages.received"
        - from: "aws.sqs.metrics.NumberOfMessagesSent.avg"
          to: "aws.sqs.messages.sent"
        - from: "aws.sqs.metrics.NumberOfEmptyReceives.avg"
          to: "aws.sqs.empty_receives"
        - from: "aws.sqs.metrics.SentMessageSize.avg"
          to: "aws.sqs.sent_message_size.bytes"

  - drop_fields:
      ignore_missing: true
      fields:
        - "aws.sqs.metrics"

```

Has anyone else faced this problem and are there any thoughts on what might be failing?

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 25, 2022, 8:17am UTC](https://discuss.elastic.co/t/missing-field-from-my-sqs-queue/312746/2 "2022-08-25T08:17:40Z")

</div>

Hey @nexus1, welcome to discuss 🙂

Have you tried to use the [`sqs` metricset](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-aws-sqs.html) instead of the `cloudwatch` one? Though it seems to use a similar configuration to yours.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 25, 2022, 8:30am UTC](https://discuss.elastic.co/t/missing-field-from-my-sqs-queue/312746/3 "2022-08-25T08:30:16Z")

</div>

> [@nexus1](#):
>
> it has text like "a few seconds"

Btw, where do you see this text? In the metrics document itself or somewhere in kibana?

---

<div class="post-metadata">

**Author:** ![nexus1](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@nexus1](https://discuss.elastic.co/u/nexus1)\
**Post date:** [September 6, 2022, 7:20pm UTC](https://discuss.elastic.co/t/missing-field-from-my-sqs-queue/312746/4 "2022-09-06T19:20:05Z")

</div>

Thanks, Jaime.  
We are currently using the sqs metricset, actually.  
This is our metricbeat deployment:

```auto
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: metricbeat-deployment
  namespace: ***-sqs-metricbeat
  labels:
    app: metricbeat
spec:
  type: metricbeat
  version: 7.17.3
  elasticsearchRef:
    name: <name>
    namespace: <namespace_name>
  kibanaRef:
    name: <name>
    namespace: <namespace_name>
  config:
    setup.template.name: "metricbeat-sqs-metrics"
    setup.template.overwrite: "true"
    setup.dashboards.index: "metricbeat-sqs-metrics-*"
    setup.ilm.enabled: auto
    setup.ilm.rollover_alias: "metricbeat-sqs-metrics"
    setup.ilm.pattern: "{now/d}-000001"
    metricbeat:
      modules:
        - module: aws
          period: 60s
          regions: us-east-1
          metricsets:
          - sqs
          - kinesis
          role_arn: arn:aws:iam:: ***:role/*** -sqs-metricbeat
    logging:
      json: true
  deployment:
    podTemplate:
      spec:
        nodeSelector:
          k8s.clio.com/node-usage: default
        securityContext:
          runAsUser: 0

```

Any idea what we might be doing wrong?

---

<div class="post-metadata">

**Author:** ![nexus1](https://avatars.discourse-cdn.com/v4/letter/n/85e7bf/32.png) [@nexus1](https://discuss.elastic.co/u/nexus1)\
**Post date:** [September 6, 2022, 7:21pm UTC](https://discuss.elastic.co/t/missing-field-from-my-sqs-queue/312746/5 "2022-09-06T19:21:14Z")

</div>

We see "a few seconds" in the in the "ApproximateAgeOfOldestMessage" field on Kibana.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 4, 2022, 9:21pm UTC](https://discuss.elastic.co/t/missing-field-from-my-sqs-queue/312746/6 "2022-10-04T21:21:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.