# SQS input plugin only pulls one JSON object from SQS Payload

**URL:** <https://discuss.elastic.co/t/sqs-input-plugin-only-pulls-one-json-object-from-sqs-payload/50352>\
**Category:** Logstash\
**Created:** [May 18, 2016, 3:04pm UTC](https://discuss.elastic.co/t/sqs-input-plugin-only-pulls-one-json-object-from-sqs-payload/50352 "2016-05-18T15:04:10Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![packplusplus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/packplusplus/32/29784_2.png) [@packplusplus](https://discuss.elastic.co/u/packplusplus)\
**Post date:** [May 18, 2016, 3:04pm UTC](https://discuss.elastic.co/t/sqs-input-plugin-only-pulls-one-json-object-from-sqs-payload/50352/1 "2016-05-18T15:04:10Z")

</div>

When using the SQS input plugin from the 2.0 branch, and (some of the) 1.5 branch. Payloads with multiple json objects only return a single json object. When using the json codec, it returns the last json object in the payload. When using the json\_lines codec, it returns the first object in the payload.

Here's a simplified config.

```
input {
  sqs {
    queue => "some_sqs_queue"
    #codec => "json_lines"
    codec => "json"
  }
}

output {
  stdout { 
    codec => rubydebug
  }
}

```

and a sample payload

```
{"obj1": 1}
{"obj2": 2}
{"obj3": 3}

```

Tested:

- 5.0.0-apha2 - behavior described above
- 2.3.2 - behavior described above
- 2.2.4 - behavior described above
- 2.1.1 - behavior described above
- 2.0.0 - behavior described above
- 1.5.5 - behavior described above (need to manually fix [https://github.com/logstash-plugins/logstash-input-sqs/pull/23](https://github.com/logstash-plugins/logstash-input-sqs/pull/23))
- 1.5.4 - json\_lines behaves as expected if {"obj3": 3} ends in a new line (no new line and the obj1 is dropped)
- 1.5.3 - json\_lines behaves as expected if {"obj3": 3} ends in a new line (no new line and the obj3 is dropped)

---

<div class="post-metadata">

**Author:** ![packplusplus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/packplusplus/32/29784_2.png) [@packplusplus](https://discuss.elastic.co/u/packplusplus)\
**Post date:** [May 18, 2016, 3:36pm UTC](https://discuss.elastic.co/t/sqs-input-plugin-only-pulls-one-json-object-from-sqs-payload/50352/2 "2016-05-18T15:36:27Z")

</div>

I don't know the logstash codebase that well, but looking at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-sqs-2.0.5/lib/logstash/inputs/sqs.rb I see this chunk of code

```
 def decode_event(message)
    @logger.info(message.body)
    @codec.decode(message.body) do |event|
      @logger.info(event.to_json)
      return event
    end
  end

```

I think that return should be an enqueue (or something like that), or wouldn't it always take one item, return and dump the rest on the ground?

---

<div class="post-metadata">

**Author:** ![packplusplus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/packplusplus/32/29784_2.png) [@packplusplus](https://discuss.elastic.co/u/packplusplus)\
**Post date:** [May 18, 2016, 4:15pm UTC](https://discuss.elastic.co/t/sqs-input-plugin-only-pulls-one-json-object-from-sqs-payload/50352/3 "2016-05-18T16:15:10Z")

</div>

I hacked on /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-sqs-2.0.5/lib/logstash/inputs/sqs.rb and collapsed some of the nice small functions into something that copes with the json\_lines codec. Off to write an issue.

```
def run(output_queue)
  @logger.debug("Polling SQS queue", :polling_options => polling_options)

  run_with_backoff do
    poller.poll(polling_options) do |messages, stats|
      break if stop?

      messages.each do |message|
        @codec.decode(message.body) do |event|
          @logger.info(event.to_json)
          add_sqs_data(event, message)
          decorate(event)
          output_queue << event
        end
      end

      @logger.debug("SQS Stats:", :request_count => stats.request_count,
                     :received_message_count => stats.received_message_count,
                    :last_message_received_at => stats.last_message_received_at) if @logger.debug?
    end
  end
end
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:57am UTC](https://discuss.elastic.co/t/sqs-input-plugin-only-pulls-one-json-object-from-sqs-payload/50352/4 "2017-07-06T04:57:07Z")

</div>