Stackoverflow error on logstash when using es_bulk codec

[eeijlar]

Using the following pipeline with logstash:

- pipeline.id: export-process
      pipeline.workers: 4
      config.string: |
        input {
          elasticsearch {
            hosts => "http://elastic:80/elasticsearch/"
            user => "elastic"
            password => ""
            ssl => "false"
            index => "metricbeat-*"
            docinfo => true
            query => '{
                "query": {
                  "bool": {
                    "filter": {
                      "range": {
                          "@timestamp": {
                          "gte": "now-35m",
                          "lte": "now",
                          "format": "strict_date_optional_time||epoch_millis"
                          }
                      }
                    }
                  }
              }
            }'
          }
        }
        output {
          file {
            gzip => "true"
            codec => es_bulk
            path => "/usr/share/logstash/export/export_%{[@metadata][_index]}.json.gz"
          }
        }

I get the following error:

Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
warning: thread "[export-process]>worker1" terminated with exception (report_on_exception is true):
warning: thread "[export-process]>worker2" terminated with exception (report_on_exception is true):
java.lang.StackOverflowError
        at usr.share.logstash.logstash_minus_core.lib.logstash.codecs.base.encode(/usr/share/logstash/logstash-core/lib/logstash/codecs/base.rb:48)
        at usr.share.logstash.logstash_minus_core.lib.logstash.codecs.base.multi_encode(/usr/share/logstash/logstash-core/lib/logstash/codecs/base.rb:64)

I am using logstash 7.17.7. I have checked the codec is listed as plugin:

logstash@60961b0afe3b:~$ logstash-plugin list
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash-codec-avro
logstash-codec-cef
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines

I see this issue listed:

java.lang.StackOverflowError when codec does not have an encode method

but that is for a different codec.

[Badger]

@eeijlar

but that is for a different codec

Not exactly. The base codec class assumes that a codec class will override the encode or multi_encode method. If it does not then it will get a stack overflow. The issue you linked to used the example of the netflow codec, but it applies to any other codec as well.

The fix would be for the es_bulk codec to implement an encode method that raises a "not implemented" exception (similar to the decode in a dots codec)