Elastic logging plugin for docker fails to start containers

Elastic Stack Beats
,
1

Hello,

After upgrading the Elasticsearch to 9.1.3, the docker containers cannot start due to issue with elastic logging plugin.

Background:
We have been using Elasticsearch 8.15.4 with docker plugin elastic/elastic-logging-plugin:8.17.3 to ship logs to our on prem elastic installation. This has been working very well. We have been using Debian 12 to host Elastic and for the application containers.

Issue:
We did a massive upgrade to the whole system now. We are now using Ubuntu 24.04 for both Elasticsearch and the application containers. None of the Docker containers can start now with an error from Docker plugin.

$ sudo docker version

Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:40:59 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:40:59 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ sudo docker plugin ls

ID             NAME                                    DESCRIPTION              ENABLED
3b39d4ab580b   elastic/elastic-logging-plugin:9.1.3    A beat for docker logs   true

$ sudo docker run --log-driver elastic/elastic-logging-plugin:9.1.3 --log-opt hosts="``http://REDACTED:9200``" --log-opt user="elastic" --log-opt password="REDACTED" alpine echo "test log"

docker: Error response from daemon: failed to create task for container: failed to initialize logging driver: error creating logger: LogDriver.StartLogging: error decoding json request: EOF.

$ sudo journalctl -eu docker

Sep 04 01:49:37 compute-test-1 dockerd[22110]: time="2025-09-04T01:49:37.620237513Z" level=warning msg="Unable to connect to plugin: plugin.moby.localhost/LogDriver.StartLogging: Post "http://plugin.moby.localhost/LogDriver.StartLogging\": EOF, retrying in 1s"
Sep 04 01:49:38 compute-test-1 dockerd[22110]: time="2025-09-04T01:49:38.644386356Z" level=error msg="Handler for POST /v1.47/containers/a66b6cf12b1462d0ff6e07ae86f1a54cb32ac27251eafdb13e54d61c4f5f08a1/start returned error: failed to create task for container: failed to initialize logging driver: error creating logger: LogDriver.StartLogging: error decoding json request: EOF\n" spanID=7ffbb880624c34ea traceID=ea474a7cf4ed3fe506c21757e51131ce

$ curl -u elastic:REDACTED http://REDACTED:9200

 {"name" : "elastic-test-1","cluster_name" : "elastic-test","cluster_uuid" : "REDACTED","version" : {"number" : "9.1.3","build_flavor" : "default","build_type" : "deb","build_hash" : "0c781091a2f57de895a73a1391ff8426c0153c8d","build_date" : "2025-08-24T22:05:04.526302670Z","build_snapshot" : false,"lucene_version" : "10.2.2","minimum_wire_compatibility_version" : "8.19.0","minimum_index_compatibility_version" : "8.0.0"},"tagline" : "You Know, for Search"}

I really appreciate any help you can provide. Thanks.