Logstash Docker Image locations and size differences

[kevingimbel]

Hello there,

@pmoust pointed out in a recent post of me (https://discuss.elastic.co/t/cant-connect-to-elastic-cloud-from-logstash-docker-image-logstash/75517/4) that the official Logstash Docker Images from Elastic are not hosted on hub.docker.com but on docker.elastic.co instead.

Can anyone clarify why this is the case? Why does elastic host its own Docker Registry and why are there differences in image sizes?

Logstash from hub.docker.com is ~493MB 
Logstash from docker.elastic.co is ~719MB

It appears to me that docker.elastic.co has no Front-End so I need to know what tags I can fetch or where the software I want is located. With Docker Hub's Front-End it is a lot easier to see images and image tags / versions. So, my main questions are:

1.) Why do you use a private Registry? Are there any benefits?
2.) Why is there a difference in Image size and why is the official Elastic Image bigger?

Thanks in advance,

• Kevin

[jarpy (Nina Dowland)]

Hi Kevin,

I'm the lead maintainer on logstash-docker. Let me see if I can help.

The two main reasons for docker.elastic.co are:

1. We needed to build our own registry to host images for the Elastic Cloud Enterprise product (self-hosted Elastic Cloud). Essentially, we need to own the uptime of the registry becuase we own the SLA to customers of that product. It just wouldn't be cool for us to depend on Docker Hub, because we wouldn't be able to do anything if there was a problem. Since we already have a really resilient registry, we figured we'd use that for our public images too.

2. Running our own registry gives us really cool metrics on when and where our images are being downloaded. We love metrics. (No prizes for guessing how we analyse logs from the registry).

Now your other question...

Question: Why is there a difference in Image size and why is the official Elastic Image bigger?

Answer: OMG thank you!

This pesky line is largely why. A terrible rookie mistake on my part which is now fixed.

The next image release will be about 170Mb smaller.
Nice. Thanks again .

[kevingimbel]

Hey @jarpy,

Thank you for the explanation! Makes perfect sense to run your own registry for SLA compliance.

If I am building my own docker images for logstash should I base them on the docker.elastic.co registry version of Logstash? Or is there - beside this little bug - no difference? I currently created one base Logstash Image for use in our company, switching from hub.docker.com to docker.elastic.co shouldn't be too much of a hassle.

Glad I could help find a bug! And thank you for mentioning me in your commit message - highly appreciated!

[jarpy (Nina Dowland)]

No worries, and thank you again.

There are some differences between the two images. Naturally, I like to think that we ship the best image. Actually, a recent development is that the Docker team are taking steps to deprecate their images in favour of ours, now that we are shipping images directly as the "upstream vendor". The Docker engineers have been extremely gracious and cool about this.

Some differences that I can think of off the top of my head are:

1. We ship the X-Pack extension pre-installed, which allows for use of the new Logstash Monitoring capabilities with a no cost, basic subscription.

2. We have a (small, but growing) acceptance test suite that helps us guarantee the quality of the image.

3. Our image never has a latest tag. We think this is really dangerous. Elastic products release often and we don't want our users to get a nasty surprise when the image changes without them asking for it.

4. Our image doesn't have an ENTRYPOINT defined. This is something that the image from Docker does better, in my opinion. I'd like to fix this soon. Having a good ENTRYPOINT can make docker run commands more compact, which would be good because having our own registry already makes those commands longer than I'd like them to be.

Anyway, soon, our image should be the "one true image", so I recommend extending that one. Also, please keep the awesome bug reports coming in . Feel free to raise issues on the GitHub project for any bugs you see or features that you would like to see.

Oh, I never answered your question about versions. No, sorry, we don't have a GUI/browser for our registry. That would be really cool; I'd like to have one. In terms of what is available, we release new, version-tagged images for Elasticsearch, Kibana and Logstash simultaneously with the traditional packages. You will always be able to pull docker.elastic.co/logstash/logstash:$VERSION, where $VERSION is a released Logstash version number.

You may have noticed that I didn't mention Beats there. Images for version 5.2.1 of the (non-Windows) Beats are actually on the registry now, but they are still "alpha", and are thus subject to change and not ready for production. Bug reports and suggestions are welcome here too, naturally!

Cheers,
Toby/Jarpy.

[kevingimbel]

@jarpy thanks for this in-depth answer! If Docker is going to deprecate their images I'll switch ours over to the official images immediately - better now than later!

Since I'll work quite a lot with the Docker images in the next weeks/months I'll make sure to report every issue or "weird behaviour" I encounter.

One last question: If X-Pack is pre-installed with your images and we have a license for X-Pack from our Elastic Cloud subscription (I think it's included with the Gold plan and up) can we use it in multiple containers with Logstash? Or would we need to get a X-Pack License for each Logstash Instance separately?

I still don't entirely grasp the pricing of Elastic Products with their different plans/subscription models.

Anyway, thank you a lot for all your answers! They've been really helpful.

[jarpy (Nina Dowland)]

Great! You've been an excellent source of real-world feedback already. 'Really appreciate it.

Feel free to run as many X-Pack enabled Logstash instances as you and your fleet would like.