Failed to upgrade on Ubuntu (displaying older version)

Elastic Stack Logstash
1

Problem

Logstash upgrade appears to fail with apt package installer. After updating apt sources to latest 2.4 for Logstash, and then performing update and upgrade, it does not appear in upgrade list and says on latest version even though a version check reveals 2.1.1

Question

Unsure how to verify upgrade to latest version on Ubuntu 14.04.x LTS using apt package installer. I am on ES 2.4.0 and want logstash and plugins to be on latest too.

Details

Check version

/opt/logstash/bin/logstash -V
logstash 2.1.1

Attempt to install again using package manager

sudo apt-get install logstash
Reading package lists... Done
Building dependency tree       
Reading state information... Done
logstash is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.

Verify using latest sources file for repo

sudo cat /etc/apt/sources.list.d/packages_elastic_co_logstash_2_4_debian.list 
deb http://packages.elastic.co/logstash/2.4/debian stable main
2

What version of the logstash package does dpkg think you have (dpkg -s logstash)?

3 
Package: logstash
Status: install ok installed
Priority: extra
Section: default
Installed-Size: 126342
Maintainer: <jenkins@slave12.build.ci.hetz.es.io>
Architecture: all
Version: 1:2.1.1-1
Depends: logrotate
Conffiles:
 /etc/default/logstash ac72257cd45e5de8f6799eaa5875bc04
 /etc/init.d/logstash 52f375dede199880173ed5c3413b995f
 /etc/logrotate.d/logstash 610253dfc0c70965bbd9d4698df7f244
Description: An extensible logging pipeline
License: Apache 2.0
Vendor: Elasticsearch
Homepage: http://www.elasticsearch.org/overview/logstash/
4

Should I purge on every box and re-run playbooks (Ansible) to install? I am just confused why apt-get update/upgrade after updating sources didn't work.

5

Okay, so you don't actually have 2.4 then. You have run apt-get update I hope? What's the output of apt-cache showpkg logstash?

6 
Package: logstash
Versions: 
1:2.4.0-1 (/var/lib/apt/lists/packages.elastic.co_logstash_2.4_debian_dists_stable_main_binary-amd64_Packages)
 Description Language: 
                 File: /var/lib/apt/lists/packages.elastic.co_logstash_2.4_debian_dists_stable_main_binary-amd64_Packages
                  MD5: 2cc8837023e818d510770f6e573a838a

1:2.1.1-1 (/var/lib/dpkg/status)
 Description Language: 
                 File: /var/lib/apt/lists/packages.elastic.co_logstash_2.4_debian_dists_stable_main_binary-amd64_Packages
                  MD5: 2cc8837023e818d510770f6e573a838a


Reverse Depends: 
Dependencies: 
1:2.4.0-1 - logrotate (0 (null)) 
1:2.1.1-1 - logrotate (0 (null)) 
Provides: 
1:2.4.0-1 - 
1:2.1.1-1 - 
Reverse Provides:
7

I perform a purge and have to manually delete 4 directories. Thereafter the older version disappears from the cache? Yes, I have done apt-get updates/upgrades and was expected this to update along with the ES/Beats updates as we update the pinned versions and sources. I only found out yesterday this was not updating like all the other packages.

sudo apt-get purge logstash
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  logstash*
0 upgraded, 0 newly installed, 1 to remove and 16 not upgraded.
After this operation, 129 MB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 118026 files and directories currently installed.)
Removing logstash (1:2.1.1-1) ...
Purging configuration files for logstash (1:2.1.1-1) ...
dpkg: warning: while removing logstash, directory '/etc/logstash/conf.d' not empty so not removed
dpkg: warning: while removing logstash, directory '/opt/logstash/bin' not empty so not removed
dpkg: warning: while removing logstash, directory '/var/lib/logstash' not empty so not removed
dpkg: warning: while removing logstash, directory '/var/log/logstash' not empty so not removed
ubuntu@app1:~$ sudo rm -rf /opt/logstash/
ubuntu@app1:~$ sudo rm -rf /var/lib/logstash/
ubuntu@app1:~$ sudo rm -rf /var/log/logstash/
ubuntu@app1:~$ sudo rm -rf /etc/logstash/conf.d/

After this I re-run the apt-cache and following:

sudo apt-cache showpkg logstash
Package: logstash
Versions: 
1:2.4.0-1 (/var/lib/apt/lists/packages.elastic.co_logstash_2.4_debian_dists_stable_main_binary-amd64_Packages)
 Description Language: 
                 File: /var/lib/apt/lists/packages.elastic.co_logstash_2.4_debian_dists_stable_main_binary-amd64_Packages
                  MD5: 2cc8837023e818d510770f6e573a838a


Reverse Depends: 
Dependencies: 
1:2.4.0-1 - logrotate (0 (null)) 
Provides: 
1:2.4.0-1 - 
Reverse Provides:
8

I purged every box, then deleted the mentioned directories and re-installed and now all are 2.4.0. There must be something odd where updates didn't occur or multiple versions (as noted in original cache output) so a heads up to others. I'm hoping not same issue on future updates but unsure cause.

Works now.

/opt/logstash/bin/logstash -V
logstash 2.4.0