After upgrade, marvel reports two different versions of nodes

Elastic Stack Elasticsearch
1

I just upgraded my cluster from 1.1.1 to 1.5.2. 99.9 % of the upgrade went well and the system is stable. However, there is a small splinter that is giving me grief. logstash and elasticsearch are running on the same server with a secondary server as a data node.

When I look at the health on either of the two servers, it is reporting the expected version of the updated elasticsearch:

 root@logstash:~# curl -XGET 'http://localhost:9200'
 {
   "status" : 200,
   "name" : "es-logstash-n1",
   "cluster_name" : "es-logstash",
   "version" : {
     "number" : "1.5.2",
     "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
     "build_timestamp" : "2015-04-27T09:21:06Z",
     "build_snapshot" : false,
     "lucene_version" : "4.10.4"
   },
   "tagline" : "You Know, for Search"
 }
 root@logstash:~#

When I check on the versions of the nodes, it reports back this:

 root@logstash:~# curl -XGET 'http://localhost:9200/_cat/nodes?v&h=ip,port,v'
 ip           port v
 192.168.1.72 9302 1.1.1
 192.168.1.72 9307 1.1.1
 192.168.1.72 9300 1.5.2
 192.168.1.72 9305 1.1.1
 192.168.1.72 9309 1.1.1
 192.168.1.72 9311 1.1.1
 192.168.1.72 9304 1.1.1
 192.168.1.72 9310 1.1.1
 192.168.1.72 9301 1.5.2
 192.168.1.72 9303 1.1.1
 192.168.1.72 9306 1.1.1
 192.168.1.72 9301 1.1.1
 192.168.1.72 9308 1.1.1
 root@logstash:~#

Do I need to upgrade the nodes in addition to this upgrade, Am I missing the point on something? I do not see any docs on if this is needed or not.

2

Hi there,

How long has it been since the upgrade? We've seen Marvel take several (~10) minutes to completely update with the new meta data from the nodes.

Marty

3

It has been 4+ hours as of this post.

4

Are you using the node protocol in the output? If so it's probably Logstash that is reporting that version.

5

I think you are correct. When I look at the ports that are listening and track the actual PID, it is indeed logstash.

root@logstash:~# netstat -plunt | grep 930[1-9]
tcp6       0      0 :::9301                 :::*                    LISTEN      24821/java
tcp6       0      0 :::9302                 :::*                    LISTEN      24821/java
tcp6       0      0 :::9303                 :::*                    LISTEN      24821/java
tcp6       0      0 :::9304                 :::*                    LISTEN      24821/java
tcp6       0      0 :::9305                 :::*                    LISTEN      24821/java
tcp6       0      0 :::9306                 :::*                    LISTEN      24821/java
tcp6       0      0 :::9307                 :::*                    LISTEN      24821/java
tcp6       0      0 :::9308                 :::*                    LISTEN      24821/java
tcp6       0      0 :::9309                 :::*                    LISTEN      24821/java
root@logstash:~# ps faux | grep 24821
root     32284  0.0  0.0   7832   872 pts/0    S+   15:50   0:00          \_ grep 24821
logstash 24821 10.9 12.6 12127852 2041788 ?    SNl  May11 318:31 /usr/bin/java -Djava.io.tmpdir=/var/lib/logstash -Xmx10g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash-1.4.2/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash-1.4.2/lib /opt/logstash-1.4.2/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
root@logstash:~#

My logstash is at the latest and greatest. I am simply pushing logstash information into the elasticsearch cluster. I am not certain how they are reporting back that they are 1.1.1.

6

LS uses an older version of some libraries that report this, if you don't want to show this then just use the HTTP protocol.