# After performing "Rolling Upgrade" from 6.0 to 6.3.1 Elasticsearch has stopped working **URL:** \ **Category:** Elasticsearch\ **Created:** [August 7, 2018, 2:08pm UTC](https://discuss.elastic.co/t/after-performing-rolling-upgrade-from-6-0-to-6-3-1-elasticsearch-has-stopped-working/143345 "2018-08-07T14:08:59Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Reena\_Deberry](https://avatars.discourse-cdn.com/v4/letter/r/f4b2a3/32.png) [@Reena\_Deberry](https://discuss.elastic.co/u/Reena_Deberry)\ **Post date:** [August 7, 2018, 2:08pm UTC](https://discuss.elastic.co/t/after-performing-rolling-upgrade-from-6-0-to-6-3-1-elasticsearch-has-stopped-working/143345/1 "2018-08-07T14:08:59Z")
Rolling upgrade did not work as documented, I have 1 node, elastic log shows this warning below. Any other info needed pls let me know. We are sort of stuck right now and i'm so new to this that i'm not sure how to proceed. Also, now old logs have been ingested and after 7 days are gzipped, if i need to re-parse old logs they have to be uncompressed. [2018-08-07T13:50:12,153][WARN][r.suppressed] path: /.kibana/\_search, params: {ignore\_unavailable=true, index=.kibana, filter\_path=aggregations.types.buckets} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed Here are all the curl commands and their responses: 1. curl "localhost:9200/.kibana/\_search?q=type:index-pattern&size=9999&\_source=false&pretty" { "error" : { "root\_cause" : [], "type" : "search\_phase\_execution\_exception", "reason" : "all shards failed", "phase" : "query", "grouped" : true, "failed\_shards" : [] }, "status" : 503 } 2. curl elastic\_ip:9200/\_cluster/health?pretty { "cluster\_name" : "elasticsearch", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 1, "number\_of\_data\_nodes" : 1, "active\_primary\_shards" : 1306, "active\_shards" : 1306, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 1309, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 49.942638623326964 } 3. curl localhost:9200/\_cat/templates?pretty .monitoring-beats [.monitoring-beats-6-_] 0 6020099 .monitoring-alerts [.monitoring-alerts-6] 0 6020099 .ml-state [.ml-state] 0 6030199 security\_audit\_log [.security\_audit\_log_] 1000 .watches [.watches\*] 2147483647 .ml-notifications [.ml-notifications] 0 6030199 .triggered\_watches [.triggered\_watches\*] 2147483647 logstash-index-template [.logstash] 0 .monitoring-es [.monitoring-es-6-_] 0 6020099 .ml-anomalies- [.ml-anomalies-_] 0 6030199 security-index-template [.security-_] 1000 .monitoring-kibana [.monitoring-kibana-6-_] 0 6020099 .monitoring-logstash [.monitoring-logstash-6-_] 0 6020099 logstash [logstash-_] 0 60001 .watch-history-7 [.watcher-history-7\*] 2147483647 kibana\_index\_template:.kibana [.kibana] 0 .ml-meta [.ml-meta] 0 6030199 4. curl localhost:9200/\_cat/indices?pretty (tons of these) yellow open logstash-2018.05.15 pfJad-p5SreuojqCD6--tA 5 1 4173661 0 1.3gb 1.3gb yellow open logstash-2018.01.31 WFx6NRT8TFqNhZSoMmQVyw 5 1 4067388 0 1.2gb 1.2gb yellow open logstash-2018.02.02 18cueP5aQ2y0Cyo\_qJYP0A 5 1 4585440 0 1.4gb 1.4gb yellow open logstash-2017.12.01 -U4ribxcSUms08FrkOJlUQ 5 1 21758379 0 5.6gb 5.6gb 5. curl localhost:9200/logstash-2018.08.06/\_mapping {"logstash-2018.08.06":{"mappings":{"_default_":{"dynamic\_templates":[{"message\_field":{"path\_match":"message","match\_mapping\_type":"string","mapping":{"norms":false,"type":"text"}}},{"string\_fields":{"match":"_","match\_mapping\_type":"string","mapping":{"fields":{"keyword":{"ignore\_above":256,"type":"keyword"}},"norms":false,"type":"text"}}}],"properties":{"@timestamp":{"type":"date"},"@version":{"type":"keyword"},"geoip":{"dynamic":"true","properties":{"ip":{"type":"ip"},"latitude":{"type":"half\_float"},"location":{"type":"geo\_point"},"longitude":{"type":"half\_float"}}}}},"doc":{"dynamic\_templates":[{"message\_field":{"path\_match":"message","match\_mapping\_type":"string","mapping":{"norms":false,"type":"text"}}},{"string\_fields":{"match":"_","match\_mapping\_type":"string","mapping":{"fields":{"keyword":{"ignore\_above":256,"type":"keyword"}},"norms":false,"type":"text"}}}],"properties":{"@timestamp":{"type":"date"},"@version":{"type":"keyword"},"appname":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"date2":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword","ignore\_above":256}}},"geoip":{"dynamic":"true","properties":{"ip": 6. curl -s 'localhost:9200/\_cat/allocation?v' shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 1306 655.7gb 3.3tb 4.1tb 7.4tb 45 172.20.0.19 172.20.0.19 X4gBF-q 1309 UNASSIGNED 8.curl -XGET 'localhost:9200/\_cluster/health?pretty' { "cluster\_name" : "elasticsearch", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 1, "number\_of\_data\_nodes" : 1, "active\_primary\_shards" : 1306, "active\_shards" : 1306, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 1309, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 49.942638623326964 } 7. curl -XGET '[http://localhost:9200/\_settings?pretty](http://localhost:9200/_settings?pretty)' (one for every log) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{ "logstash-2018.05.15" : { "settings" : { "index" : { "refresh\_interval" : "5s", "number\_of\_shards" : "5", "provided\_name" : "logstash-2018.05.15", "creation\_date" : "1526342400295", "number\_of\_replicas" : "1", "uuid" : "pfJad-p5SreuojqCD6--tA", "version" : { "created" : "6000099", "upgraded" : "6030199" } } } }, "logstash-2018.02.02" : { "settings" : { "index" : { "refresh\_interval" : "5s", "number\_of\_shards" : "5", "provided\_name" : "logstash-2018.02.02", "creation\_date" : "1517529600318", "number\_of\_replicas" : "1", "uuid" : "18cueP5aQ2y0Cyo\_qJYP0A", "version" : { "created" : "6000099", "upgraded" : "6030199" } } } } ---
**Author:** ![Reena\_Deberry](https://avatars.discourse-cdn.com/v4/letter/r/f4b2a3/32.png) [@Reena\_Deberry](https://discuss.elastic.co/u/Reena_Deberry)\ **Post date:** [August 7, 2018, 2:10pm UTC](https://discuss.elastic.co/t/after-performing-rolling-upgrade-from-6-0-to-6-3-1-elasticsearch-has-stopped-working/143345/2 "2018-08-07T14:10:38Z")
Input file to my issue: input { file { path =\> ["/var/log/adc/2018/_/_/adc.log", "/var/log/adc/2018/_/_/asdi.log", "/var/log/adc/2018/_/_/edct\_cdm\_flight\_data.log", "/var/log/adc/2018/_/_/flightaware.log", "/var/log/adc/2018/_/_/flight\_manager.log", "/var/log/adc/2018/_/_/fp.log", "/var/log/adc/2018/_/_/invalid\_outgoing.log", "/var/log/adc/2018/_/_/iridium.log", "/var/log/adc/2018/_/_/met\_error.log", "/var/log/adc/2018/_/_/microservice.log", "/var/log/adc/2018/_/_/mq\_output.log", "/var/log/adc/2018/_/_/performance.log", "/var/log/adc/2018/_/_/position\_data.log", "/var/log/adc/2018/_/_/rmqapps.log", "/var/log/adc/2018/_/_/sbbtraffic.log", "/var/log/adc/2018/_/_/schneider.log", "/var/log/adc/2018/_/_/skyguide\_notams.log", "/var/log/adc/2018/_/_/sql.log", "/var/log/adc/2018/_/_/unparsed.log", "/var/log/adc/2018/_/_/wx.log" ] tags =\> ["standard\_adc\_format"] ``` # default discover_interval is 15 sec codec => plain { charset => "ISO-8859-1" } discover_interval => 60 # file where indexes into the current log file positions are stored # sincedb_path => "/tmp/logstash-sincedb.db" sincedb_path => "/dev/null" ignore_older => 0 # when a new log is first found, begin reading from the first line start_position => "beginning" #codec => "gzip_lines" ``` } file { path =\> ["/var/log/adc/2018/_/_/api.log", "/var/log/adc/2018/_/_/dashboard.log" ] tags =\> ["alt\_adc\_format"] ``` # default discover_interval is 15 sec codec => plain { charset => "ISO-8859-1" } discover_interval => 60 # file where indexes into the current log file positions are stored #sincedb_path => "/tmp/logstash-sincedb2.db" sincedb_path => "/dev/null" ignore_older => 0 # when a new log is first found, begin reading from the first line start_position => "beginning" #codec => "gzip_lines" ``` } file { path =\> ["/var/log/sys/2018/_/_/maillog" ] tags =\> ["syslog\_format"] ``` # default discover_interval is 15 sec codec => plain { charset => "ISO-8859-1" } discover_interval => 60 # file where indexes into the current log file positions are stored #sincedb_path => "/tmp/logstash-sincedb3.db" sincedb_path => "/dev/null" ignore_older => 0 # when a new log is first found, begin reading from the first line start_position => "beginning" #codec => "gzip_lines" ``` } } filter { ``` if "standard_adc_format" in [tags] { if ".py" in [message] { # it's a log line from a python app with extra info grok { match => ["message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} %{NOTSPACE:appname}\[%{USERNAME:process_id}\] %{NOTSPACE:serverdate} %{NOTSPACE:servertime} %{WORD:loglevel} %{NUMBER:thread_id} %{NOTSPACE:source_file} %{POSINT:source_line} %{GREEDYDATA:message}" ] overwrite => ["message"] } } else { # it's a standard syslog format not generated by our python logging libs grok { match => ["message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} %{NOTSPACE:appname}\[%{USERNAME:process_id}\] %{GREEDYDATA:message}" ] } } mutate { gsub => [ "message", "", " ``` " ] } } ``` if "alt_adc_format" in [tags] { grok { match => ["message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} #\|%{NOTSPACE:date2} %{NOTSPACE:time2} %{WORD:loglevel} %{NUMBER:thread_id} %{NOTSPACE:source_file} %{POSINT:source_line} %{GREEDYDATA:message}"] overwrite => ["message"] } mutate { gsub => [ "message", "", " ``` " ] } } ``` if "syslog_format" in [tags] { grok { match => ["message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} %{NOTSPACE:appname} %{GREEDYDATA:message}"] overwrite => ["message"] } } ``` } output { if "\_grokparsefailure" in [tags] { # write events that didn't match to a file file { "path" =\> "/tmp/grok\_failures.txt" } } else { elasticsearch { hosts =\> ["localhost:9200"] } } # for debugging: # stdout { codec =\> rubydebug } } ent preformatted text by 4 spaces ---
**Author:** ![Reena\_Deberry](https://avatars.discourse-cdn.com/v4/letter/r/f4b2a3/32.png) [@Reena\_Deberry](https://discuss.elastic.co/u/Reena_Deberry)\ **Post date:** [August 7, 2018, 2:23pm UTC](https://discuss.elastic.co/t/after-performing-rolling-upgrade-from-6-0-to-6-3-1-elasticsearch-has-stopped-working/143345/3 "2018-08-07T14:23:52Z")
Disk usage on the node: ``` Filesystem Size Used Avail Use% Mounted on ``` /dev/mapper/vg\_log02sat-lv\_root 7.5T 3.1T 4.2T 43% / tmpfs 7.8G 0 7.8G 0% /dev/shm /dev/sda1 477M 68M 385M 15% /boot ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 4, 2018, 2:23pm UTC](https://discuss.elastic.co/t/after-performing-rolling-upgrade-from-6-0-to-6-3-1-elasticsearch-has-stopped-working/143345/4 "2018-09-04T14:23:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.