After performing "Rolling Upgrade" from 6.0 to 6.3.1 Elasticsearch has stopped working

Reena_Deberry · August 7, 2018, 2:08pm

Rolling upgrade did not work as documented, I have 1 node, elastic log shows this warning below. Any other info needed pls let me know. We are sort of stuck right now and i'm so new to this that i'm not sure how to proceed.

Also, now old logs have been ingested and after 7 days are gzipped, if i need to re-parse old logs they have to be uncompressed.

[2018-08-07T13:50:12,153][WARN ][r.suppressed ] path: /.kibana/_search, params: {ignore_unavailable=true, index=.kibana, filter_path=aggregations.types.buckets}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

Here are all the curl commands and their responses:

curl "localhost:9200/.kibana/_search?q=type:index-pattern&size=9999&_source=false&pretty"
{
"error" : {
"root_cause" : [ ],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [ ]
},
"status" : 503
}
curl elastic_ip:9200/_cluster/health?pretty
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1306,
"active_shards" : 1306,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1309,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 49.942638623326964
}
curl localhost:9200/_cat/templates?pretty
.monitoring-beats [.monitoring-beats-6-] 0 6020099
.monitoring-alerts [.monitoring-alerts-6] 0 6020099
.ml-state [.ml-state] 0 6030199
security_audit_log [.security_audit_log] 1000
.watches [.watches*] 2147483647
.ml-notifications [.ml-notifications] 0 6030199
.triggered_watches [.triggered_watches*] 2147483647
logstash-index-template [.logstash] 0
.monitoring-es [.monitoring-es-6-] 0 6020099
.ml-anomalies- [.ml-anomalies-] 0 6030199
security-index-template [.security-] 1000
.monitoring-kibana [.monitoring-kibana-6-] 0 6020099
.monitoring-logstash [.monitoring-logstash-6-] 0 6020099
logstash [logstash-] 0 60001
.watch-history-7 [.watcher-history-7*] 2147483647
kibana_index_template:.kibana [.kibana] 0
.ml-meta [.ml-meta] 0 6030199
curl localhost:9200/_cat/indices?pretty (tons of these)
yellow open logstash-2018.05.15 pfJad-p5SreuojqCD6--tA 5 1 4173661 0 1.3gb 1.3gb
yellow open logstash-2018.01.31 WFx6NRT8TFqNhZSoMmQVyw 5 1 4067388 0 1.2gb 1.2gb
yellow open logstash-2018.02.02 18cueP5aQ2y0Cyo_qJYP0A 5 1 4585440 0 1.4gb 1.4gb
yellow open logstash-2017.12.01 -U4ribxcSUms08FrkOJlUQ 5 1 21758379 0 5.6gb 5.6gb
curl localhost:9200/logstash-2018.08.06/_mapping
{"logstash-2018.08.06":{"mappings":{"default":{"dynamic_templates":[{"message_field":{"path_match":"message","match_mapping_type":"string","mapping":{"norms":false,"type":"text"}}},{"string_fields":{"match":"","match_mapping_type":"string","mapping":{"fields":{"keyword":{"ignore_above":256,"type":"keyword"}},"norms":false,"type":"text"}}}],"properties":{"@timestamp":{"type":"date"},"@version":{"type":"keyword"},"geoip":{"dynamic":"true","properties":{"ip":{"type":"ip"},"latitude":{"type":"half_float"},"location":{"type":"geo_point"},"longitude":{"type":"half_float"}}}}},"doc":{"dynamic_templates":[{"message_field":{"path_match":"message","match_mapping_type":"string","mapping":{"norms":false,"type":"text"}}},{"string_fields":{"match":"","match_mapping_type":"string","mapping":{"fields":{"keyword":{"ignore_above":256,"type":"keyword"}},"norms":false,"type":"text"}}}],"properties":{"@timestamp":{"type":"date"},"@version":{"type":"keyword"},"appname":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword","ignore_above":256}}},"date2":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword","ignore_above":256}}},"geoip":{"dynamic":"true","properties":{"ip":
curl -s 'localhost:9200/_cat/allocation?v'
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
1306 655.7gb 3.3tb 4.1tb 7.4tb 45 172.20.0.19 172.20.0.19 X4gBF-q
1309 UNASSIGNED
8.curl -XGET 'localhost:9200/_cluster/health?pretty'
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1306,
"active_shards" : 1306,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1309,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 49.942638623326964
}
curl -XGET 'http://localhost:9200/_settings?pretty' (one for every log)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{
"logstash-2018.05.15" : {
"settings" : {
"index" : {
"refresh_interval" : "5s",
"number_of_shards" : "5",
"provided_name" : "logstash-2018.05.15",
"creation_date" : "1526342400295",
"number_of_replicas" : "1",
"uuid" : "pfJad-p5SreuojqCD6--tA",
"version" : {
"created" : "6000099",
"upgraded" : "6030199"
}
}
}
},
"logstash-2018.02.02" : {
"settings" : {
"index" : {
"refresh_interval" : "5s",
"number_of_shards" : "5",
"provided_name" : "logstash-2018.02.02",
"creation_date" : "1517529600318",
"number_of_replicas" : "1",
"uuid" : "18cueP5aQ2y0Cyo_qJYP0A",
"version" : {
"created" : "6000099",
"upgraded" : "6030199"
}
}
}
}

Reena_Deberry · August 7, 2018, 2:10pm

Input file to my issue:

input {

file {
path => ["/var/log/adc/2018///adc.log",
"/var/log/adc/2018///asdi.log",
"/var/log/adc/2018///edct_cdm_flight_data.log",
"/var/log/adc/2018///flightaware.log",
"/var/log/adc/2018///flight_manager.log",
"/var/log/adc/2018///fp.log",
"/var/log/adc/2018///invalid_outgoing.log",
"/var/log/adc/2018///iridium.log",
"/var/log/adc/2018///met_error.log",
"/var/log/adc/2018///microservice.log",
"/var/log/adc/2018///mq_output.log",
"/var/log/adc/2018///performance.log",
"/var/log/adc/2018///position_data.log",
"/var/log/adc/2018///rmqapps.log",
"/var/log/adc/2018///sbbtraffic.log",
"/var/log/adc/2018///schneider.log",
"/var/log/adc/2018///skyguide_notams.log",
"/var/log/adc/2018///sql.log",
"/var/log/adc/2018///unparsed.log",
"/var/log/adc/2018///wx.log"
]
tags => [ "standard_adc_format" ]

  # default discover_interval is 15 sec
  codec => plain {
          charset => "ISO-8859-1"
  }
  discover_interval => 60

  # file where indexes into the current log file positions are stored
  # sincedb_path => "/tmp/logstash-sincedb.db"
  sincedb_path => "/dev/null"
  ignore_older => 0

  # when a new log is first found, begin reading from the first line
  start_position => "beginning"
  #codec => "gzip_lines"

}

file {
path => ["/var/log/adc/2018///api.log",
"/var/log/adc/2018///dashboard.log"
]
tags => [ "alt_adc_format" ]

  # default discover_interval is 15 sec
   codec => plain {
           charset => "ISO-8859-1"
   }

  discover_interval => 60

  # file where indexes into the current log file positions are stored
  #sincedb_path => "/tmp/logstash-sincedb2.db"
  sincedb_path => "/dev/null"
  ignore_older => 0

  # when a new log is first found, begin reading from the first line
  start_position => "beginning"
  #codec => "gzip_lines"

}

file {
path => ["/var/log/sys/2018///maillog"
]
tags => [ "syslog_format" ]

  # default discover_interval is 15 sec
  codec => plain {
          charset => "ISO-8859-1"
  }
  discover_interval => 60

  # file where indexes into the current log file positions are stored
  #sincedb_path => "/tmp/logstash-sincedb3.db"
  sincedb_path => "/dev/null"
  ignore_older => 0

  # when a new log is first found, begin reading from the first line
  start_position => "beginning"
  #codec => "gzip_lines"

}
}

filter {

if "standard_adc_format" in [tags] {
    if ".py" in [message] {
        # it's a log line from a python app with extra info
        grok {
            match => [ "message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} %{NOTSPACE:appname}\[%{USERNAME:process_id}\]  %{NOTSPACE:serverdate} %{NOTSPACE:servertime} %{WORD:loglevel} %{NUMBER:thread_id} %{NOTSPACE:source_file} %{POSINT:source_line} %{GREEDYDATA:message}" ]

            overwrite => [ "message" ]
        }
    } else {
        # it's a standard syslog format not generated by our python logging libs
        grok {
            match => [ "message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} %{NOTSPACE:appname}\[%{USERNAME:process_id}\] %{GREEDYDATA:message}" ]
        }
    }
    mutate  {
        gsub => [ "message", "<nl>", "

" ]
}
}

if "alt_adc_format" in [tags] {
    grok {
        match => [ "message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} #\|%{NOTSPACE:date2}  %{NOTSPACE:time2} %{WORD:loglevel} %{NUMBER:thread_id} %{NOTSPACE:source_file} %{POSINT:source_line} %{GREEDYDATA:message}" ]

        overwrite => [ "message" ]
    }
    mutate  {
        gsub => [ "message", "<nl>", "

" ]
}
}

if "syslog_format" in [tags] {
    grok {
        match => [ "message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} %{NOTSPACE:appname} %{GREEDYDATA:message}" ]
        overwrite => [ "message" ]
    }
}

}

output {
if "_grokparsefailure" in [tags] {
# write events that didn't match to a file
file { "path" => "/tmp/grok_failures.txt" }
} else {
elasticsearch { hosts => ["localhost:9200"] }
}

for debugging:

stdout { codec => rubydebug }

}
ent preformatted text by 4 spaces

Reena_Deberry · August 7, 2018, 2:23pm

Disk usage on the node:

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/vg_log02sat-lv_root
7.5T 3.1T 4.2T 43% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 477M 68M 385M 15% /boot

system · September 4, 2018, 2:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch continually fails after upgrade Elasticsearch	8	1077	July 11, 2019
All shards failed-yellow status Elasticsearch	5	1904	February 11, 2020
[error] All shards failed after migration to 6.7.1 Elasticsearch	5	5154	May 13, 2019
Upgraded from 7.1.0 to 7.3.1 and stoped working Elasticsearch	2	552	October 9, 2019
Elasticserach6.1.1 restart and i got "all shards failed" Elasticsearch	4	51570	February 23, 2018

After performing "Rolling Upgrade" from 6.0 to 6.3.1 Elasticsearch has stopped working

for debugging:

stdout { codec => rubydebug }

Related topics