After performing "Rolling Upgrade" from 6.0 to 6.3.1 Elasticsearch has stopped working

Reena_Deberry · August 7, 2018, 2:08pm

Rolling upgrade did not work as documented, I have 1 node, elastic log shows this warning below. Any other info needed pls let me know. We are sort of stuck right now and i'm so new to this that i'm not sure how to proceed.

Also, now old logs have been ingested and after 7 days are gzipped, if i need to re-parse old logs they have to be uncompressed.

[2018-08-07T13:50:12,153][WARN ][r.suppressed ] path: /.kibana/_search, params: {ignore_unavailable=true, index=.kibana, filter_path=aggregations.types.buckets}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

Here are all the curl commands and their responses:

curl "localhost:9200/.kibana/_search?q=type:index-pattern&size=9999&_source=false&pretty"
{
"error" : {
"root_cause" : [ ],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [ ]
},
"status" : 503
}
curl elastic_ip:9200/_cluster/health?pretty
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1306,
"active_shards" : 1306,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1309,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 49.942638623326964
}
curl localhost:9200/_cat/templates?pretty
.monitoring-beats [.monitoring-beats-6-] 0 6020099
.monitoring-alerts [.monitoring-alerts-6] 0 6020099
.ml-state [.ml-state] 0 6030199
security_audit_log [.security_audit_log] 1000
.watches [.watches*] 2147483647
.ml-notifications [.ml-notifications] 0 6030199
.triggered_watches [.triggered_watches*] 2147483647
logstash-index-template [.logstash] 0
.monitoring-es [.monitoring-es-6-] 0 6020099
.ml-anomalies- [.ml-anomalies-] 0 6030199
security-index-template [.security-] 1000
.monitoring-kibana [.monitoring-kibana-6-] 0 6020099
.monitoring-logstash [.monitoring-logstash-6-] 0 6020099
logstash [logstash-] 0 60001
.watch-history-7 [.watcher-history-7*] 2147483647
kibana_index_template:.kibana [.kibana] 0
.ml-meta [.ml-meta] 0 6030199
curl localhost:9200/_cat/indices?pretty (tons of these)
yellow open logstash-2018.05.15 pfJad-p5SreuojqCD6--tA 5 1 4173661 0 1.3gb 1.3gb
yellow open logstash-2018.01.31 WFx6NRT8TFqNhZSoMmQVyw 5 1 4067388 0 1.2gb 1.2gb
yellow open logstash-2018.02.02 18cueP5aQ2y0Cyo_qJYP0A 5 1 4585440 0 1.4gb 1.4gb
yellow open logstash-2017.12.01 -U4ribxcSUms08FrkOJlUQ 5 1 21758379 0 5.6gb 5.6gb
curl localhost:9200/logstash-2018.08.06/_mapping
{"logstash-2018.08.06":{"mappings":{"default":{"dynamic_templates":[{"message_field":{"path_match":"message","match_mapping_type":"string","mapping":{"norms":false,"type":"text"}}},{"string_fields":{"match":"","match_mapping_type":"string","mapping":{"fields":{"keyword":{"ignore_above":256,"type":"keyword"}},"norms":false,"type":"text"}}}],"properties":{"@timestamp":{"type":"date"},"@version":{"type":"keyword"},"geoip":{"dynamic":"true","properties":{"ip":{"type":"ip"},"latitude":{"type":"half_float"},"location":{"type":"geo_point"},"longitude":{"type":"half_float"}}}}},"doc":{"dynamic_templates":[{"message_field":{"path_match":"message","match_mapping_type":"string","mapping":{"norms":false,"type":"text"}}},{"string_fields":{"match":"","match_mapping_type":"string","mapping":{"fields":{"keyword":{"ignore_above":256,"type":"keyword"}},"norms":false,"type":"text"}}}],"properties":{"@timestamp":{"type":"date"},"@version":{"type":"keyword"},"appname":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword","ignore_above":256}}},"date2":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword","ignore_above":256}}},"geoip":{"dynamic":"true","properties":{"ip":
curl -s 'localhost:9200/_cat/allocation?v'
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
1306 655.7gb 3.3tb 4.1tb 7.4tb 45 172.20.0.19 172.20.0.19 X4gBF-q
1309 UNASSIGNED
8.curl -XGET 'localhost:9200/_cluster/health?pretty'
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1306,
"active_shards" : 1306,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1309,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 49.942638623326964
}
curl -XGET 'http://localhost:9200/_settings?pretty' (one for every log)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{
"logstash-2018.05.15" : {
"settings" : {
"index" : {
"refresh_interval" : "5s",
"number_of_shards" : "5",
"provided_name" : "logstash-2018.05.15",
"creation_date" : "1526342400295",
"number_of_replicas" : "1",
"uuid" : "pfJad-p5SreuojqCD6--tA",
"version" : {
"created" : "6000099",
"upgraded" : "6030199"
}
}
}
},
"logstash-2018.02.02" : {
"settings" : {
"index" : {
"refresh_interval" : "5s",
"number_of_shards" : "5",
"provided_name" : "logstash-2018.02.02",
"creation_date" : "1517529600318",
"number_of_replicas" : "1",
"uuid" : "18cueP5aQ2y0Cyo_qJYP0A",
"version" : {
"created" : "6000099",
"upgraded" : "6030199"
}
}
}
}

Reena_Deberry · August 7, 2018, 2:10pm

Input file to my issue:

input {

file {
path => ["/var/log/adc/2018///adc.log",
"/var/log/adc/2018///asdi.log",
"/var/log/adc/2018///edct_cdm_flight_data.log",
"/var/log/adc/2018///flightaware.log",
"/var/log/adc/2018///flight_manager.log",
"/var/log/adc/2018///fp.log",
"/var/log/adc/2018///invalid_outgoing.log",
"/var/log/adc/2018///iridium.log",
"/var/log/adc/2018///met_error.log",
"/var/log/adc/2018///microservice.log",
"/var/log/adc/2018///mq_output.log",
"/var/log/adc/2018///performance.log",
"/var/log/adc/2018///position_data.log",
"/var/log/adc/2018///rmqapps.log",
"/var/log/adc/2018///sbbtraffic.log",
"/var/log/adc/2018///schneider.log",
"/var/log/adc/2018///skyguide_notams.log",
"/var/log/adc/2018///sql.log",
"/var/log/adc/2018///unparsed.log",
"/var/log/adc/2018///wx.log"
]
tags => [ "standard_adc_format" ]

  # default discover_interval is 15 sec
  codec => plain {
          charset => "ISO-8859-1"
  }
  discover_interval => 60

  # file where indexes into the current log file positions are stored
  # sincedb_path => "/tmp/logstash-sincedb.db"
  sincedb_path => "/dev/null"
  ignore_older => 0

  # when a new log is first found, begin reading from the first line
  start_position => "beginning"
  #codec => "gzip_lines"

}

file {
path => ["/var/log/adc/2018///api.log",
"/var/log/adc/2018///dashboard.log"
]
tags => [ "alt_adc_format" ]

  # default discover_interval is 15 sec
   codec => plain {
           charset => "ISO-8859-1"
   }

  discover_interval => 60

  # file where indexes into the current log file positions are stored
  #sincedb_path => "/tmp/logstash-sincedb2.db"
  sincedb_path => "/dev/null"
  ignore_older => 0

  # when a new log is first found, begin reading from the first line
  start_position => "beginning"
  #codec => "gzip_lines"

}

file {
path => ["/var/log/sys/2018///maillog"
]
tags => [ "syslog_format" ]

  # default discover_interval is 15 sec
  codec => plain {
          charset => "ISO-8859-1"
  }
  discover_interval => 60

  # file where indexes into the current log file positions are stored
  #sincedb_path => "/tmp/logstash-sincedb3.db"
  sincedb_path => "/dev/null"
  ignore_older => 0

  # when a new log is first found, begin reading from the first line
  start_position => "beginning"
  #codec => "gzip_lines"

}
}

filter {

if "standard_adc_format" in [tags] {
    if ".py" in [message] {
        # it's a log line from a python app with extra info
        grok {
            match => [ "message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} %{NOTSPACE:appname}\[%{USERNAME:process_id}\]  %{NOTSPACE:serverdate} %{NOTSPACE:servertime} %{WORD:loglevel} %{NUMBER:thread_id} %{NOTSPACE:source_file} %{POSINT:source_line} %{GREEDYDATA:message}" ]

            overwrite => [ "message" ]
        }
    } else {
        # it's a standard syslog format not generated by our python logging libs
        grok {
            match => [ "message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} %{NOTSPACE:appname}\[%{USERNAME:process_id}\] %{GREEDYDATA:message}" ]
        }
    }
    mutate  {
        gsub => [ "message", "<nl>", "

" ]
}
}

if "alt_adc_format" in [tags] {
    grok {
        match => [ "message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} #\|%{NOTSPACE:date2}  %{NOTSPACE:time2} %{WORD:loglevel} %{NUMBER:thread_id} %{NOTSPACE:source_file} %{POSINT:source_line} %{GREEDYDATA:message}" ]

        overwrite => [ "message" ]
    }
    mutate  {
        gsub => [ "message", "<nl>", "

" ]
}
}

if "syslog_format" in [tags] {
    grok {
        match => [ "message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} %{NOTSPACE:appname} %{GREEDYDATA:message}" ]
        overwrite => [ "message" ]
    }
}

}

output {
if "_grokparsefailure" in [tags] {
# write events that didn't match to a file
file { "path" => "/tmp/grok_failures.txt" }
} else {
elasticsearch { hosts => ["localhost:9200"] }
}

for debugging:

stdout { codec => rubydebug }

}
ent preformatted text by 4 spaces

Reena_Deberry · August 7, 2018, 2:23pm

Disk usage on the node:

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/vg_log02sat-lv_root
7.5T 3.1T 4.2T 43% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 477M 68M 385M 15% /boot

system · September 4, 2018, 2:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.