After performing "Rolling Upgrade" from 6.0 to 6.3.1 Elasticsearch has stopped working

Rolling upgrade did not work as documented, I have 1 node, elastic log shows this warning below. Any other info needed pls let me know. We are sort of stuck right now and i'm so new to this that i'm not sure how to proceed.

Also, now old logs have been ingested and after 7 days are gzipped, if i need to re-parse old logs they have to be uncompressed.

[2018-08-07T13:50:12,153][WARN ][r.suppressed ] path: /.kibana/_search, params: {ignore_unavailable=true, index=.kibana, filter_path=aggregations.types.buckets}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

Here are all the curl commands and their responses:

  1. curl "localhost:9200/.kibana/_search?q=type:index-pattern&size=9999&_source=false&pretty"
    {
    "error" : {
    "root_cause" : [ ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [ ]
    },
    "status" : 503
    }

  2. curl elastic_ip:9200/_cluster/health?pretty
    {
    "cluster_name" : "elasticsearch",
    "status" : "red",
    "timed_out" : false,
    "number_of_nodes" : 1,
    "number_of_data_nodes" : 1,
    "active_primary_shards" : 1306,
    "active_shards" : 1306,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 1309,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "task_max_waiting_in_queue_millis" : 0,
    "active_shards_percent_as_number" : 49.942638623326964
    }

  3. curl localhost:9200/_cat/templates?pretty
    .monitoring-beats [.monitoring-beats-6-] 0 6020099
    .monitoring-alerts [.monitoring-alerts-6] 0 6020099
    .ml-state [.ml-state] 0 6030199
    security_audit_log [.security_audit_log
    ] 1000
    .watches [.watches*] 2147483647
    .ml-notifications [.ml-notifications] 0 6030199
    .triggered_watches [.triggered_watches*] 2147483647
    logstash-index-template [.logstash] 0
    .monitoring-es [.monitoring-es-6-] 0 6020099
    .ml-anomalies- [.ml-anomalies-
    ] 0 6030199
    security-index-template [.security-] 1000
    .monitoring-kibana [.monitoring-kibana-6-
    ] 0 6020099
    .monitoring-logstash [.monitoring-logstash-6-] 0 6020099
    logstash [logstash-
    ] 0 60001
    .watch-history-7 [.watcher-history-7*] 2147483647
    kibana_index_template:.kibana [.kibana] 0
    .ml-meta [.ml-meta] 0 6030199

  4. curl localhost:9200/_cat/indices?pretty (tons of these)
    yellow open logstash-2018.05.15 pfJad-p5SreuojqCD6--tA 5 1 4173661 0 1.3gb 1.3gb
    yellow open logstash-2018.01.31 WFx6NRT8TFqNhZSoMmQVyw 5 1 4067388 0 1.2gb 1.2gb
    yellow open logstash-2018.02.02 18cueP5aQ2y0Cyo_qJYP0A 5 1 4585440 0 1.4gb 1.4gb
    yellow open logstash-2017.12.01 -U4ribxcSUms08FrkOJlUQ 5 1 21758379 0 5.6gb 5.6gb

  5. curl localhost:9200/logstash-2018.08.06/_mapping
    {"logstash-2018.08.06":{"mappings":{"default":{"dynamic_templates":[{"message_field":{"path_match":"message","match_mapping_type":"string","mapping":{"norms":false,"type":"text"}}},{"string_fields":{"match":"","match_mapping_type":"string","mapping":{"fields":{"keyword":{"ignore_above":256,"type":"keyword"}},"norms":false,"type":"text"}}}],"properties":{"@timestamp":{"type":"date"},"@version":{"type":"keyword"},"geoip":{"dynamic":"true","properties":{"ip":{"type":"ip"},"latitude":{"type":"half_float"},"location":{"type":"geo_point"},"longitude":{"type":"half_float"}}}}},"doc":{"dynamic_templates":[{"message_field":{"path_match":"message","match_mapping_type":"string","mapping":{"norms":false,"type":"text"}}},{"string_fields":{"match":"","match_mapping_type":"string","mapping":{"fields":{"keyword":{"ignore_above":256,"type":"keyword"}},"norms":false,"type":"text"}}}],"properties":{"@timestamp":{"type":"date"},"@version":{"type":"keyword"},"appname":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword","ignore_above":256}}},"date2":{"type":"text","norms":false,"fields":{"keyword":{"type":"keyword","ignore_above":256}}},"geoip":{"dynamic":"true","properties":{"ip":

  6. curl -s 'localhost:9200/_cat/allocation?v'
    shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
    1306 655.7gb 3.3tb 4.1tb 7.4tb 45 172.20.0.19 172.20.0.19 X4gBF-q
    1309 UNASSIGNED
    8.curl -XGET 'localhost:9200/_cluster/health?pretty'
    {
    "cluster_name" : "elasticsearch",
    "status" : "red",
    "timed_out" : false,
    "number_of_nodes" : 1,
    "number_of_data_nodes" : 1,
    "active_primary_shards" : 1306,
    "active_shards" : 1306,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 1309,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "task_max_waiting_in_queue_millis" : 0,
    "active_shards_percent_as_number" : 49.942638623326964
    }

  7. curl -XGET 'http://localhost:9200/_settings?pretty' (one for every log)
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{
    "logstash-2018.05.15" : {
    "settings" : {
    "index" : {
    "refresh_interval" : "5s",
    "number_of_shards" : "5",
    "provided_name" : "logstash-2018.05.15",
    "creation_date" : "1526342400295",
    "number_of_replicas" : "1",
    "uuid" : "pfJad-p5SreuojqCD6--tA",
    "version" : {
    "created" : "6000099",
    "upgraded" : "6030199"
    }
    }
    }
    },
    "logstash-2018.02.02" : {
    "settings" : {
    "index" : {
    "refresh_interval" : "5s",
    "number_of_shards" : "5",
    "provided_name" : "logstash-2018.02.02",
    "creation_date" : "1517529600318",
    "number_of_replicas" : "1",
    "uuid" : "18cueP5aQ2y0Cyo_qJYP0A",
    "version" : {
    "created" : "6000099",
    "upgraded" : "6030199"
    }
    }
    }
    }

Input file to my issue:

input {

file {
path => ["/var/log/adc/2018///adc.log",
"/var/log/adc/2018///asdi.log",
"/var/log/adc/2018///edct_cdm_flight_data.log",
"/var/log/adc/2018///flightaware.log",
"/var/log/adc/2018///flight_manager.log",
"/var/log/adc/2018///fp.log",
"/var/log/adc/2018///invalid_outgoing.log",
"/var/log/adc/2018///iridium.log",
"/var/log/adc/2018///met_error.log",
"/var/log/adc/2018///microservice.log",
"/var/log/adc/2018///mq_output.log",
"/var/log/adc/2018///performance.log",
"/var/log/adc/2018///position_data.log",
"/var/log/adc/2018///rmqapps.log",
"/var/log/adc/2018///sbbtraffic.log",
"/var/log/adc/2018///schneider.log",
"/var/log/adc/2018///skyguide_notams.log",
"/var/log/adc/2018///sql.log",
"/var/log/adc/2018///unparsed.log",
"/var/log/adc/2018///wx.log"
]
tags => [ "standard_adc_format" ]

  # default discover_interval is 15 sec
  codec => plain {
          charset => "ISO-8859-1"
  }
  discover_interval => 60

  # file where indexes into the current log file positions are stored
  # sincedb_path => "/tmp/logstash-sincedb.db"
  sincedb_path => "/dev/null"
  ignore_older => 0

  # when a new log is first found, begin reading from the first line
  start_position => "beginning"
  #codec => "gzip_lines"

}

file {
path => ["/var/log/adc/2018///api.log",
"/var/log/adc/2018///dashboard.log"
]
tags => [ "alt_adc_format" ]

  # default discover_interval is 15 sec
   codec => plain {
           charset => "ISO-8859-1"
   }

  discover_interval => 60

  # file where indexes into the current log file positions are stored
  #sincedb_path => "/tmp/logstash-sincedb2.db"
  sincedb_path => "/dev/null"
  ignore_older => 0

  # when a new log is first found, begin reading from the first line
  start_position => "beginning"
  #codec => "gzip_lines"

}

file {
path => ["/var/log/sys/2018///maillog"
]
tags => [ "syslog_format" ]

  # default discover_interval is 15 sec
  codec => plain {
          charset => "ISO-8859-1"
  }
  discover_interval => 60

  # file where indexes into the current log file positions are stored
  #sincedb_path => "/tmp/logstash-sincedb3.db"
  sincedb_path => "/dev/null"
  ignore_older => 0

  # when a new log is first found, begin reading from the first line
  start_position => "beginning"
  #codec => "gzip_lines"

}
}

filter {

if "standard_adc_format" in [tags] {
    if ".py" in [message] {
        # it's a log line from a python app with extra info
        grok {
            match => [ "message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} %{NOTSPACE:appname}\[%{USERNAME:process_id}\]  %{NOTSPACE:serverdate} %{NOTSPACE:servertime} %{WORD:loglevel} %{NUMBER:thread_id} %{NOTSPACE:source_file} %{POSINT:source_line} %{GREEDYDATA:message}" ]

            overwrite => [ "message" ]
        }
    } else {
        # it's a standard syslog format not generated by our python logging libs
        grok {
            match => [ "message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} %{NOTSPACE:appname}\[%{USERNAME:process_id}\] %{GREEDYDATA:message}" ]
        }
    }
    mutate  {
        gsub => [ "message", "<nl>", "

" ]
}
}

if "alt_adc_format" in [tags] {
    grok {
        match => [ "message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} #\|%{NOTSPACE:date2}  %{NOTSPACE:time2} %{WORD:loglevel} %{NUMBER:thread_id} %{NOTSPACE:source_file} %{POSINT:source_line} %{GREEDYDATA:message}" ]

        overwrite => [ "message" ]
    }
    mutate  {
        gsub => [ "message", "<nl>", "

" ]
}
}

if "syslog_format" in [tags] {
    grok {
        match => [ "message", "^%{TIMESTAMP_ISO8601:logdate} <%{NOTSPACE:syslog}> %{NOTSPACE:hostname} %{NOTSPACE:appname} %{GREEDYDATA:message}" ]
        overwrite => [ "message" ]
    }
}

}

output {
if "_grokparsefailure" in [tags] {
# write events that didn't match to a file
file { "path" => "/tmp/grok_failures.txt" }
} else {
elasticsearch { hosts => ["localhost:9200"] }
}

for debugging:

stdout { codec => rubydebug }

}
ent preformatted text by 4 spaces

Disk usage on the node:

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/vg_log02sat-lv_root
7.5T 3.1T 4.2T 43% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 477M 68M 385M 15% /boot

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.