Syslog are not indexing to logstash-2019-x.x

[wcpoon (Woody)]

Hi guys,

after upgrade from 6.7 to 7.0, i can received syslog from my remote devices syslog to /var/log/messages, however, i cannot display all the latest logs in kibana discover tab.
all latest data is not indexing to logstash-2019.x.x

anyone can help?

[wcpoon (Woody)]

Why no one want to help on this?

[staodd (Staale)]

Maybe because you don't give any information that can be used to solve your problem. You just state its not working without including any logs or config.

[wcpoon (Woody)]

Sorry,

this is the error message,

[2019-05-06T10:10:35,215][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2019.05.06", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x18120ee4>], :response=>{"index"=>{"_index"=>"logstash-2019.05.06", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"The [default] mapping cannot be updated on index [logstash-2019.05.06]: defaults mappings are not useful anymore now that indices can have at most one type."}}}}
[2019-05-06T10:11:07,506][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2019.05.06", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x5fd8c8e7>], :response=>{"index"=>{"_index"=>"logstash-2019.05.06", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"The [default] mapping cannot be updated on index [logstash-2019.05.06]: defaults mappings are not useful anymore now that indices can have at most one type."}}}}
[2019-05-06T15:16:46,782][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://10.3.3.30:9200/][Manticore::SocketException] Connection reset {:url=>http://10.3.3.30:9200/, :error_message=>"Elasticsearch Unreachable: [http://10.3.3.30:9200/][Manticore::SocketException] Connection reset", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2019-05-06T15:16:46,799][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://10.3.3.30:9200/][Manticore::SocketException] Connection reset", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2019-05-06T15:16:46,898][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.3.3.30:9200/"}
[2019-05-06T15:17:48,863][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[logstash-2019.05.06][1] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[logstash-2019.05.06][1]] containing [index {[logstash-2019.05.06][doc][itAAjGoBnMerGV6spaR7], source[{\"@version\":\"1\",\"received_from\":\"10.3.3.214\",\"syslog_timestamp\":\"May  6 15:16:51\",\"@timestamp\":\"2019-05-06T07:16:51.000Z\",\"syslog_message\":\"%SYS-5-CONFIG_I: Configured from console by igsadmin on vty4 (10.3.2.26)\",\"received_at\":\"2019-05-06T07:16:46.605Z\",\"type\":\"syslog\",\"host\":\"10.3.3.214\",\"syslog_program\":\"Cli\",\"syslog_hostname\":\"z3leo-r01\",\"message\":\"<165>May  6 15:16:51 z3leo-r01 Cli: %SYS-5-CONFIG_I: Configured from console by igsadmin on vty4 (10.3.2.26)\"}]}]]"})
[2019-05-06T15:17:48,867][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[2019-05-06T15:18:11,897][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://10.3.3.30:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://10.3.3.30:9200/, :error_message=>"Elasticsearch Unreachable: [http://10.3.3.30:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2019-05-06T15:18:11,898][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://10.3.3.30:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2019-05-06T15:18:11,992][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.3.3.30:9200/"}
[2019-05-06T15:18:48,867][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[logstash-2019.05.06][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[logstash-2019.05.06][0]] containing [index {[logstash-2019.05.06][doc][nNABjGoBnMerGV6sj6Ts], source[{\"@version\":\"1\",\"received_from\":\"10.3.3.214\",\"syslog_timestamp\":\"May  6 15:17:53\",\"@timestamp\":\"2019-05-06T07:17:53.000Z\",\"syslog_message\":\"%SYS-5-CONFIG_STARTUP: Startup config saved from system:/running-config by igsadmin on vty4 (10.3.2.26).\",\"received_at\":\"2019-05-06T07:17:48.754Z\",\"type\":\"syslog\",\"host\":\"10.3.3.214\",\"syslog_program\":\"Cli\",\"syslog_hostname\":\"z3leo-r01\",\"message\":\"<165>May  6 15:17:53 z3leo-r01 Cli: %SYS-5-CONFIG_STARTUP: Startup config saved from system:/running-config by igsadmin on vty4 (10.3.2.26).\"}]}]]"})
[2019-05-06T15:18:48,868][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[2019-05-06T15:18:50,898][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[logstash-2019.05.06][1] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[logstash-2019.05.06][1]] containing [index {[logstash-2019.05.06][doc][ndABjGoBnMerGV6sl6Te], source[{\"@version\":\"1\",\"received_from\":\"10.3.3.214\",\"syslog_timestamp\":\"May  6 15:16:51\",\"@timestamp\":\"2019-05-06T07:16:51.000Z\",\"syslog_message\":\"%SYS-5-CONFIG_I: Configured from console by igsadmin on vty4 (10.3.2.26)\",\"received_at\":\"2019-05-06T07:16:46.605Z\",\"type\":\"syslog\",\"host\":\"10.3.3.214\",\"syslog_program\":\"Cli\",\"syslog_hostname\":\"z3leo-r01\",\"message\":\"<165>May  6 15:16:51 z3leo-r01 Cli: %SYS-5-CONFIG_I: Configured from console by igsadmin on vty4 (10.3.2.26)\"}]}]]"})
[2019-05-06T15:18:50,899][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[2019-05-06T15:19:13,908][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
~