Elasticsearch Errors - Failed action [500]

kopacko · March 20, 2017, 4:25pm

I am setting up a new index on my cluster just for my firewall logs. I average about 5 million per hour across all my sites. Upon getting the index defined and configuring Logstash to target that index, I am now getting this message on the ES cluster.

[2017-03-20T11:06:33,053][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>500,
 :action=>["index",
 {:_id=>nil,
 :_index=>"logstash-network-firewall-2017.03.20",
 :_type=>"logs",
 :_routing=>nil},

 <13>2017-03-20T16:06:25.536Z [ *** CISCO SYSLOG ***],

 :response=>{"index"=>{"_index"=>"logstash-network-firewall-2017.03.20",
 "_type"=>"logs",
 "_id"=>"AVrseFL00CXnw_LoCboQ",
 "status"=>500,
 "error"=>{"type"=>"illegal_state_exception",
 "reason"=>"Message not fully read (request) for requestId [265947], action [indices:data/write/bulk[s]], available [319600]; resetting"}}}}

kopacko · March 20, 2017, 4:49pm

UPDATE:

I appears that a few documents get added to the index and then no more.

kopacko · March 20, 2017, 4:52pm

Here is my index config:

$ curl -XGET 'http://localhost:9200/_template/*?pretty'
    {
      "logstash-network-firewall" : {
        "order" : 0,
        "template" : "logstash-network-firewall*",
        "settings" : {
          "index" : {
            "mapping" : {
              "total_fields" : {
                "limit" : "2000"
              }
            },
            "refresh_interval" : "10s",
            "number_of_shards" : "3",
            "number_of_replicas" : "0"
          }
        },
        "mappings" : { },
        "aliases" : { }
      }
    }

kopacko · March 20, 2017, 5:13pm

Here is my Logstash config:

##########
# ELASTICSAERCH Output Parameters
##########
output {
  if [pipeline] =~ /(?i)(LOGFIREWALL|LOGSFR)/ {
    elasticsearch {
      hosts => ["http://x.x.x.x:9200","http://x.x.x.x:9200"]
      index => "logstash-network-firewall-%{+YYYY.MM.dd}"
    }
#   stdout { codec => rubydebug }
  }
}
##################################################

kopacko · March 21, 2017, 2:55pm

Anyone? Any suggestions?

warkolm · March 21, 2017, 9:37pm

Is there anything in your ES logs?

kopacko · March 21, 2017, 10:11pm

Shoot, thought I had posted one. Hang on.

kopacko · March 22, 2017, 7:06pm

Elasticsearch log (Part 1):

[2017-03-22T14:26:43,320][DEBUG][o.e.a.b.TransportShardBulkAction] [DATANODE-06] [logstash-2017.03.22][0] failed to execute bulk item (index) index {[logstash-2017.03.22][logs][AVr3RV-up8MpPZKftHcN], source[{"responder_packets":"1",

[***SYSLOG MESSAGE***]

org.elasticsearch.index.mapper.MapperParsingException: failed to parse
        at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:175) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:275) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:533) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:510) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:196) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:201) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:348) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.index(TransportShardBulkAction.java:155) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.handleItem(TransportShardBulkAction.java:134) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:120) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:73) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:76) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:49) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:914) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:884) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:327) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:262) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:864) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:861) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:147) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1652) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:873) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:92) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:279) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:258) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:250) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.2.jar:5.2.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

kopacko · March 22, 2017, 7:07pm

Elasticsearch log (Part 2):

Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in name
 at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@6a8b26d7; line: 1, column: 359]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._throwUnquotedSpace(ParserMinimalBase.java:522) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(UTF8StreamJsonParser.java:1963) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseName(UTF8StreamJsonParser.java:1925) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1747) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:776) ~[jackson-core-2.8.6.jar:2.8.6]
        at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:55) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:397) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:372) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:66) ~[elasticsearch-5.2.2.jar:5.2.2]
        ... 34 more

FYI, the character limit in these posts is ridiculously small.

kopacko · March 22, 2017, 8:02pm

Hoping this helps.

Whatever is wrong, I am completely down now and have been for a couple days.

warkolm · March 22, 2017, 11:04pm

Looks like some bad chars somewhere.

Use gist/pastebin/etc

kopacko · March 22, 2017, 11:20pm

I am getting it from all sources. No matter what is sending in, syslog, winlogbeat, packetbeat, etc.

I am sure it is something I screwed up on my cluster config....I just don't know where.

kopacko · March 23, 2017, 5:32pm

Any other suggestions on things to try or do?

kopacko · March 24, 2017, 2:15pm

This is really getting old.

Topic		Replies	Views
Logstash errors retrying failed action with response code 500 Logstash	3	789	September 15, 2020
Failed Action Elasticsearch	11	1443	April 3, 2017
Failed action with response of 500, dropping action Logstash1.5.4 Logstash	3	1742	July 6, 2017
:message=>"Failed action. ", :status=>404, :action=>["index", Logstash	13	3198	July 6, 2017
Indexing error "Index -1 out of bounds for length 0" Logstash	10	2811	December 31, 2020

Elasticsearch Errors - Failed action [500]

Related topics