Elasticsearch Errors - Failed action [500]

kopacko · March 20, 2017, 4:25pm

I am setting up a new index on my cluster just for my firewall logs. I average about 5 million per hour across all my sites. Upon getting the index defined and configuring Logstash to target that index, I am now getting this message on the ES cluster.

[2017-03-20T11:06:33,053][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>500,
 :action=>["index",
 {:_id=>nil,
 :_index=>"logstash-network-firewall-2017.03.20",
 :_type=>"logs",
 :_routing=>nil},

 <13>2017-03-20T16:06:25.536Z [ *** CISCO SYSLOG ***],

 :response=>{"index"=>{"_index"=>"logstash-network-firewall-2017.03.20",
 "_type"=>"logs",
 "_id"=>"AVrseFL00CXnw_LoCboQ",
 "status"=>500,
 "error"=>{"type"=>"illegal_state_exception",
 "reason"=>"Message not fully read (request) for requestId [265947], action [indices:data/write/bulk[s]], available [319600]; resetting"}}}}

kopacko · March 20, 2017, 4:49pm

UPDATE:

I appears that a few documents get added to the index and then no more.

kopacko · March 20, 2017, 4:52pm

Here is my index config:

$ curl -XGET 'http://localhost:9200/_template/*?pretty'
    {
      "logstash-network-firewall" : {
        "order" : 0,
        "template" : "logstash-network-firewall*",
        "settings" : {
          "index" : {
            "mapping" : {
              "total_fields" : {
                "limit" : "2000"
              }
            },
            "refresh_interval" : "10s",
            "number_of_shards" : "3",
            "number_of_replicas" : "0"
          }
        },
        "mappings" : { },
        "aliases" : { }
      }
    }

kopacko · March 20, 2017, 5:13pm

Here is my Logstash config:

##########
# ELASTICSAERCH Output Parameters
##########
output {
  if [pipeline] =~ /(?i)(LOGFIREWALL|LOGSFR)/ {
    elasticsearch {
      hosts => ["http://x.x.x.x:9200","http://x.x.x.x:9200"]
      index => "logstash-network-firewall-%{+YYYY.MM.dd}"
    }
#   stdout { codec => rubydebug }
  }
}
##################################################

kopacko · March 21, 2017, 2:55pm

Anyone? Any suggestions?

warkolm · March 21, 2017, 9:37pm

Is there anything in your ES logs?

kopacko · March 21, 2017, 10:11pm

Shoot, thought I had posted one. Hang on.

kopacko · March 22, 2017, 7:06pm

Elasticsearch log (Part 1):

[2017-03-22T14:26:43,320][DEBUG][o.e.a.b.TransportShardBulkAction] [DATANODE-06] [logstash-2017.03.22][0] failed to execute bulk item (index) index {[logstash-2017.03.22][logs][AVr3RV-up8MpPZKftHcN], source[{"responder_packets":"1",

[***SYSLOG MESSAGE***]

org.elasticsearch.index.mapper.MapperParsingException: failed to parse
        at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:175) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:275) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:533) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:510) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:196) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:201) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:348) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.index(TransportShardBulkAction.java:155) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.handleItem(TransportShardBulkAction.java:134) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:120) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:73) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:76) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:49) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:914) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:884) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:327) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:262) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:864) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:861) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:147) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1652) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:873) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:92) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:279) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:258) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:250) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.2.jar:5.2.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

kopacko · March 22, 2017, 7:07pm

Elasticsearch log (Part 2):

Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in name
 at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@6a8b26d7; line: 1, column: 359]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._throwUnquotedSpace(ParserMinimalBase.java:522) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(UTF8StreamJsonParser.java:1963) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseName(UTF8StreamJsonParser.java:1925) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1747) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:776) ~[jackson-core-2.8.6.jar:2.8.6]
        at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:55) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:397) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:372) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:66) ~[elasticsearch-5.2.2.jar:5.2.2]
        ... 34 more

FYI, the character limit in these posts is ridiculously small.

kopacko · March 22, 2017, 8:02pm

Hoping this helps.

Whatever is wrong, I am completely down now and have been for a couple days.

warkolm · March 22, 2017, 11:04pm

Looks like some bad chars somewhere.

Use gist/pastebin/etc

kopacko · March 22, 2017, 11:20pm

I am getting it from all sources. No matter what is sending in, syslog, winlogbeat, packetbeat, etc.

I am sure it is something I screwed up on my cluster config....I just don't know where.

kopacko · March 23, 2017, 5:32pm

Any other suggestions on things to try or do?

kopacko · March 24, 2017, 2:15pm

This is really getting old.

system · April 21, 2017, 2:15pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Failed Action Elasticsearch	11	1375	April 3, 2017
Failed to index Bulk Items Error in ES Logs Elasticsearch	2	3402	July 5, 2017
Logstash errors retrying failed action with response code 500 Logstash	3	729	September 15, 2020
:message=>"Failed action. ", :status=>404, :action=>["index", Logstash	13	3127	July 6, 2017
How to rectify this error? Logstash	6	1350	May 17, 2019

Elasticsearch Errors - Failed action [500]

Related topics