Elasticsearch Errors - Failed action [500]

I am setting up a new index on my cluster just for my firewall logs. I average about 5 million per hour across all my sites. Upon getting the index defined and configuring Logstash to target that index, I am now getting this message on the ES cluster.

[2017-03-20T11:06:33,053][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>500,
 :action=>["index",
 {:_id=>nil,
 :_index=>"logstash-network-firewall-2017.03.20",
 :_type=>"logs",
 :_routing=>nil},

 <13>2017-03-20T16:06:25.536Z [ *** CISCO SYSLOG ***],

 :response=>{"index"=>{"_index"=>"logstash-network-firewall-2017.03.20",
 "_type"=>"logs",
 "_id"=>"AVrseFL00CXnw_LoCboQ",
 "status"=>500,
 "error"=>{"type"=>"illegal_state_exception",
 "reason"=>"Message not fully read (request) for requestId [265947], action [indices:data/write/bulk[s]], available [319600]; resetting"}}}}

UPDATE:

I appears that a few documents get added to the index and then no more.

Here is my index config:

$ curl -XGET 'http://localhost:9200/_template/*?pretty'
    {
      "logstash-network-firewall" : {
        "order" : 0,
        "template" : "logstash-network-firewall*",
        "settings" : {
          "index" : {
            "mapping" : {
              "total_fields" : {
                "limit" : "2000"
              }
            },
            "refresh_interval" : "10s",
            "number_of_shards" : "3",
            "number_of_replicas" : "0"
          }
        },
        "mappings" : { },
        "aliases" : { }
      }
    }

Here is my Logstash config:

##########
# ELASTICSAERCH Output Parameters
##########
output {
  if [pipeline] =~ /(?i)(LOGFIREWALL|LOGSFR)/ {
    elasticsearch {
      hosts => ["http://x.x.x.x:9200","http://x.x.x.x:9200"]
      index => "logstash-network-firewall-%{+YYYY.MM.dd}"
    }
#   stdout { codec => rubydebug }
  }
}
##################################################

Anyone? Any suggestions?

Is there anything in your ES logs?

Shoot, thought I had posted one. Hang on.

Elasticsearch log (Part 1):

[2017-03-22T14:26:43,320][DEBUG][o.e.a.b.TransportShardBulkAction] [DATANODE-06] [logstash-2017.03.22][0] failed to execute bulk item (index) index {[logstash-2017.03.22][logs][AVr3RV-up8MpPZKftHcN], source[{"responder_packets":"1",

[***SYSLOG MESSAGE***]

org.elasticsearch.index.mapper.MapperParsingException: failed to parse
        at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:175) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:275) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:533) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:510) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:196) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:201) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:348) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.index(TransportShardBulkAction.java:155) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.handleItem(TransportShardBulkAction.java:134) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:120) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:73) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:76) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:49) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:914) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:884) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:327) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:262) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:864) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:861) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:147) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1652) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:873) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:92) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:279) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:258) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:250) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.2.jar:5.2.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

Elasticsearch log (Part 2):

Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in name
 at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@6a8b26d7; line: 1, column: 359]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._throwUnquotedSpace(ParserMinimalBase.java:522) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(UTF8StreamJsonParser.java:1963) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseName(UTF8StreamJsonParser.java:1925) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1747) ~[jackson-core-2.8.6.jar:2.8.6]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:776) ~[jackson-core-2.8.6.jar:2.8.6]
        at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:55) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:397) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:372) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:66) ~[elasticsearch-5.2.2.jar:5.2.2]
        ... 34 more

FYI, the character limit in these posts is ridiculously small.

Hoping this helps.

Whatever is wrong, I am completely down now and have been for a couple days.

Looks like some bad chars somewhere.

Use gist/pastebin/etc :slight_smile:

I am getting it from all sources. No matter what is sending in, syslog, winlogbeat, packetbeat, etc.

I am sure it is something I screwed up on my cluster config....I just don't know where.

Any other suggestions on things to try or do?

This is really getting old.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.