Thanks a lot. Let me try.
@guyboertje,
I just tested the config and getting the below error.
None of the log from first firewall was hit. Only logs from second firewall are showing in ELK.
It said date format can't parse. Any idea?
Error From Logstash
[2018-05-02T16:20:18,245][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2018.05.02", :_type=>"t_nettraffic", :_routing=>nil}, 2018-05-02T08:20:18.123Z 10.10.10.10 %{message}], :response=>{"index"=>{"_index"=>"logstash-2018.05.02", "_type"=>"t_nettraffic", "_id"=>"AGEREERTEREWDFWERSDFRSD", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [date]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "2018-05-02 08:20:18 +0000" is malformed at " 08:20:18 +0000""}}}}}
+++++++++++++++++++++++++++++++++++++++++++
Here is the log from elasticsearch.log
[2018-05-02T16:29:27,523][DEBUG][o.e.a.b.TransportShardBulkAction] [70m8uRn] [logstash-2018.05.02][0] failed to execute bulk item (index) BulkShardRequest [[logstash-2018.05.02][0]] containing [index {[logstash-2018.05.02][t_nettraffic][AWMf-HhjeHqNa56vZixR], source[{"date":"2018-05-02 08:29:27 +0000","bytes_written_to_client":"41667","srcip":"10.10.10.10","dstport":"443","bytes_written_to_server":"2426","pid":"18895","type":"t_nettraffic","hostname":"XXX.local.host","host":"x.x.x.x","devname":"FIREWALL1","dstip":"x.x.x.x.x","event":"session end","rule_name":"TESTING RULE","pri":"p_major","priority":"45","dst_geo":"HK","srczone":"VL_1","start_time":"2018-05-02 08:27:48 +0000","@timestamp":"2018-05-02T08:29:27.515Z","application":"TCP 443","proto":"6","srcport":"21396","dstzone":"VL_EXT","logid":"0","cmd":"httpp"}]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [date]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:298) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:468) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.DocumentParser.parseDynamicValue(DocumentParser.java:816) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:598) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:396) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:373) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:66) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:277) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:530) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:507) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.bulk.TransportShardBulkAction.prepareIndexOperationOnPrimary(TransportShardBulkAction.java:459) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:467) ~[elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:146) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:115) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:70) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:975) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:944) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:345) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:270) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:924) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:921) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1659) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:933) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:92) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:291) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:266) [elasticsearch-5.6.9.jar:5.6.9]
at
... 37 moreThis topic was automatically closed 28 days after the last reply. New replies are no longer allowed.