Can't get it working with cowrie->filebeat->logstash->elasticsearch->kibana
My cowrie logs to file, which is read by filebeat.
Filebeat version
$ /usr/share/filebeat/bin/filebeat version
filebeat version 6.1.1 (arm), libbeat 6.1.1
Filebeat config
$ sudo cat /etc/filebeat/filebeat.yml
filebeat.modules:
filebeat.prospectors:
- input_type: log
type: log
enabled: true
paths:
- /home/cowrie/cowrie/var/log/cowrie/cowrie.json*
encoding: plain
fields:
document_type: cowrie
registry_file: /var/lib/filebeat/registry
output.logstash:
hosts: ["192.168.10.6:5044"]
shipper:
logging:
to_syslog: false
to_files: true
files:
path: /var/log/filebeat/
name: mybeat
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7
level: info
Logstash version
# /opt/logstash/bin/logstash --version
logstash 6.4.3
Logstash config
# cat /etc/logstash/conf.d/cowrie.conf
input {
beats {
type => "beats"
port => 5044 # Pick an available port to listen on
host => "0.0.0.0"
}
}
filter {
if [type] == "cowrie" {
json {
source => message
}
date {
match => [ "timestamp", "ISO8601" ]
}
if [src_ip] {
dns {
reverse => [ "src_host", "src_ip" ]
action => "append"
}
geoip {
source => "src_ip" # With the src_ip field
target => "geoip" # Add the geoip one
# Using the database we previously saved
database => "/opt/logstash/vendor/geoip/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
# Get the ASN code as well
#geoip {
#source => "src_ip"
#database => "/opt/logstash/vendor/geoip/GeoIPASNum.dat"
#}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
}
}
}
output {
if [type] == "cowrie" {
# Output to elasticsearch
elasticsearch {
hosts => ["127.0.0.1:9200"] # Provided elasticsearch is listening on that host:port
#sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
file {
path => "/tmp/cowrie-logstash.log"
codec => json
}
# For debugging
stdout {
codec => rubydebug
}
}
}
Logstash seems to be picking up something
Logstash logfile excerpt
...
[DEBUG] 2019-01-26 21:54:14.493 [Ruby-0-Thread-8: :1] pipeline - filter received {"event"=>{"host"=>"hunnipi", "source"=>"/home/cowrie/cowrie/var/log/cowrie/cowrie.json", "@version"=>"1", "message"=>"{\"eventid\": \"cowrie.direct-tcpip.request\", \"timestamp\": \"2019-01-26T20:54:12.854963Z\", \"dst_ip\": \"ya.ru\", \"src_ip\": \"5.188.86.208\", \"session\": \"72993b90b7b8\", \"dst_port\": 443, \"src_port\": 0, \"message\": \"direct-tcp connection request to ya.ru:443 from ::1:0\", \"sensor\": \"hunnipi\"}", "tags"=>["beats_input_codec_plain_applied"], "type"=>"beats", "beat"=>{"version"=>"6.1.1", "name"=>"hunnipi", "hostname"=>"hunnipi"}, "@timestamp"=>2019-01-26T20:54:13.347Z, "offset"=>53203349, "prospector"=>{"type"=>"log"}, "fields"=>{"document_type"=>"cowrie"}}}
[DEBUG] 2019-01-26 21:54:14.494 [Ruby-0-Thread-8: :1] pipeline - output received {"event"=>{"host"=>"hunnipi", "source"=>"/home/cowrie/cowrie/var/log/cowrie/cowrie.json", "@version"=>"1", "message"=>"{\"eventid\": \"cowrie.direct-tcpip.request\", \"timestamp\": \"2019-01-26T20:54:12.854963Z\", \"dst_ip\": \"ya.ru\", \"src_ip\": \"5.188.86.208\", \"session\": \"72993b90b7b8\", \"dst_port\": 443, \"src_port\": 0, \"message\": \"direct-tcp connection request to ya.ru:443 from ::1:0\", \"sensor\": \"hunnipi\"}", "tags"=>["beats_input_codec_plain_applied"], "type"=>"beats", "beat"=>{"version"=>"6.1.1", "name"=>"hunnipi", "hostname"=>"hunnipi"}, "@timestamp"=>2019-01-26T20:54:13.347Z, "offset"=>53203349, "prospector"=>{"type"=>"log"}, "fields"=>{"document_type"=>"cowrie"}}}
...
No matching index found in elasticsearch
# curl 'http://localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-2019.01.26 HZ9AdcbSSM69MQw-4xXDhA 5 1 241 0 232.7kb 232.7kb
green open .kibana _GwWwwVuS_O-wA3mEkPenw 1 0 2 0 10.7kb 10.7kb
Thanks
/jon