I have filebeat running on one of my servers which collects iptables and telnet logs, both feeds use the geoip filter plugin and both feed into the logstash for parsing and then subsequently into elasticsearch for storing. However, I am getting the following error when trying to index the telnet logs though the iptables ones work fine:
All running 5.0.1
Error on Logstash:
[2016-11-22T11:07:08,956][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"telnet-2016.11.22", :_type=>"telnet", :_routing=>nil}, 2016-11-22T11:06:59.000Z lon-01-collect-001 %{message}], :response=>{"index"=>{"_index"=>"telnet-2016.11.22", "_type"=>"telnet", "_id"=>"AViLt88Y87jAoXX9m1im", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"parse_exception", "reason"=>"geo_point expected"}}}}}
filebeat.yml
filebeat.prospectors:
- input_type: log
paths:
- /var/log/iptables.log
document_type: iptables
- input_type: log
paths:
- /var/log/telnet.log
document_type: telnet
output.logstash:
hosts: ["MY_LOGSTASH_SERVER:5044"]
logstashconfig.conf
input {
beats {
port => 5044
}
}
filter {
if [type] == "iptables" {
grok {
patterns_dir => ["/etc/logstash/patterns/"]
match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{SYSLOGHOST} %{SYSLOGPROG} IPTables-Dropped: %{IPTABLES}" }
}
date {
match => [ "log_timestamp", "ISO8601" ]
}
} else if [type] == "telnet" {
grok {
match => { "message" => "%{INT:log_timestamp},%{IP:src_ip},%{WORD:username},%{WORD:password}" }
}
date {
match => [ "log_timestamp", "UNIX" ]
}
}
geoip {
source => "src_ip"
}
mutate {
remove_field => [ "log_timestamp", "message" ]
}
}
output {
if [type] == "iptables" {
elasticsearch {
hosts => [ "MY_ELASTICSEARH_SERVER:9200" ]
index => [ "iptables-%{+YYYY.MM.dd}" ]
manage_template => false
}
} else if [type] == "telnet" {
elasticsearch {
hosts => [ "MY_ELASTICSEARH_SERVER:9200" ]
index => [ "telnet-%{+YYYY.MM.dd}" ]
manage_template => false
}
}
}
Sample telnet log:
1479813223,220.133.42.168,admin,xmhdipc
Logstash file output of the log:
{
"geoip": {
"timezone": null,
"ip": "81.19.139.197",
"latitude": 55.7386,
"continent_code": "EU",
"city_name": null,
"country_code2": "RU",
"country_name": "Russia",
"dma_code": null,
"country_code3": "RU",
"region_name": null,
"location": [37.6068, 55.7386],
"postal_code": null,
"longitude": 37.6068,
"region_code": null
},
"offset": 346958,
"input_type": "log",
"source": "/var/log/telnet.log",
"type": "telnet",
"tags": ["beats_input_codec_plain_applied"],
"src_ip": "81.19.139.197",
"password": "1111",
"@timestamp": "2016-11-22T11:18:17.000Z",
"@version": "1",
"beat": {
"hostname": "lon-01-collect-001",
"name": "lon-01-collect-001",
"version": "5.0.1"
},
"host": "lon-01-collect-001",
"username": "root"
}