Logstash proftpd log

Elastic Stack Logstash
1

Hi guys,

I'm working on logstash and I want to send proftpd and sftp logs to elasticsearch. But I'm facing some issues with doing that.

No index is genreated in ES.

Can any one help me :slightly_smiling:

File conf:

#input {

file {

'path' => ['/var/log/proftpd/sftp.log']

'type' => 'system logs'

}

#}

input {
file {
type => "proftpd-common"
path => "/var/log/proftpd/*log"
}
}
filter {
if [type] == 'proftpd-common' {
grok {
match => [
"message", "%{IPORHOST:clientip} %{WORD:indent} %{USER:ftpuser} [%{HTTPDATE:timestamp}] "%{WORD:command}(?:%{SPACE}%{DATA:request}|%{SPACE})" (?:%{NUMBER:response}|-) (?:%{NUMBER:bytes}|-)"
]
}

    date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
    }
}

}
output {
elasticsearch {
codec => json
hosts => "localhost:9200"
index => "logstash"
}

}

log.jpg1342×865 520 KB

2

Try to add index in the following format:
index => "logstash-test-%{+YYYY.MM.dd}"

3

There should not be any JSON codec within the Elasticsearch output plugin, as it converts the event to JSON automatically.

4

Thanks guys I'm testing right now :slightly_smiling:

5

Hi,

I made some tests and I have the same issue.

I have this message:

Using mapping template from {:path=>nil, :level=>:info}
Attempting to install template {:manage_template=>{"template"=>"logstash-", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true, "ignore_above"=>256}}}}}, {"float_fields"=>{"match"=>"", "match_mapping_type"=>"float", "mapping"=>{"type"=>"float", "doc_values"=>true}}}, {"double_fields"=>{"match"=>"", "match_mapping_type"=>"double", "mapping"=>{"type"=>"double", "doc_values"=>true}}}, {"byte_fields"=>{"match"=>"", "match_mapping_type"=>"byte", "mapping"=>{"type"=>"byte", "doc_values"=>true}}}, {"short_fields"=>{"match"=>"", "match_mapping_type"=>"short", "mapping"=>{"type"=>"short", "doc_values"=>true}}}, {"integer_fields"=>{"match"=>"", "match_mapping_type"=>"integer", "mapping"=>{"type"=>"integer", "doc_values"=>true}}}, {"long_fields"=>{"match"=>"", "match_mapping_type"=>"long", "mapping"=>{"type"=>"long", "doc_values"=>true}}}, {"date_fields"=>{"match"=>"", "match_mapping_type"=>"date", "mapping"=>{"type"=>"date", "doc_values"=>true}}}, {"geo_point_fields"=>{"match"=>"", "match_mapping_type"=>"geo_point", "mapping"=>{"type"=>"geo_point", "doc_values"=>true}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "doc_values"=>true}, "@version"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true}, "geoip"=>{"type"=>"object", "dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip", "doc_values"=>true}, "location"=>{"type"=>"geo_point", "doc_values"=>true}, "latitude"=>{"type"=>"float", "doc_values"=>true}, "longitude"=>{"type"=>"float", "doc_values"=>true}}}}}}}, :level=>:info}

Best regards

Ismael

6

Hi guys,

Thanks for your help. I solved the issue, it's related to my filter and didn't add start_position => "beginning" in the input file.

Have a nice day.

Best regards

Ismael

7

Do you mind sharing your proftpd ELK config files with the community?

8

Hi,

input {
file {
path => "/var/log/proftpd/sftp-xferlog"
start_position => "beginning"
}
}
filter {
grok {
match => [ "message", "(?\w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} \d{4}) (?\d*) (?\b\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\b) (?\d*) (?\S*) (?[a|b]) (?\S{1}) (?\S{1}) (?\S{1}) (?\S*) (?\S*) (?\S*) (?\S*) (?\S*)" ]
}

 date {
	 match    => [ "time", "EEE MMM dd HH:mm:ss YYYY" ]
	 timezone => [ "Europe/Paris" ]  # Change to your local timezone
 }

 geoip {
	 source   => [ "remoteHost" ]
 }

}
output {
elasticsearch {
hosts => "localhost:9200"
index => "mft_r7"
}
stdout {}
}

Best regards

Ismael