Import txt log data to cluster


(Amin) #1

Hello all,
I want to import some .TXT log to my elasticsearch cluster via Logstash.
my logstash configuration is as follow :
please help me how config my logstash and which filter is suitable for this job ?

Data Format :

date=2018-05-04 time=23:49:32 itime=1525461572 threat=ip_src_session threatlevel=4 threattype=ips logver=52 logid=18432 type=anomaly subtype=anomaly level=alert devid=FG600C3913802373 sessionid=0 attackid=16777322 severity=critical srcip=5.127.45.0 dstip=192.168.148.24 srcport=18137 dstport=8010 srcintf=Internet action=clear_session proto=6 service=IntelliLAN_8010/TCP ref=http://www.fortinet.com/ids/VID16777322 count=13 msg="anomaly: ip_src_session, 41 > threshold 40, repeats 13 times" vd=root attack=ip_src_session crscore=50 crlevel=critical policyid=1 srccountry="Iran, Islamic Republic of"
date=2018-05-04 time=23:29:58 itime=1525460399 threat=ip_src_session threatlevel=4 threattype=ips logver=52 logid=18432 type=anomaly subtype=anomaly level=alert devid=FG600C3913802373 sessionid=0 attackid=16777322 severity=critical srcip=5.123.168.194 dstip=192.168.148.8 srcport=19934 dstport=8011 srcintf=Internet action=clear_session proto=6 service=IntelliLAN_8011/TCP ref=http://www.fortinet.com/ids/VID16777322 count=8 msg="anomaly: ip_src_session, 41 > threshold 40, repeats 8 times" vd=root attack=ip_src_session crscore=50 crlevel=critical policyid=1 srccountry="Iran, Islamic Republic of"

Logstash configuration :

input {
redis {
host => "172.24.69.9"
data_type => "list"
port => "6379"
key => "sadra"
}

redis {
host => "172.24.69.10"
data_type => "list"
port=> "6379"
key => "sadra"
}
}

filter {
kv { }
}

output {

elasticsearch {
hosts => [ "172.24.69.21:9200","172.24.69.14:9200"]
index => "pasdaranlog-%{+xxxx.ww}"
user => *****
password => *******

}
}


(Mark Walkom) #2

Using the KV filter makes sense, what problems are you having?


(Amin) #3

hello
how import log from file ?
best


(Mark Walkom) #4

Use the file input and you should be good.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.