I'm parsing syslog from an AV server. Every log goes fine from the AV to a syslog server, then to a RMQ, and then to ElasticSearch (1.6) via Logstash (1.5.1). There is only log type that is giving me problems: the one that are similar to this:
Aug 26 18:28:18 SymantecServer tetsuo: Virus found,IP Address: 192.168.10.58,Computer name: kerrigan,Source: Real Time Scan,Risk name: EICAR Test String,Occurrences: 1,D:\Users\sacha\Desktop\lala.txt,'',Actual action: Cleaned by deletion,Requested action: Cleaned,Secondary action: Quarantined,Event time: 2015-08-26 22:27:01,Inserted: 2015-08-26 22:28:18,End: 2015-08-26 22:27:01,Last update time: 2015-08-26 22:28:18,Domain: Default,Group: My Company\Equipos de Trabajo,Server: tetsuo,User: sacha,Source computer: ,Source IP: ,Disposition: Reputation was not used in this detection.,Download site: ,Web domain: ,Downloaded by: ,Prevalence: Reputation was not used in this detection.,Confidence: Reputation was not used in this detection.,URL Tracking Status: Off,,First Seen: Reputation was not used in this detection.,Sensitivity: ,MDS,Application hash: 275A021BBFB6489E54D471899F7DB9D1663FC695EC2FE2A2C4538AABF651FD0F,Hash type: SHA2,Company name: ,Application name: eicar.txt,Application version: ,Application type: 127,File size (bytes): 68,Category set: Malware,Category type: Virus
Part of the filter that parses this kind of logs is:
grok { match => ["message", "^(?<risk_type>[^,]+),IP\sAddress:\s(?<dest_ip>[^,]+),Computer name: %{USERNAME:computer},Source:\s(?<scan_type>[^,]+),Risk\sname:\s(?<signature>[^,]+),Occurrences:\s(?<occurrences>\d+),(?<process>\w:\\[^,]+),(?<file_path>(.+)(\\|/))?(?<file_name>[^,]+),Actual\saction:\s(?<actual_action>[^,]+),Requested action:\s(?<requested_action>[^,]+),Secondary action:\s(?<secondary_action>[^,]+),Event\stime:\s(?<event_time>[^,]+),Inserted:\s(?<inserted>[^,]+),End:\s(?<end_time>[^,]+),Last\supdate\stime:\s(?<last_update_time>[^,]+)...
Here is the whole grok: http://paste.ubuntu.com/12202957/
I tested the GROK on http://grokdebug.herokuapp.com/ and works.
I also have this mapping on ES for this kind of logs:
"sep":{"properties":{
"action":{"index":"not_analyzed","type":"string"},
"action_field":{"index":"no","type":"string"},
"action_type":{"index":"no","type":"string"},
"actual_action":{"index":"no","type":"string"},
"admin":{"index":"not_analyzed","type":"string"},
"agt_info":{"index":"no","type":"string"},
"api":{"index":"not_analyzed","type":"string"},
"application":{"index":"not_analyzed","type":"string"},
"application_hash":{"index":"not_analyzed","type":"string"},
"application_name":{"index":"not_analyzed","type":"string"},
...
Here is the whole mapping: http://paste.ubuntu.com/12202956/
Every log goes fine but this ones... this is what I get:
failed action with response of 400, dropping action: ["index", {:_id=>nil, :_index=>"syslog_sep-euroamerica-2015.08.26", :_type=>"sep", :_routing=>nil}, #<LogStash::Event:0x49f27104 @metadata_accessors=#<LogStash::Util::Accessors:0x32e140a6 @store={"retry_count"=>0}, @lut={}>, @cancelled=false, @data={"message"=>"Virus found,IP Address: ...,
Here is the whole error: http://paste.ubuntu.com/12202964/
Even is only a warning, this is the most important logs I have (the one that detected virus!).
I changed the parsing in every way I imagine, and I still don't know where is the problem.
Any clues?
Sacha,.
)