I am receiving syslogs like this: Oct 5 13:37:13 10.20.30.40 %ASA-4-733100: [ HTTP 80] drop rate-1 exceeded. Current burst rate is 7 per second, max configured rate is 40; Current average rate is 23 per second, max configured rate is 20; Cumulative total count is 14207
My grok statement, which works on herokuapp grok debugger, is:
The message field in kibana only shows the below, ie the fields that I want parsed no longer exist but have not been parsed. (I do NOT have overwrite configured in my config file anywhere and other logs that are parsed or grokparsefailure always include the full message). I was expecting the message to show the entire log, but it seems the leading [ seems to cause some odd behavior.:
] drop rate-1 exceeded. Current burst rate is 7 per second, max configured rate is 40; Current average rate is 23 per second, max configured rate is 20; Cumulative total count is 14207
Are you saying it should be : %{CISCOTIMESTAMP:log_date} %{IP:log_source_ip} \%%{DATA:asa_code}%:{GREEDYDATA:message}
This fails in the herokuapp because, I believe, the % is meant to preceed the {} declaration of a new field and with the semi-colon moved it no longer does. ie I want it to match anything after (but not including) the literal : which follows the asa_code field and capture that as the message.
I appreciate your continued help but I'm not sure I understand what you're getting at here. You seem to have removed the % which is required for the {GREEDYDATA:message} grok filter, rendering it invalid. Unless you're referring to one of the two % in \%%{DATA:asa_code} which are used to match a literal % in the string, and to declare the asa_code filter respectively.
Can you clarify for me? I should mention that this grok filter works great for all messages I receive from this device, except these ones with the leading square bracket.
There was a space missing... apologies I am doing too many things
Here is tha from Kibana Dev Tools
Oct 5 13:37:13 10.20.30.40 %ASA-4-733100: [ HTTP 80] drop rate-1 exceeded. Current burst rate is 7 per second, max configured rate is 40; Current average rate is 23 per second, max configured rate is 20; Cumulative total count is 14207
Note there is a space after the : in the \%%{DATA:asa_code}: before %{GREEDYDATA:message}
{
"asa_code": "ASA-4-733100",
"log_date": "Oct 5 13:37:13",
"log_source_ip": "10.20.30.40",
"message": "[ HTTP 80] drop rate-1 exceeded. Current burst rate is 7 per second, max configured rate is 40; Current average rate is 23 per second, max configured rate is 20; Cumulative total count is 14207"
}
Hi Stephen,
I noticed i still have the issue, but after looking directly at the syslog-ng file the logs are read from I can see that, for some reason, these particular messages with the leading [ have a line feed and numerous spaces inserted into them. This is the reason the parsing is failing as the message falls over multiple lines.
So on another syslog server the message shows as:
[ LOCAL\user@user.com ] drop rate-1 exceeded. Current burst rate is 41 per second, max configured rate is 40; Current average rate is 1 per second, max configured rate is 20; Cumulative total count is 835
But on syslog-ng (where logstash reads from) it shows as:
Oct 7 08:46:09 10.210.13.3 %ASA-4-733100: [ LOCAL\user@user.com
] drop rate-1 exceeded. Current burst rate is 64 per second, max configured rate is 40; Current average rate is 2 per second, max configured rate is 20; Cumulative total count is 1294
This only occurs on a tiny minority of my logs so i'm content to just leave it as is rather than tshoot syslog-ng. Just thought I'd post an update.
Cheers again for the help.
Joe
Interesting... there would probably be a way to fix that as well... There are some tricks with multiline that might work. @badger is the master parser perhaps if / when they, take' a look there might be a suggestion.
Are you suggesting I use that line in the grok config? This type of log is a tiny sub percentage of the logs I am receiving, so wary of putting in any config that might affect logs from the others.
If I open the log file in any text editor that shows these characters then it shows as a LF followed by about 5 spaces so it looks like an embedded line feed to me and i'd be surprised if it was coming that way from the source - although without doing a packet capture (which would be very large as the source is very talkative) I can't confirm for sure.
If you have any suspects spring to mind then let me know, otherwise I wouldn't worry about it. I'm happy to let these anomalies occur as they're quite isolated.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.