Hello, everyone!
I have a lot of syslog messages. And when I trying to filter them I found that logstash losing some messages. How can I investigate why? And how can I found howq many of messages are missed ?
What does you config look like? Which inputs are you using?
input
{
syslog
{
port => 514
type => "syslog_Check"
}
}
filter
{
if [type] == "syslog_Check"
{
mutate {
gsub => [ "message", ".<134>1 ", "date:=", "message", "= ", " ", "message", ":"", "=", "message", ":=", "=", "message", "";", ";", "message", "; ", ";", "message", " .[", ";" ]
}
kv {field_split => ";"}
if
[message] =~ ".could not get SAs from packet." or
[message] =~ ".no proposal chosen." or
[message] =~ ".No valid SA." or
[message] =~ ".Packet is dropped because there is no valid SA." or
[message] =~ ".Phase one received notification from peer; payload malformed." or
[message] =~ ".Quick Mode failed to match proposal." or
[message] =~ ".Quick Mode Received Notification from Peer: invalid id information." or
[message] =~ ".Reject Reason: Gateway to Gateway authentication failure." or
[message] =~ ".Unknown SPI for IPsec packet." or
[message] =~ ".Virtual defragmentation error: Timeout." or
[message] =~ ".encryption failure: Failed to enforce VPN Policy (11)." or
[message] =~ ".encryption failure: no response from peer." or
[message] =~ ".encryption failure: Tunnel failure, unresolved SA (VPN Error code 01)." or
[message] =~ ".encryption failure: Unknown SPI." or
[message] =~ ".encryption failure: Wrong peer gateway for decrypted." or
[message] =~ ".failed to connect: Internal error- (CCC_E_GENERAL)." or
[message] =~ ".IKE: Aggressive Mode Failed to match proposal:." or
[message] =~ ".IKE: Main Mode cannot complete certificate chain." or
[message] =~ ".IKE: Main Mode Sent Notification to Peer: invalid certificate." or
[message] =~ ".IKE: Main Mode Sent Notification to Peer: invalid information." or
[message] =~ ".Invalid ID information." or
[message] =~ ".Invalid Peer Certificate." or
[message] =~ ".no common community for myself and peer." or
[message] =~ ".no proposal chosen." or
[message] =~ ".No valid SA." or
[message] =~ ".authentication failed." or
([user] and [action] == "Log In")
{
mutate {
rename => { "ike" => "metric"
"community" => "object_name"
"service" => "serv" }
add_field => {
"value" => 1
"service" => "CheckP"
"description" => "Syslog time = %{date}. Detected an ERROR. Obj: %{object_name}. ERROR - %{metric}"
"object_category" => "Check_syslog"
"metric_category" => "predicate" }
remove_field => ["tags", "message" ]
}
if ([object_name] == "" or [object_name] == "obj")
{mutate{
rename => {"object_name" => "to_delete_obj" }}
}
if
[action] == "Log In" and
([user] =~ ".user1" or
[user] =~ ".user2" or
[user] =~ ".user3" or
)
{
mutate {
update => {
"metric" => "CheckUserLogOn"
"object_name" => "%{user}"
"value" => 1
"service" => "CheckUser"
"description" => "There was the login attempt for user - %{user} at - %{date}. Please, provide information why VPN-connection was used."
"object_category" => "Check_user"
"metric_category" => "predicate" }
add_field =>
"object_name" => "%{user}"
"metric" => "CheckUserLogOn"
}
}
}
else
{drop{}}
}
}
output
{
if [type] == "syslog_Check"
{
# { stdout {codec => rubydebug}}
elasticsearch
{
hosts => "localhost:9200"
index => "syslog_check-%{+YYYY.MM.dd}"
}
}
}
Hey, Christian it seems like that I found why it doesn't work, but when I get my expresion in brackets
like this
if (expresion )
{
if (expresion )
}
else
{drop {}}
it's look like it's work...
No, it's still losses some logs. Maybe my filter is too big for logstash?
Are you sending your data over TCP or UDP? If you are using UDP, have you verified that all data is actually reaching Logstash?
It looks like this section could potentially be VERY slow as it might require a lot of regular expression parsing. Have you got X-Pack monitoring installed so you can see how much processing time is spent here?
All data received over UDP. Yes, we verified that all data receied on target host.
No, I don't have X-Pack.
But maybe I can see some message in logs of logstash that it dropped some events?
Instead of dropping the messages in the config, why not write them to a separate file so you can verify it is dropping whet you expect it to?
hmmm....I never do that...
You mean that I need to write my logs to file, and then parse it? Or make separeted config files?
How I can implement this?
I was suggesting it as a way to debug this, not necessarily a permanent feature.
Instead of the drop filter, instead add e.g. a tag. The only send events without this tag to elasticsearch and write all events with a tag to e.g. a file output so you can see exactly what is being dropped.
Hi, Christian!
I removed drop filter, but for my sorry logstash still loses messages.
But when I replased input plugin for this :
udp
{
type => 'syslog'
port => 514
queue_size => 72000
receive_buffer_bytes => 31457280
}
It works fine! Logstash don't lose messages anymore !
my input lugin was :
input
{
syslog
{
port => 514
type => "syslog"
}
}
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.