# Logstash losing messages **URL:** \ **Category:** Logstash\ **Created:** [July 12, 2018, 10:35am UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730 "2018-07-12T10:35:01Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![M\_B\_I](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@M\_B\_I](https://discuss.elastic.co/u/M_B_I)\ **Post date:** [July 12, 2018, 10:35am UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/1 "2018-07-12T10:35:01Z")
Hello, everyone! I have a lot of syslog messages. And when I trying to filter them I found that logstash losing some messages. How can I investigate why? And how can I found howq many of messages are missed ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 12, 2018, 10:50am UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/2 "2018-07-12T10:50:52Z")
What does you config look like? Which inputs are you using? ---
**Author:** ![M\_B\_I](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@M\_B\_I](https://discuss.elastic.co/u/M_B_I)\ **Post date:** [July 12, 2018, 11:43am UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/3 "2018-07-12T11:43:37Z")
input { syslog { port =\> 514 type =\> "syslog\_Check" } } filter { if [type] == "syslog\_Check" { mutate { gsub =\> ["message", "._\<134\>1 ", "date:=", "message", "= ", " ", "message", ":"", "=", "message", ":=", "=", "message", "";", ";", "message", "; ", ";", "message", " ._[", ";"] } kv {field\_split =\> ";"} if [message] =~ "._could not get SAs from packet._" or [message] =~ "._no proposal chosen._" or [message] =~ "._No valid SA._" or [message] =~ "._Packet is dropped because there is no valid SA._" or [message] =~ "._Phase one received notification from peer; payload malformed._" or [message] =~ "._Quick Mode failed to match proposal._" or [message] =~ "._Quick Mode Received Notification from Peer: invalid id information._" or [message] =~ "._Reject Reason: Gateway to Gateway authentication failure._" or [message] =~ "._Unknown SPI for IPsec packet._" or [message] =~ "._Virtual defragmentation error: Timeout._" or [message] =~ "._encryption failure: Failed to enforce VPN Policy (11)._" or [message] =~ "._encryption failure: no response from peer._" or [message] =~ "._encryption failure: Tunnel failure, unresolved SA (VPN Error code 01)._" or [message] =~ "._encryption failure: Unknown SPI._" or [message] =~ "._encryption failure: Wrong peer gateway for decrypted._" or [message] =~ "._failed to connect: Internal error- (CCC\_E\_GENERAL)._" or [message] =~ "._IKE: Aggressive Mode Failed to match proposal:._" or [message] =~ "._IKE: Main Mode cannot complete certificate chain._" or [message] =~ "._IKE: Main Mode Sent Notification to Peer: invalid certificate._" or [message] =~ "._IKE: Main Mode Sent Notification to Peer: invalid information._" or [message] =~ "._Invalid ID information._" or [message] =~ "._Invalid Peer Certificate._" or [message] =~ "._no common community for myself and peer._" or [message] =~ "._no proposal chosen._" or [message] =~ "._No valid SA._" or [message] =~ "._authentication failed._" or ([user] and [action] == "Log In") { mutate { rename =\> { "ike" =\> "metric" "community" =\> "object\_name" "service" =\> "serv" } add\_field =\> { "value" =\> 1 "service" =\> "CheckP" "description" =\> "Syslog time = %{date}. Detected an ERROR. Obj: %{object\_name}. ERROR - %{metric}" "object\_category" =\> "Check\_syslog" "metric\_category" =\> "predicate" } remove\_field =\> ["tags", "message"] } if ([object\_name] == "" or [object\_name] == "obj") {mutate{ rename =\> {"object\_name" =\> "to\_delete\_obj" }} } if [action] == "Log In" and ([user] =~ "._user1_" or [user] =~ "._user2_" or [user] =~ "._user3_" or ) { mutate { update =\> { "metric" =\> "CheckUserLogOn" "object\_name" =\> "%{user}" "value" =\> 1 "service" =\> "CheckUser" "description" =\> "There was the login attempt for user - %{user} at - %{date}. Please, provide information why VPN-connection was used." "object\_category" =\> "Check\_user" "metric\_category" =\> "predicate" } add\_field =\> "object\_name" =\> "%{user}" "metric" =\> "CheckUserLogOn" } } } ``` else {drop{}} } ``` } output { if [type] == "syslog\_Check" { # { stdout {codec =\> rubydebug}} elasticsearch { hosts =\> "localhost:9200" index =\> "syslog\_check-%{+YYYY.MM.dd}" } } } ---
**Author:** ![M\_B\_I](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@M\_B\_I](https://discuss.elastic.co/u/M_B_I)\ **Post date:** [July 12, 2018, 11:56am UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/4 "2018-07-12T11:56:12Z")
Hey, Christian it seems like that I found why it doesn't work, but when I get my expresion in brackets like this if (expresion ) { if (expresion ) } else {drop {}} it's look like it's work... ---
**Author:** ![M\_B\_I](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@M\_B\_I](https://discuss.elastic.co/u/M_B_I)\ **Post date:** [July 12, 2018, 1:33pm UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/5 "2018-07-12T13:33:23Z")
No, it's still losses some logs. Maybe my filter is too big for logstash? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 12, 2018, 1:35pm UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/6 "2018-07-12T13:35:35Z")
Are you sending your data over TCP or UDP? If you are using UDP, have you verified that all data is actually reaching Logstash? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 12, 2018, 1:38pm UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/7 "2018-07-12T13:38:31Z")
> [@M\_B\_I](#): > > if > [message] =~ ". _could not get SAs from packet._ " or > [message] =~ ". _no proposal chosen._ " or > [message] =~ ". _No valid SA._ " or > [message] =~ ". _Packet is dropped because there is no valid SA._ " or > [message] =~ ". _Phase one received notification from peer; payload malformed._ " or > [message] =~ ". _Quick Mode failed to match proposal._ " or > [message] =~ ". _Quick Mode Received Notification from Peer: invalid id information._ " or > [message] =~ ". _Reject Reason: Gateway to Gateway authentication failure._ " or > [message] =~ ". _Unknown SPI for IPsec packet._ " or > [message] =~ ". _Virtual defragmentation error: Timeout._ " or > [message] =~ ". _encryption failure: Failed to enforce VPN Policy (11)._ " or > [message] =~ ". _encryption failure: no response from peer._ " or > [message] =~ ". _encryption failure: Tunnel failure, unresolved SA (VPN Error code 01)._ " or > [message] =~ ". _encryption failure: Unknown SPI._ " or > [message] =~ ". _encryption failure: Wrong peer gateway for decrypted._ " or > [message] =~ ". _failed to connect: Internal error- (CCC\_E\_GENERAL)._ " or > [message] =~ ". _IKE: Aggressive Mode Failed to match proposal:._ " or > [message] =~ ". _IKE: Main Mode cannot complete certificate chain._ " or > [message] =~ ". _IKE: Main Mode Sent Notification to Peer: invalid certificate._ " or > [message] =~ ". _IKE: Main Mode Sent Notification to Peer: invalid information._ " or > [message] =~ ". _Invalid ID information._ " or > [message] =~ ". _Invalid Peer Certificate._ " or > [message] =~ ". _no common community for myself and peer._ " or > [message] =~ ". _no proposal chosen._ " or > [message] =~ ". _No valid SA._ " or > [message] =~ ". _authentication failed._ " or > ([user] and [action] == "Log In") > { It looks like this section could potentially be VERY slow as it might require a lot of regular expression parsing. Have you got X-Pack monitoring installed so you can see how much processing time is spent here? ---
**Author:** ![M\_B\_I](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@M\_B\_I](https://discuss.elastic.co/u/M_B_I)\ **Post date:** [July 12, 2018, 1:55pm UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/8 "2018-07-12T13:55:49Z")
All data received over UDP. Yes, we verified that all data receied on target host. No, I don't have X-Pack. But maybe I can see some message in logs of logstash that it dropped some events? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 12, 2018, 2:05pm UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/9 "2018-07-12T14:05:04Z")
Instead of dropping the messages in the config, why not write them to a separate file so you can verify it is dropping whet you expect it to? ---
**Author:** ![M\_B\_I](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@M\_B\_I](https://discuss.elastic.co/u/M_B_I)\ **Post date:** [July 12, 2018, 2:16pm UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/10 "2018-07-12T14:16:52Z")
hmmm....I never do that... You mean that I need to write my logs to file, and then parse it? Or make separeted config files? How I can implement this? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 12, 2018, 2:26pm UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/11 "2018-07-12T14:26:04Z")
I was suggesting it as a way to debug this, not necessarily a permanent feature. Instead of the `drop` filter, instead add e.g. a tag. The only send events without this tag to elasticsearch and write all events with a tag to e.g. a file output so you can see exactly what is being dropped. ---
**Author:** ![M\_B\_I](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@M\_B\_I](https://discuss.elastic.co/u/M_B_I)\ **Post date:** [August 7, 2018, 8:54am UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/12 "2018-08-07T08:54:27Z")
Hi, Christian! I removed drop filter, but for my sorry logstash still loses messages. But when I replased input plugin for this : udp { type =\> 'syslog' port =\> 514 queue\_size =\> 72000 receive\_buffer\_bytes =\> 31457280 } It works fine! Logstash don't lose messages anymore ! my input lugin was : input { syslog { port =\> 514 type =\> "syslog" } } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 4, 2018, 8:54am UTC](https://discuss.elastic.co/t/logstash-losing-messages/139730/13 "2018-09-04T08:54:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.