# How to drop a message

**URL:** https://discuss.elastic.co/t/how-to-drop-a-message/69548
**Category:** Logstash
**Created:** [December 20, 2016, 10:01am UTC](https://discuss.elastic.co/t/how-to-drop-a-message/69548 "2016-12-20T10:01:28Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![houfan](https://avatars.discourse-cdn.com/v4/letter/h/47e85d/32.png) [@houfan](https://discuss.elastic.co/u/houfan)
#### Post date: [December 20, 2016, 10:01am UTC](https://discuss.elastic.co/t/how-to-drop-a-message/69548/1 "2016-12-20T10:01:28Z")

</div>

the logstash give me three message:

> "message" =\> "# administrator command: Prepare;"

> "message" =\> "# User@Host: gcenter[gcenter] @ [192.168.168.100] Id: 4882345700\n# Schema: gcenter Last\_errno: 0 Killed: 0\n# Query\_time: 5.883876 Lock\_time: 0.000000 Rows\_sent: 0 Rows\_examined: 0 Rows\_affected: 0\n# Bytes\_sent: 435\nuse gcenter;\nSET timestamp=1481691552;\n# administrator command: Prepare;",

> "message" =\> "# User@Host: acenter[acenter] @ [192.168.168.106] Id: 4882345703\n# Schema: appcenter Last\_errno: 0 Killed: 0\n# Query\_time: 5.879482 Lock\_time: 0.000000 Rows\_sent: 0 Rows\_examined: 0 Rows\_affected: 0\n# Bytes\_sent: 1421\nuse appcenter;\nSET timestamp=1481691552;",

how to configure the logstash.conf to drop the first message???

> if [message] =~ "^# Time:" {  
> drop {}  
> }  
> this above didn't work as expect. Is there anyway to fix this problem?  
> really appreciate your help 🙂

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 20, 2016, 10:10am UTC](https://discuss.elastic.co/t/how-to-drop-a-message/69548/2 "2016-12-20T10:10:39Z")

</div>

The correct syntax is

```
if [message] =~ /^# Time:/ {

```

but otherwise it should work. Of course, none of the messages in your example begin with "# Time:".

---

<div class="post-metadata">

### Author: ![houfan](https://avatars.discourse-cdn.com/v4/letter/h/47e85d/32.png) [@houfan](https://discuss.elastic.co/u/houfan)
#### Post date: [December 21, 2016, 2:21am UTC](https://discuss.elastic.co/t/how-to-drop-a-message/69548/3 "2016-12-21T02:21:07Z")

</div>

I modified the logstash.conf like this:

> if [message] =~ /^# Time:/ {  
> drop {}  
> }

and this is mysql-slow.log:

> ```
> # Time: 161214 4:07:56
> # User@Host: root[root] @ [192.168.168.10] Id: 4866759372
> # Schema: dcenter Last_errno: 0 Killed: 0
> # Query_time: 128.778717 Lock_time: 0.000000 Rows_sent: 44812656 Rows_examined: 44812656 Rows_affected: 0
> # Bytes_sent: 5333525169
> SET timestamp=1481659676;
> SELECT /*!40001 SQL_NO_CACHE */ * FROM `game_action_logs`;
> # Time: 161214 4:08:07
> # User@Host: root[root] @ [192.168.168.10] Id: 4866759372
> # Schema: dcenter Last_errno: 0 Killed: 0
> # Query_time: 11.018205 Lock_time: 0.000000 Rows_sent: 4164122 Rows_examined: 4164122 Rows_affected: 0
> # Bytes_sent: 416694449
> SET timestamp=1481659687;
> SELECT /*!40001 SQL_NO_CACHE */ * FROM `games_summary_d`;
> 
> ```

logstash give me one result:

`> "message" => "# User@Host: root[root] @ [192.168.168.10] Id: 4866759372\n# Schema: dcenter Last_errno: 0 Killed: 0\n# Query_time: 11.018205 Lock_time: 0.000000 Rows_sent: 4164122 Rows_examined: 4164122 Rows_affected: 0\n# Bytes_sent: 416694449\nSET timestamp=1481659687;\nSELECT /*!40001 SQL_NO_CACHE */ * FROM `games\_summary\_d`;"`

cos I add this part at the end of grok pattern:**(?:# Time:.\*\n)?**  
this is the whole grok pattern:

`> ^# User@Host: %{USER:User}\[[^\]]+\] @ (?:(?<clienthost>\S*) )?\[(?:%{IP:Client_IP})?\]\s+Id:\s+%{NUMBER:row_id:int}\n# Schema: %{WORD:Schema}\s+Last_errno: %{NUMBER}\s+Killed: %{NUMBER}\n# Query_time: %{NUMBER:Query_Time:float}\s+Lock_time: %{NUMBER:Lock_Time:float}\s+Rows_sent: %{NUMBER:Rows_Sent:int}\s+Rows_examined: %{NUMBER:Rows_Examined:int}\s+Rows_affected: %{NUMBER:Rows_affected:int}\n(?:# Bytes_sent: %{NUMBER:Byte_sent:int}\n)+(?:use %{DATA:database};\s*\n)?SET\s+timestamp=%{NUMBER:timestamp};(?:\n)?(?<sql>(?<action>\w+)\b.*;)?(?:\n)?(?:# Time:.*\n)?`

after I have tried many times, I found out that it would drop not only the first line start with "# Time",and also the whole message that the grok pattern matched next!!!

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 21, 2016, 6:35am UTC](https://discuss.elastic.co/t/how-to-drop-a-message/69548/4 "2016-12-21T06:35:25Z")

</div>

Well, yes. It seems you're using a multiline codec to join the physical lines of a logical event. If you do that you can't use the drop filter.

---

<div class="post-metadata">

### Author: ![houfan](https://avatars.discourse-cdn.com/v4/letter/h/47e85d/32.png) [@houfan](https://discuss.elastic.co/u/houfan)
#### Post date: [December 21, 2016, 6:47am UTC](https://discuss.elastic.co/t/how-to-drop-a-message/69548/5 "2016-12-21T06:47:18Z")

</div>

eeeeee.......really thans for your reply  
cos this is mysql-slow.log.so I have to use the multiline codec .  
now ,how can I drop this line: 😭 ....."# Time: 161214 1:31:33"

I tried this:

> grok {  
> match =\> { "message" =\> "^# Time:\s+\d{6}\s+\d{1,2}:\d{2}:\d{2}\n" }  
> add\_tag =\> ["time"]  
> tag\_on\_failure =\>   
> }  
> if "time" in [tags] {  
> drop {}  
> }

but it didn't work. and I tried another way like this:

> if "\_grokparsefailure" in [tags] {  
> drop {}  
> }

this did work,but I think it is not a proper way to drop this message.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 21, 2016, 6:48am UTC](https://discuss.elastic.co/t/how-to-drop-a-message/69548/6 "2016-12-21T06:48:34Z")

</div>

Why do you need to drop that line? Just ignore the timestamp in your grok filter if you don't care about it.

---

<div class="post-metadata">

### Author: ![houfan](https://avatars.discourse-cdn.com/v4/letter/h/47e85d/32.png) [@houfan](https://discuss.elastic.co/u/houfan)
#### Post date: [December 21, 2016, 7:05am UTC](https://discuss.elastic.co/t/how-to-drop-a-message/69548/7 "2016-12-21T07:05:40Z")

</div>

cos mysql-slow.log has two different heads. one is :

> ```
> # Time: 161214 10:21:28
> # User@Host: alltechremotecon[alltechremotecon] @ [218.17.162.125] Id: 4876018124
> # Schema: gcenter3x Last_errno: 0 Killed: 0
> # Query_time: 194.159645 Lock_time: 0.000040 Rows_sent: 50810 Rows_examined: 50810 Rows_affected: 0
> # Bytes_sent: 8707551
> SET timestamp=1481682088;
> SELECT * FROM `december_data_upload`;
> 
> ```

another one is this:

> ```
> # administrator command: Prepare;
> # User@Host: gcenter[gcenter] @ [192.168.168.100] Id: 4882345700
> # Schema: gcenter Last_errno: 0 Killed: 0
> # Query_time: 5.883876 Lock_time: 0.000000 Rows_sent: 0 Rows_examined: 0 Rows_affected: 0
> # Bytes_sent: 435
> use gcenter;
> SET timestamp=1481691552;
> 
> ```

and I think I can only set the pattern in multiline codec like this:

> pattern: ^# User@Host

I add this part at the end of my grok pattern:

> (?:# Time:.\*\n)?(?:#\s+%{WORD}\s+%{WORD}:\s+\w+;\n)?

it will ignore these two messages "# Time: 161214 10:21:28 " "# administrator command: Prepare;" only if they are not at the first line.  
now I am considering the situation when the log rotate,and one of these two messages comes to the first line.....how to solve this 😭  
and really thank you

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 21, 2016, 7:29am UTC](https://discuss.elastic.co/t/how-to-drop-a-message/69548/8 "2016-12-21T07:29:07Z")

</div>

I still don't understand. If you look for the User@Host line in the multiline pattern and don't care about the Time value, why does it matter where the Time value ends up? Just adjust your grok expression to handle all cases. You can also use a mutate filter's gsub option to remove the Time value regardless of where in the string it occurs.

---

<div class="post-metadata">

### Author: ![houfan](https://avatars.discourse-cdn.com/v4/letter/h/47e85d/32.png) [@houfan](https://discuss.elastic.co/u/houfan)
#### Post date: [December 21, 2016, 8:31am UTC](https://discuss.elastic.co/t/how-to-drop-a-message/69548/9 "2016-12-21T08:31:34Z")

</div>

...........I am just a rookie here. I still don't know how to modify the logstash.conf . Could you give me a template or example...😭  
really really thanks 😀

I bought a book about ELK...wrote by a chinese author.and it didn't help me a lot. so far ,I could only get help from the internet ,especially from this forum,from you...it's a sad story 😭

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 18, 2017, 8:31am UTC](https://discuss.elastic.co/t/how-to-drop-a-message/69548/10 "2017-01-18T08:31:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
