# Problem dropping multiple log lines with match statement

**URL:** <https://discuss.elastic.co/t/problem-dropping-multiple-log-lines-with-match-statement/197144>\
**Category:** Logstash\
**Created:** [August 28, 2019, 3:01pm UTC](https://discuss.elastic.co/t/problem-dropping-multiple-log-lines-with-match-statement/197144 "2019-08-28T15:01:40Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Neje](https://avatars.discourse-cdn.com/v4/letter/n/8e7dd6/32.png) [@Neje](https://discuss.elastic.co/u/Neje)\
**Post date:** [August 28, 2019, 3:01pm UTC](https://discuss.elastic.co/t/problem-dropping-multiple-log-lines-with-match-statement/197144/1 "2019-08-28T15:01:41Z")

</div>

I'm using LogStash to process AWS Cloudfront logs and I've got a bit of a problem with the log headers.

At the start of each log file are two header lines before the main log-data itself. Both start with # and take the format:

#Version: 1.0  
#Fields:

I added a match before my grok filter and a check for a parser error after but it only drops one of the two lines, not both. This results in having a single entry per log file containing one of the header value as "message".

```
# drop anything starting with #
if [message] =~ /^#/ {
    drop{}
}

grok {
  match => {
    "message" => "%{DATE_EU:date}[\t]%{TIME:time}[\t](?<edge_location>\b[\w\-]+\b)[\t](?:%{INT:resp_bytes}|-)[\t]%{IPORHOST:client_ip}[\t]%{WORD:req_method}[\t]%{HOSTNAME:cf_host}[\t]%{URIPATH:req_path}[\t]%{INT:resp_status}[\t](?:%{URI:referrer}|-)[\t]%{NOTSPACE:User_Agent}[\t]%{NOTSPACE:req_query}[\t]%{NOTSPACE:req_cookies}[\t]%{WORD:edge_resp_type}[\t]%{NOTSPACE:req_id}[\t]%{HOSTNAME:req_hostname}[\t]%{URIPROTO:req_protocol}[\t]%{INT:req_bytes}[\t]%{NUMBER:time_taken:float}[\t]%{NOTSPACE:x_forwarded_for}[\t]%{NOTSPACE:tls_ver}[\t]%{NOTSPACE:tls_cipher}[\t]%{WORD:edge_response_result_type}[\t]%{NOTSPACE:req_protocol_ver}[\t]%{NOTSPACE:fle_status}[\t]%{NOTSPACE:fle_encrypted_fields}"
  }
}

# drop lines we can't parse
if "_grokparsefailure" in [tags] {
  drop{}
}

```

I've tried putting the filter after the grok match and the tag match but it still only filters out one of the headers.

Can anyone shed any light?

---

<div class="post-metadata">

**Author:** ![naveenrt23](https://avatars.discourse-cdn.com/v4/letter/n/47e85d/32.png) [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\
**Post date:** [August 28, 2019, 3:29pm UTC](https://discuss.elastic.co/t/problem-dropping-multiple-log-lines-with-match-statement/197144/2 "2019-08-28T15:29:25Z")

</div>

Can you provide a test message ?

---

<div class="post-metadata">

**Author:** ![naveenrt23](https://avatars.discourse-cdn.com/v4/letter/n/47e85d/32.png) [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\
**Post date:** [August 28, 2019, 3:39pm UTC](https://discuss.elastic.co/t/problem-dropping-multiple-log-lines-with-match-statement/197144/3 "2019-08-28T15:39:58Z")

</div>

> [@Neje](#):
>
> if "\_grokparsefailure" in [tags] { drop{} }

I've tested this with my filters and it drops both version and fields

---

<div class="post-metadata">

**Author:** ![Neje](https://avatars.discourse-cdn.com/v4/letter/n/8e7dd6/32.png) [@Neje](https://discuss.elastic.co/u/Neje)\
**Post date:** [August 30, 2019, 8:18am UTC](https://discuss.elastic.co/t/problem-dropping-multiple-log-lines-with-match-statement/197144/4 "2019-08-30T08:18:22Z")

</div>

Not much to share really, start of the log ingest looks like this:

```
#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields
<grokable log lines follow>

```

What I'm finding in my index's is the following:

```
{
  "_index": "cloudfront-prod-logs-2019.08.29",
  "_type": "doc",
  "_id": "%{[req_id]}",
  "_version": 476,
  "_score": null,
  "_source": {
    "message": "#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields\n",
    "@timestamp": "2019-08-29T12:54:08.189Z",
    "@version": "1"
  },
  "fields": {
    "@timestamp": [
      "2019-08-29T12:54:08.189Z"
    ]
  },
  "sort": [
    1567083248189
  ]
}

```

So it's catching the first line with a # but not the second and inserting it into my index. 😕

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 27, 2019, 8:18am UTC](https://discuss.elastic.co/t/problem-dropping-multiple-log-lines-with-match-statement/197144/5 "2019-09-27T08:18:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.