# Grok Multiline file

**URL:** <https://discuss.elastic.co/t/grok-multiline-file/248302>\
**Category:** Logstash\
**Created:** [September 11, 2020, 10:27am UTC](https://discuss.elastic.co/t/grok-multiline-file/248302 "2020-09-11T10:27:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![LuigiDelavega](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LuigiDelavega](https://discuss.elastic.co/u/LuigiDelavega)\
**Post date:** [September 11, 2020, 10:27am UTC](https://discuss.elastic.co/t/grok-multiline-file/248302/1 "2020-09-11T10:27:17Z")

</div>

Hi there,

I'm looking to grok mails which were not sent. So i would like the entire file to be read as one log with 5 fields : From, To, Date, Subject and the rest as Message.

Here is what it looks like:

```auto
From: TEXT
To: TEXT
Date: Thu, 30 Jul 2020 19:00:25 +0200
TEXT
TEXT
	TEXT
TEXT
TEXT
Subject: TEXT
TEXT
TEXT
	TEXT
etc...

```

How can i write my grok to get those FROM field, then TO field then DATE field, then between DATE and SUBJECT, we put everything in MESSAGE, then SUBJECT then again in MESSAGE

I'm using a pipeline which has 10 filters depending on a field i add in filebeat.yml. So my first question is : can i use codec multiline with a condition in the input of my pipeline ?

Right now it looks like

```auto
  beats {
    port => 5047
	client_inactivity_timeout => 1200
  }
}

```

and i would like to do this

```auto
  beats {
    port => 5047
	client_inactivity_timeout => 1200
  }
  if [fields][log_type] == "bad-mail" {   
    codec => multiline {
          pattern => "From:%{DATA:from} To:%{DATA:to} Date:%{DATA:date} %{GREEDYDATA:message} Subject:%{Subject:from} %{GREEDYDATA:message}"
          negate => "false"
      }
    }
}

```

To use multiline only with this log\_type, i don't know if i can put \n or else.

I've looked to a lot of example or others post on this website and others, but i didn't find the solution yet 😕 , i m still testing all the filters either in filebeat or in logstash.

Thanks already for your time and your help, have a great day,

Louis Vince.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 11, 2020, 1:23pm UTC](https://discuss.elastic.co/t/grok-multiline-file/248302/2 "2020-09-11T13:23:11Z")

</div>

> [@LuigiDelavega](#):
>
> can i use codec multiline with a condition in the input of my pipeline ?

No, you cannot. The codec is used to generate events, so it cannot be conditional on events which only exist after the codec has been applied.

---

<div class="post-metadata">

**Author:** ![LuigiDelavega](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LuigiDelavega](https://discuss.elastic.co/u/LuigiDelavega)\
**Post date:** [September 11, 2020, 1:55pm UTC](https://discuss.elastic.co/t/grok-multiline-file/248302/3 "2020-09-11T13:55:34Z")

</div>

Ok thanks for your answer @Badger, so i have to put the multiline.pattern and others settings into my filebeat.yml.  
But i can't find any patterns matching what i need. I can get the line where there are the terms "From: To: Subject: and Date:" but i don't know how to write "take the rest of the document into a Message field". The number of line and the content will be different for every mail.

Moreover, the mail can be a response to another mail.. so it can be fullfilled with:

```auto
From:
To:
Date:
TExt ....
Subject:
text
text 
...
... 
         from:
         to:
         etc....

```

Those fields can even be at the same level of indentation... but i only want to get "From: To: Subject: and Date:" fields of the top of the mail. I can understand and i don't want u to do the regex filter, but i only want to know if it's possible 😃

Thanks again for your time, best regards,

Louis Vince

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 11, 2020, 2:26pm UTC](https://discuss.elastic.co/t/grok-multiline-file/248302/4 "2020-09-11T14:26:49Z")

</div>

If you are saying that in the text

```
From:
To:
Date:
TExt ....
Subject:
text
text 
From:
To:
Date:
TExt ....
Subject:
text
text 

```

that the second From is sometimes a new mail message and sometimes a quoted message then no, I see no way for logstash or filebeat to determine which it is.

---

<div class="post-metadata">

**Author:** ![LuigiDelavega](https://avatars.discourse-cdn.com/v4/letter/l/9de0a6/32.png) [@LuigiDelavega](https://discuss.elastic.co/u/LuigiDelavega)\
**Post date:** [September 11, 2020, 2:57pm UTC](https://discuss.elastic.co/t/grok-multiline-file/248302/5 "2020-09-11T14:57:28Z")

</div>

No no, one file is always one log, sometimes in the mail, there is a quoted mail, but i dont want to get this quoted mail as a log.

i've changed my grok debugger and it seems to work, i will test it next monday.

But my Message field isnt a multiline field so it"s not very clean to read when there are 40 lines with line 1 \n line2 \n line 3...

I would like the \n to be applied when i display the data in a table or in the discover interface in Kibana.

```auto
From: %{GREEDYDATA:From}\nTo: %{GREEDYDATA:To}\nDate: %{GREEDYDATA:Date}\n(?m)%{GREEDYDATA:Message}\nMessage-ID: %{DATA:MessageID}\nSubject: %{DATA:Subject}\n(?m)%{GREEDYDATA:Message}\n

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 19, 2020, 8:35am UTC](https://discuss.elastic.co/t/grok-multiline-file/248302/7 "2020-10-19T08:35:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.