# Filebeat with new line

**URL:** <https://discuss.elastic.co/t/filebeat-with-new-line/130523>\
**Category:** Beats\
**Created:** [May 3, 2018, 8:29pm UTC](https://discuss.elastic.co/t/filebeat-with-new-line/130523 "2018-05-03T20:29:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maddy\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddy_m/32/45638_2.png) [@Maddy\_M](https://discuss.elastic.co/u/Maddy_M)\
**Post date:** [May 3, 2018, 8:29pm UTC](https://discuss.elastic.co/t/filebeat-with-new-line/130523/1 "2018-05-03T20:29:06Z")

</div>

Hi

I have following lines, and I am using filebeat to publish this data to logstash and then trying to extract values. Could you please advise?

2017-09-03 16:01:28,574 85732M0=\>{"DATALIN": {  
"EVA:": "P",  
"INTRO": "",  
"BACKUP": "",  
"NEUTRAL": "",  
"ID": "005706",  
"FON": 0,  
}}  
2017-09-03 16:01:28,574 85732M0=\>{"DATALIN": {  
"EVA:": "P",  
"INTRO": "",  
"BACKUP": "",  
"NEUTRAL": "",  
"ID": "005706",  
"FON": 0,  
}}

---

<div class="post-metadata">

**Author:** ![atira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atira/32/28699_2.png) [@atira](https://discuss.elastic.co/u/atira)\
**Post date:** [May 3, 2018, 9:31pm UTC](https://discuss.elastic.co/t/filebeat-with-new-line/130523/2 "2018-05-03T21:31:59Z")

</div>

You mean you would like to combine multiple lines into one message?  
That's what the [multiline settings](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html) are for.

You could use this for example:

```auto
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after

```

---

<div class="post-metadata">

**Author:** ![Maddy\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddy_m/32/45638_2.png) [@Maddy\_M](https://discuss.elastic.co/u/Maddy_M)\
**Post date:** [May 4, 2018, 12:42pm UTC](https://discuss.elastic.co/t/filebeat-with-new-line/130523/3 "2018-05-04T12:42:11Z")

</div>

Hi Atira,

I used multiline earlier in filbeat and it generated this message with lot of backslashs and \n characters. Is there a way to eliminate this and generate following format?

"message": "2017-09-03 16:01:28,574 85732M0=\u003e{"DATALIN": {\n "EVA": "P",\n "INTRO": "",\n "BACKUP": "",\n "NEUTRAL": "",\n "ID": 005706,\n "FON": ""\n}}"

target format:

Time: 2017-09-03 16:01:28,574  
username: 85732M0  
EVA: "P"  
INTRO:  
BACKUP  
NEUTRAL  
ID: 005706  
FON:

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![atira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atira/32/28699_2.png) [@atira](https://discuss.elastic.co/u/atira)\
**Post date:** [May 4, 2018, 4:41pm UTC](https://discuss.elastic.co/t/filebeat-with-new-line/130523/4 "2018-05-04T16:41:01Z")

</div>

I've looked up on it a little.  
Taking a better look, your events consist of date + user + json document.  
There is a codec plugin that handles json inputs with newlines, the [json\_lines codec plugin](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json_lines.html).

However, your events are only partly json documents, so the codec would fail. This makes the solution a little bit, well, uglier. Maybe someone knows better (I'm a beginner at the ELK stack myself).

But this should work.

So, let's take this input:

> [@Maddy\_M](#):
>
> "message": "2017-09-03 16:01:28,574 85732M0=\u003e{"DATALIN": {\n "EVA": "P",\n "INTRO": "",\n "BACKUP": "",\n "NEUTRAL": "",\n "ID": 005706,\n "FON": ""\n}}"

This, while looks bad for the human eye, is quite standard format, so using the dissect filter should work, which is good because the dissect filter has a much lower performance need.  
I added the date filter too, you'll probably need it anyway. I presume the dissect filter will eliminate the space between the date and the time.

```auto
filter {
    dissect {
        mapping => {
            "message" => "%{Time} %{+Time} %{username}=\u003e{\"DATALIN\": {\n \"EVA\": \"%{EVA}\",\n \"INTRO\": \"%{INTRO}\",\n \"BACKUP\": \"%{BACKUP}\",\n \"NEUTRAL\": \"%{NEUTRAL}\",\n \"ID\": %{ID},\n \"FON\": \"%{FON}\"\n}}"
            }
        }
    }
    date {
        match => ["Time", "yyyy-MM-ddHH:mm:ss,SSS"]
    }
}

```

I hope it'll work. I can't test it myself, so please come back with the result 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 1, 2018, 6:41pm UTC](https://discuss.elastic.co/t/filebeat-with-new-line/130523/5 "2018-06-01T18:41:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.