# Multiline read in filebeat
**URL:** https://discuss.elastic.co/t/multiline-read-in-filebeat/66365
**Category:** Beats
**Tags:** filebeat
**Created:** [November 17, 2016, 10:46am UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365 "2016-11-17T10:46:36Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@Aditya\_Srivastava](https://discuss.elastic.co/u/Aditya_Srivastava)
#### Post date: [November 17, 2016, 10:46am UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/1 "2016-11-17T10:46:36Z")
I have a custom log file which has multiple lines getting logged to it. All those multiple lines do not have any similar pattern to it. Those multilines are random. Which or rather what kind of parser or anything should I write in filebeat conf file. How do I read those multi lines??
Log file ex.
```auto
Nov 17 16:25:30 1.2.3.4 appData [app:16:25:28,115] INFO [application level detail. Got response
Nov 17 16:25:30 1.2.3.4 appData [app:16:25:28,115] DEBUG [application-level detail. Response from app is :: {}029B
1.0
ABC
DEF
161117162528
2016-11-17T16:25:32.313+05:30
0810
20161117
Nov 17 16:25:30 1.2.3.4 app [App:16:25:28,116] INFO [application level data]
```
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [November 17, 2016, 1:23pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/2 "2016-11-17T13:23:10Z")
I think you could you the timestamp to identify where new log lines start. See the [Timestamp](https://www.elastic.co/guide/en/beats/filebeat/5.0/multiline-examples.html#_timestamps) example in the documentation.
---
### Author:  [@Aditya\_Srivastava](https://discuss.elastic.co/u/Aditya_Srivastava)
#### Post date: [November 17, 2016, 1:27pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/3 "2016-11-17T13:27:16Z")
> [@Aditya\_Srivastava](#):
>
> \2016-11-17T16:25:32.313+05:30\
Are you talking about this timestamp?? If yes what about the other entries..How do I get all of them as one message???
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [November 17, 2016, 1:30pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/4 "2016-11-17T13:30:50Z")
> [@Aditya\_Srivastava](#):
>
> Nov 17 16:25:30 1.2.3.4 appData [app:16:25:28,115] DEBUG
No, I'm talking about the timestamp from the logger.
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [November 17, 2016, 1:32pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/5 "2016-11-17T13:32:19Z")
Have a lookt at your logs. They are mostly plain-text starting with month, day of month and time. The multiline content is always indented. e.g. check a log-line starting with any of these characters `^[JFMASOND]` (regex captures all characters of "months"). Alternative check for log line being empty or starting with space/tab.
---
### Author:  [@Aditya\_Srivastava](https://discuss.elastic.co/u/Aditya_Srivastava)
#### Post date: [November 17, 2016, 2:14pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/6 "2016-11-17T14:14:11Z")
Thnk u so much for the reply....Now I get the desired output all the multiline as one message... Only Prob now is all my new line is coming as \n and all the spaces are coming as \t.
ex-:
{"@timestamp":"2016-11-17T14:09:33.670Z","beat":{"hostname":"ip-10-0-0-9","name":"ip-10-0-0-9","version":"5.0.0"},"input\_type":"log","message":"Nov 17 19:39:32 RCPPPCFWASN1 Wallet\_App\_\_access [WALLET:19:39:28,126] DEBUG [application-akka.actor.default-dispatcher-35959][SwitchPaymentResponseActor.java:45] Response from switch is :: {}029B\u003c?xml version="1.0" encoding="UTF-8" ?\u003e\n\t\u003cEngine\u003c/Header\u003e\n\t\t\u003cBody\u003e\n\t\t\t\u003cAuthRes\u003e\n\t\t\t\t\u003cMsgType\u003e0810\u003c/MsgType\u003e\n\t\t\t\t\u003cTranDate\u003e20161117\u003cmber\u003e\n\t\t\t\t\u003countDate\u003e20161117\u003\n\t\t\t\u003c/AuthRes\u003e\n\t\t\u003c/Body\u003e\n\t\u003c/Engine\u003e","offset":30024253,"source":"/res/1.2.3.4/App/2016-11-17.log",}
---
### Author:  [@seti321](https://discuss.elastic.co/u/seti321)
#### Post date: [November 17, 2016, 2:26pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/7 "2016-11-17T14:26:45Z")
You could use @sematext/logagent instead of filebeat, logstash etc.
It is lightweight log shipper made with nodejs.
It can parse your file out of the box:
```auto
sudo npm i -g @sematext/logagent
cat test.log | logagent --yaml
```
```auto
@timestamp: Thu Nov 17 2016 15:21:47 GMT+0100 (CET)
message: Nov 17 16:25:30 1.2.3.4 appData [app:16:25:28,115] INFO [application level detail. Got response
logSource: unknown
@timestamp: Thu Nov 17 2016 15:21:47 GMT+0100 (CET)
message:
"""
Nov 17 16:25:30 1.2.3.4 appData [app:16:25:28,115] DEBUG [application-level detail. Response from app is :: {}029B
1.0
ABC
DEF
161117162528
2016-11-17T16:25:32.313+05:30
0810
20161117
"""
logSource: unknown
@timestamp: Thu Nov 17 2016 15:21:48 GMT+0100 (CET)
message: Nov 17 16:25:30 1.2.3.4 app [App:16:25:28,116] INFO [application level data]
logSource: unknown
```
To ship logs to elasticsearch use
```auto
logagent -e http://localhost:9200 -i logs /var/log/*.log
```
More info here: [https://sematext.com/logagent/](https://sematext.com/logagent/)
---
### Author:  [@Aditya\_Srivastava](https://discuss.elastic.co/u/Aditya_Srivastava)
#### Post date: [November 17, 2016, 2:40pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/8 "2016-11-17T14:40:47Z")
thnk u sir..will look into it....We also use logstash for filtering the data and parsing it.. Can that be done using the logagent????
---
### Author:  [@seti321](https://discuss.elastic.co/u/seti321)
#### Post date: [November 17, 2016, 2:46pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/9 "2016-11-17T14:46:14Z")
Yes. You can define custom patters to structure logs, apply JavaScript functions or SQL queries to it before you ship the structured and aggregated logs to Elasticsearch.
Various log formats are supported out the box (nginx, system logs, mongodb, elasticsearch, hadoop, kafka ...).
Example pattern: [http://sematext.github.io/logagent-js/parser/](http://sematext.github.io/logagent-js/parser/)
---
### Author:  [@Aditya\_Srivastava](https://discuss.elastic.co/u/Aditya_Srivastava)
#### Post date: [November 17, 2016, 2:53pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/10 "2016-11-17T14:53:51Z")
how do we use kafka and logagent?? can we take the discussion to a new thread...
---
### Author:  [@seti321](https://discuss.elastic.co/u/seti321)
#### Post date: [November 17, 2016, 3:13pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/11 "2016-11-17T15:13:44Z")
We could create a Kafka output plugin, but you might not need it, because Logagent has a disk buffer (used when connection to Elasticsearch fails), and retransmits logs when Elasticsearch is available again.
You can ask questions in Github [https://github.com/sematext/logagent-js](https://github.com/sematext/logagent-js) or the new forum: [https://groups.google.com/forum/#!forum/logagent](https://groups.google.com/forum/#!forum/logagent)
---
### Author:  [@Aditya\_Srivastava](https://discuss.elastic.co/u/Aditya_Srivastava)
#### Post date: [November 17, 2016, 3:33pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/12 "2016-11-17T15:33:49Z")
Hii....I tried with the solution u gave . The pattern is working as expected when I try the pattern and content on the play golang. But When I try this on my file beat config it shows \n for newlines and \t for space. Please help me out on this.
---
### Author:  [@seti321](https://discuss.elastic.co/u/seti321)
#### Post date: [November 17, 2016, 4:03pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/13 "2016-11-17T16:03:07Z")
> [@Aditya\_Srivastava](#):
>
> The pattern is working as expected when I try the pattern and content on the play golang. But When I try this on my file beat conf
Sorry, can't help with filebeat mixing up white spaces ☹
Logagent pattern definition would look like this:
```auto
patterns:
- # multiline with blockStart regex
sourceName: !!js/regexp /mylogs/
blockStart: !!js/regexp /^\S+\s\d+\s\d\d:\d\d:\d\d/
match:
- type: mylogs
regex: !!js/regexp /^(\S+\s\d+\s\d\d:\d\d:\d\d)\s(\S+)\s(\S+)\s\[.+\]\s(\S+)\s([\S|\s]+)/
fields:
- ts
- ip_address:string
- app:string
- severity:string
- message:string
dateFormat: MMM D HH:mm:ss
```
You could also extract fieds from the XML inside, with a bit more work on the regex.
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [November 17, 2016, 6:24pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/14 "2016-11-17T18:24:54Z")
> [@Aditya\_Srivastava](#):
>
> But When I try this on my file beat config it shows \n for newlines and \t for space.
This is normal. The data is put into a JSON object and since newlines and tab characters are special characters they need to be encoded. You'll also notice that quotes in the message are also escaped.
What problem is this causing you?
---
### Author:  [@Aditya\_Srivastava](https://discuss.elastic.co/u/Aditya_Srivastava)
#### Post date: [November 18, 2016, 3:04pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/15 "2016-11-18T15:04:30Z")
yeah u were right..The end message I get at ES is in proper format except for there is no new line. All message gets appended in same line with space. But still I can work with it. Thank
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [November 18, 2016, 3:42pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/16 "2016-11-18T15:42:21Z")
makes me wonder how you display your message. You display the message via browser, e.g. are you using kibana? Newline characters are included in stored string (the json encoding uses `\n` for newlines). But when displaying in browser via HTML as is (without transforming `\n` to `
` or using `` tag), newlines will be ignored by HTML.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 16, 2016, 3:42pm UTC](https://discuss.elastic.co/t/multiline-read-in-filebeat/66365/17 "2016-12-16T15:42:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.