# Multiline config for filebeat
**URL:** https://discuss.elastic.co/t/multiline-config-for-filebeat/43458
**Category:** Beats
**Created:** [March 4, 2016, 12:10am UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458 "2016-03-04T00:10:29Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@kasi](https://discuss.elastic.co/u/kasi)
#### Post date: [March 4, 2016, 12:10am UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458/1 "2016-03-04T00:10:29Z")
what will be the multiline config for the following stack trace, I have tried everything nothing seems to work.
Any help much appreciated
Stack Trace:
```auto
TID: [0] [BAM] [2015-11-27 23:51:19,549] ERROR {org.wso2.carbon.hadoop.hive.jdbc.storage.db.DBOperation} - Failed to write data to database {org.wso2.carbon.hadoop.hive.jdbc.storage.db.DBOperation}
org.h2.jdbc.JdbcSQLException: NULL not allowed for column "CONSUMERKEY"; SQL statement:
INSERT INTO API_RESPONSE_SUMMARY_DAY (time,resourcepath,context,servicetime,total_response_count,version,tzoffset,consumerkey,epoch,userid,apipublisher,api) VALUES (?,?,?,?,?,?,?,?,?,?,?,?) [90006-140]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.table.Column.validateConvertUpdateSequence(Column.java:294)
at org.h2.table.Table.validateConvertUpdateSequence(Table.java:621)
at org.h2.command.dml.Insert.insertRows(Insert.java:116)
at org.h2.command.dml.Insert.update(Insert.java:82)
at org.h2.command.CommandContainer.update(CommandContainer.java:70)
at org.h2.command.Command.executeUpdate(Command.java:199)
at org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:141)
```
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [March 4, 2016, 6:49am UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458/2 "2016-03-04T06:49:02Z")
Can you share what you have tried so far?
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [March 4, 2016, 12:03pm UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458/3 "2016-03-04T12:03:43Z")
I did try to format your log using 3 backticks before and after your logs. Can you confirm formatted log is correct (especially newlines)? You can check and fix formatting by clicking the edit button (the pencil one).
Just seeing the exception is only half of the story, as one doesn't want multiline to combine normal logs by accident. Maybe you can add some more context here by adding some more log output.
Assuming all log-lines start with 'TID: ' (thread id?) one can try this:
```auto
multiline:
negate: true
pattern: '^TID:'
match: after
```
Which tool is generating the log? With some more context, maybe we can add this as example to our docs.
---
### Author:  [@kasi](https://discuss.elastic.co/u/kasi)
#### Post date: [March 4, 2016, 5:28pm UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458/4 "2016-03-04T17:28:44Z")
Thank you very much for the help steffens, for your question, I have given more info below:
The stack trace is more of the same, the exception is generated by one of our internal apps.
rufflin: I did try the following:
multiline:
pattern: "TID:^"
negate: true
match: after
with various combinations and it did not join the lines together.
steffens I will try your solution and update the post.
Thanks,
Kasi
---
### Author:  [@kasi](https://discuss.elastic.co/u/kasi)
#### Post date: [March 4, 2016, 9:07pm UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458/5 "2016-03-04T21:07:42Z")
steffens:
I am still them as a single entries, please let me know if you need more info.
Thanks,
kasi
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [March 5, 2016, 1:27am UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458/6 "2016-03-05T01:27:56Z")
> [@kasi](#):
>
> I am still them as a single entries, please let me know if you need more info.
I don't understand. Config given by me should be able to combine all lines into one event.
Check out this script applying the regex to your logs: [Go Playground - The Go Programming Language](http://play.golang.org/p/nVtzB6tem-)
You can play with `pattern`, `negate` and `content` variables. Having `match: after`, all lines starting with `false` indicates a new line event and line starting with `true` indicate lines being merged into the multiline event.
What's your filebeat config?
Can you post some more log-lines to get some more context?
Can you give an example of expected input and expected output?
---
### Author:  [@kasi](https://discuss.elastic.co/u/kasi)
#### Post date: [March 7, 2016, 7:09pm UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458/7 "2016-03-07T19:09:49Z")
Hi Steffens,
Thank you very much for the help. I have posted my file beat config below:
```auto
filebeat:
prospectors:
- input_type: log
paths:
- /Users/test/wso2.log
multiline:
negate: true
pattern: '^TID:'
match: after
```
Log lines:
TID: [0] [BAM] [2015-11-27 23:51:19,549] ERROR {org.wso2.carbon.hadoop.hive.jdbc.storage.db.DBOperation} - Failed to write data to database {org.wso2.carbon.hadoop.hive.jdbc.storage.db.DBOperation}
org.h2.jdbc.JdbcSQLException: NULL not allowed for column "CONSUMERKEY"; SQL statement:
INSERT INTO API\_RESPONSE\_SUMMARY\_DAY (time,resourcepath,context,servicetime,total\_response\_count,version,tzoffset,consumerkey,epoch,userid,apipublisher,api) VALUES (?,?,?,?,?,?,?,?,?,?,?,?) [90006-140]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.table.Column.validateConvertUpdateSequence(Column.java:294)
at org.h2.table.Table.validateConvertUpdateSequence(Table.java:621)
at org.h2.command.dml.Insert.insertRows(Insert.java:116)
at org.h2.command.dml.Insert.update(Insert.java:82)
at org.h2.command.CommandContainer.update(CommandContainer.java:70)
at org.h2.command.Command.executeUpdate(Command.java:199)
at org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:141)
at org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:127)
at org.wso2.carbon.hadoop.hive.jdbc.storage.db.DBOperation.insertData(DBOperation.java:175)
at org.wso2.carbon.hadoop.hive.jdbc.storage.db.DBOperation.writeToDB(DBOperation.java:63)
at org.wso2.carbon.hadoop.hive.jdbc.storage.db.DBRecordWriter.write(DBRecordWriter.java:35)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:589)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:467)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:758)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:467)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:758)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:467)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:758)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:964)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:781)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:707)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:467)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:248)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:518)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:419)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:257)
Expected output:
The same as above starting from TID the entire log entry should go as a single event.
Thanks,
Kasi
---
### Author:  [@kasi](https://discuss.elastic.co/u/kasi)
#### Post date: [March 7, 2016, 11:30pm UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458/8 "2016-03-07T23:30:25Z")
Also in the output I am seeing in the logstash console from filebeat, I am seeing the following
"message" =\> "\tat org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:141)",
looks like there is a tab in the beginning of the line, do not know if that will make a difference.
Thanks,
Kasi
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [March 8, 2016, 10:11am UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458/9 "2016-03-08T10:11:13Z")
It seems like the indentation of your multiline config is not correct. It should be under a prospector, but at the moment it seems to be on the top level.
I reformatted your post above to make it visible.
---
### Author:  [@kasi](https://discuss.elastic.co/u/kasi)
#### Post date: [March 9, 2016, 10:43pm UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458/10 "2016-03-09T22:43:51Z")
Thanks I checked my config it is intended only, the lines lost their format when I copied them
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [March 10, 2016, 11:43am UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458/11 "2016-03-10T11:43:40Z")
Can you try what happens if you use `pattern: ^TID:` instead of `pattern: '^TID:'`
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 9:54pm UTC](https://discuss.elastic.co/t/multiline-config-for-filebeat/43458/12 "2017-07-05T21:54:47Z")