Added new line in my log and cannot get the output correctly in filbeat (need to remove new line)

Elastic Stack Beats
1

How to remove newline using multiline config in filebeat

Log file:

2020-06-01T07:44:31.300103 H 80 DHandler.cpp:953 stdlog sql_execute 11201 9 handcrafted admin 431-856b {"query","client","execution_time","total_time"} {"SELECT user.id AS user_meta_id, user_meta.security_risk_score AS user_meta_security_risk_score
FROM user_meta
WHERE 2 = ANY user_meta.department_ids ORDER BY user_meta.security_risk_score DESC
LIMIT 10","http:10.10.10.1","8","9"}

Expect:
Every FROM, WHERE and LIMIT has a new line before this string. it should come single line sql

2

This is created to multiple lines but need to access like single line in filebeat.yml
Any help?

2020-06-01T15:57:48.499723 H 15 Handler.cpp:93 stdlog sql_execute 34697 9 handcraft user 748-Nue0 {"query_str","client","execution_time","total_time"} {"SELECT user_meta.id AS user _meta_id, user_meta.security_risk_score AS user_meta_security_risk_score
FROM user_meta
WHERE 0 = 1 ORDER BY user_meta.security_risk_score DESC
LIMIT 10","http:192.168.1.213","9","9"}

3

I just would use the Multiline config for the timestamp example here:
https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html#_timestamps

So in your case it would be to place the multiline config into you input section:

multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
4

Hi Andre,
Thanks a lot your suggestion. Actually I have used this pattern and some other pattern for my filebeat. Its work for few queries. Not all the queries (long queries with JOIN, select in side the select ). Is there any query string limitation on filebeat?

5

Hm, can you write an example that doesn't work? You can remove confidential information.

6

Hi Andre,

Sorry for the late reply. Actually, your suggestion was good. it was work for me. Thanks a lot.

7

Perfect, that's great to hear.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.