# Using multiline codec for python stacktrace **URL:** \ **Category:** Logstash\ **Created:** [September 6, 2019, 5:35am UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346 "2019-09-06T05:35:52Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![rajdeep101](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Post date:** [September 6, 2019, 5:35am UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346/1 "2019-09-06T05:35:53Z")
Basically i have three types of lines in my log file. 1. Syntax error 2. Error timestamp internal server error url blah blah 3. traceback For the traceback part i am just checking if line begins with a space and appending it into previous, thereby i am getting all of the traceback in a single event. But in real my event should consist all 3 lines above mention in a single event. With my current config i am getting all three parts in different events with "tags" = [multiline, \_grokparsefailure] in traceback event. How can i club all 3 into one event and remove that multiline tag too. **Config file:** ``` input { file { path => "/home/rajdeep/Desktop/e9" codec => multiline { pattern => "^\s" what => "previous" } start_position => "beginning" } } filter { grok { match => [ "message", "SyntaxError: %{DATA:class}", "message", "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:logdate} Internal Server Error: %{URIPATHPARAM:url}", "(?m)message", "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:logdate} Internal Server Error: %{URIPATHPARAM:url} %{GREEDYDATA:traceback}" ] } date { match => ["logdate", "YYYY-MM-dd HH:mm:ss,SSS"] } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 6, 2019, 12:22pm UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346/2 "2019-09-06T12:22:07Z")
You can supress the tag by adding ``` multiline_tag => "" ``` to the codec. There is no way to say how else to change the codec without seeing the lines you wish to combine. ---
**Author:** ![rajdeep101](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Post date:** [September 6, 2019, 12:31pm UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346/3 "2019-09-06T12:31:06Z")
Hi @Badger Thanks for taking time out. I want to club following lines into 1 event. Whats happening with the above code is all three parts are coming as separate events. I even used aggregation, i can provide code for aggregation if there is not other way out. ``` SyntaxError: invalid syntax ERROR 2019-08-12 13:56:20,544 Internal Server Error: /api/v1/userapp/notification/seen/ Traceback (most recent call last): File "/home/ubuntu/virtualenvs/parkwheels-core-api/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 119, in get_response resolver_match = resolver.resolve(request.path_info) File "/home/ubuntu/virtualenvs/parkwheels-core-api/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 368, in resolve sub_match = pattern.resolve(new_path) File "/home/ubuntu/virtualenvs/parkwheels-core-api/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 368, in resolve sub_match = pattern.resolve(new_path) File "/home/ubuntu/virtualenvs/parkwheels-core-api/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 368, in resolve sub_match = pattern.resolve(new_path) File "/home/ubuntu/virtualenvs/parkwheels-core-api/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 240, in resolve return ResolverMatch(self.callback, args, kwargs, self.name) File "/home/ubuntu/virtualenvs/parkwheels-core-api/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 247, in callback self._callback = get_callable(self._callback_str) File "/home/ubuntu/virtualenvs/parkwheels-core-api/local/lib/python2.7/site-packages/django/utils/lru_cache.py", line 101, in wrapper result = user_function(*args, **kwds) File "/home/ubuntu/virtualenvs/parkwheels-core-api/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 106, in get_callable mod = import_module(mod_name) File "/usr/lib/python2.7/importlib/ __init__.py", line 37, in import_module __import__ (name) File "./parkwheels/userapp/views/notification_views.py", line 132 seen=True, seen_time=parkwheels_utils.get_current_datetime() created=created, updated=created) ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 6, 2019, 12:37pm UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346/4 "2019-09-06T12:37:39Z")
To do that using a multiline codec you need a pattern that matches the first line. ``` codec => multiline { pattern => '^SyntaxError:' negate => true what => "previous" auto_flush_interval => 1 } } ``` If you need to match multiple errors then maybe '^[a-zA-Z]+Error:'. ---
**Author:** ![rajdeep101](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Post date:** [September 6, 2019, 12:47pm UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346/5 "2019-09-06T12:47:17Z")
Hi thank you for that. But how can i have separate fields for all three merged events? Like first line goes to `error_type` 2nd into `message` and third into `traceback`.? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 6, 2019, 1:10pm UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346/6 "2019-09-06T13:10:12Z")
You can use grok to split the lines up. You can match a line using "one or more not-newline followed by newline" ``` grok { match => { "message" => "\A(?[^ ]+) (?[^ ]+) %{GREEDYDATA:stacktrace}" } } ``` ---
**Author:** ![rajdeep101](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Post date:** [September 6, 2019, 1:36pm UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346/7 "2019-09-06T13:36:34Z")
sorry but i am not so familiar with regex patterns. can i put logstash defined patterns in `[^]` instead of whole regex pattern for the the log line? Also can i just use below mentioned grok block and changes message fields to message, message1, message2 respectively and use them to pass through `[^]` like `[{"%message1"}]`. is it logically correct? > [@rajdeep101](#): > > ``` > `grok { match => [ "message", "SyntaxError: %{DATA:class}", > "message", "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:logdate} Internal Server Error: %{URIPATHPARAM:url}", > "(?m)message", "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:logdate} Internal Server Error: %{URIPATHPARAM:url} %{GREEDYDATA:traceback}" ] }` > > ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 6, 2019, 2:23pm UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346/8 "2019-09-06T14:23:14Z")
Yes, you could use individual patterns to match each part of the event. ---
**Author:** ![rajdeep101](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Post date:** [September 6, 2019, 2:24pm UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346/9 "2019-09-06T14:24:39Z")
But this pattern is not working. I have added the codec in input file (with changes). ``` filter { grok { match => [ "message", "SyntaxError: %{DATA:class}", "message1", "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:logdate} Internal Server Error: %{URIPATHPARAM:url}", "(?m)message2", "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:logdate} Internal Server Error: %{URIPATHPARAM:url} %{GREEDYDATA:stacktrace}" ] } grok { match => { "message" => "\A(?["%{message}"]+) (?["%{message1}"]+) %{GREEDYDATA:stacktrace}" } } date { match => ["logdate", "YYYY-MM-dd HH:mm:ss,SSS"] } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 6, 2019, 3:06pm UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346/10 "2019-09-06T15:06:50Z")
If you want to match multiple patterns then use ``` grok { break_on_match => false match => { "message" => [ "SyntaxError: %{DATA:class}", "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:logdate} Internal Server Error: %{URIPATHPARAM:url}", "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:logdate} Internal Server Error: %{URIPATHPARAM:url} %{GREEDYDATA:stacktrace}" ] } } ``` The first one does not match for reasons I am unclear on. The third does not match because the %{URIPATHPARAM:url} is not followed by a space in your event. ---
**Author:** ![rajdeep101](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Post date:** [September 7, 2019, 5:22am UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346/11 "2019-09-07T05:22:19Z")
Hi @Badger first one is matching now. Apparently a space was missing in the multiline codec where we define pattern to look for `'^SyntaxError:'` as you can see after colon we need a space. Can you tell me how can i have different fields for for all the 3 matches in a single event without using this method. > [@Badger](#): > > grok { match =\> { "message" =\> "\A(?[^]+) (?[^]+) %{GREEDYDATA:stacktrace}" } } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 5, 2019, 5:22am UTC](https://discuss.elastic.co/t/using-multiline-codec-for-python-stacktrace/198346/12 "2019-10-05T05:22:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.