# Multiline problem with python stack trace **URL:** https://discuss.elastic.co/t/multiline-problem-with-python-stack-trace/129329 **Category:** Logstash **Created:** [April 24, 2018, 2:51pm UTC](https://discuss.elastic.co/t/multiline-problem-with-python-stack-trace/129329 "2018-04-24T14:51:40Z") **Posts on this page:** 7 **Page:** 1
### Author: ![romanskz](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@romanskz](https://discuss.elastic.co/u/romanskz) #### Post date: [April 24, 2018, 2:51pm UTC](https://discuss.elastic.co/t/multiline-problem-with-python-stack-trace/129329/1 "2018-04-24T14:51:40Z")
Hello, I'm using an elk stack. I have a log file from a python program. Some logs have stacktrace. I'm trying to use the multiline option in filebeat to get all stacktrace in addition to the error log. However, nothing more than just the log without the stacktrace appears. Perhaps my pattern is bad but I don't thing so A log that I want to have as one: ``` 2018-04-24 13:38:55 [scrapy.core.scraper] ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/scrapy/utils/defer.py", line 102, in iter_errback yield next(it) File "/usr/local/lib/python3.6/dist-packages/scrapy_splash/middleware.py", line 156, in process_spider_output for el in result: File "/usr/local/lib/python3.6/dist-packages/scrapy/spidermiddlewares/offsite.py", line 30, in process_spider_output for x in result: File "/usr/local/lib/python3.6/dist-packages/scrapy/spidermiddlewares/referer.py", line 339, in return (_set_referer(r) for r in result or ()) File "/usr/local/lib/python3.6/dist-packages/scrapy/spidermiddlewares/urllength.py", line 37, in return (r for r in result or () if _filter(r)) File "/usr/local/lib/python3.6/dist-packages/scrapy/spidermiddlewares/depth.py", line 58, in return (r for r in result or () if _filter(r)) File "/tmp/clothes-1524570832-w21yv00v.egg/clothes/spiders/clubmonaco.py", line 221, in parse_item if image_to_locate_script in script: TypeError: 'in ' requires string as left operand, not NoneType ``` My filebeat.yml ``` filebeat: prospectors: [{"paths": ["/data/logs/*/*/*.log"], "type": "log", "fields_under_root": true, "fields": {"index_type": "scrapy_log"}}] multiline.pattern: "^[a-zA-Z]+Error.*" multiline.negate: true multiline.match: before ``` logstash: ``` filter { if [index_type] == "scrapy_log" { grok { match => { "message" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME} \[%{NOTSPACE:user}] %{WORD:severity}\: %{GREEDYDATA:message}" } } grok { match => { "source" => "%{GREEDYDATA:folder}/%{NOTSPACE:crawler}\_%{NOTSPACE:country}/%{GREEDYDATA:filename}\.log" } } } } ``` ---
### Author: ![TimWard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timward/32/19574_2.png) [@TimWard](https://discuss.elastic.co/u/TimWard) #### Post date: [April 24, 2018, 3:10pm UTC](https://discuss.elastic.co/t/multiline-problem-with-python-stack-trace/129329/2 "2018-04-24T15:10:51Z")
For Java exceptions, which are essentially similar, I just use the "2" at the start of the timestamp. Tacky, sure, but it works, for the whole of this millenium. ``` multiline: pattern: ^2 negate: true match: after ``` ---
### Author: ![romanskz](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@romanskz](https://discuss.elastic.co/u/romanskz) #### Post date: [April 24, 2018, 3:23pm UTC](https://discuss.elastic.co/t/multiline-problem-with-python-stack-trace/129329/3 "2018-04-24T15:23:54Z")
Unfortunately, I tried with this ``` multiline.pattern : "^[0-9]{4}-[0-9]{2}-[0-9]{2}" multiline.negate: true multiline.match: after ``` that is in the same idea as yours, but nothing changed ---
### Author: ![TimWard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timward/32/19574_2.png) [@TimWard](https://discuss.elastic.co/u/TimWard) #### Post date: [April 24, 2018, 3:44pm UTC](https://discuss.elastic.co/t/multiline-problem-with-python-stack-trace/129329/4 "2018-04-24T15:44:14Z")
Which suggests that it's the grok pattern that's the problem then, not the multiline stuff. ---
### Author: ![romanskz](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@romanskz](https://discuss.elastic.co/u/romanskz) #### Post date: [April 24, 2018, 3:52pm UTC](https://discuss.elastic.co/t/multiline-problem-with-python-stack-trace/129329/5 "2018-04-24T15:52:27Z")
> [@TimWard](#): > > Which suggests that it's the grok pattern that's the problem then, not the multiline stuff I got logs in kibana for example : `2018-04-24 15:43:24 [root] INFO: start parse_item for url: http://clubmonaco.borderfree.com/product/index.jsp?productId=137926546` gave me ![exmapleKibana](https://us1.discourse-cdn.com/elastic/original/3X/a/a/aa300712602eb4019d64ce555fdaa368abcaa414.png) So I imagine if the multiline option works, I would have all the stacktrace, no? ---
### Author: ![romanskz](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@romanskz](https://discuss.elastic.co/u/romanskz) #### Post date: [May 4, 2018, 9:47am UTC](https://discuss.elastic.co/t/multiline-problem-with-python-stack-trace/129329/6 "2018-05-04T09:47:23Z")
multiline is a parameter of the prospectors, this solved my problem ``` filebeat_prospectors: - type: log paths: - "/data/logs/*/*/*.log" fields: index_type: scrapy_log fields_under_root: true multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2} multiline.negate: true multiline.match: after ``` ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [June 1, 2018, 9:47am UTC](https://discuss.elastic.co/t/multiline-problem-with-python-stack-trace/129329/7 "2018-06-01T09:47:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.