# Different lines structure for same log
**URL:** https://discuss.elastic.co/t/different-lines-structure-for-same-log/71779
**Category:** Logstash
**Created:** [January 16, 2017, 7:51pm UTC](https://discuss.elastic.co/t/different-lines-structure-for-same-log/71779 "2017-01-16T19:51:55Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@przecie](https://discuss.elastic.co/u/przecie)
#### Post date: [January 16, 2017, 7:51pm UTC](https://discuss.elastic.co/t/different-lines-structure-for-same-log/71779/1 "2017-01-16T19:51:55Z")
Hi
I have a following lines in my log:
--
- 2017-01-16 00:00:01,779 [Thread-31:SpaceChangePollerTask(SpaceChangePollerTask-1121721-Space:MAIN)] INFO (support.context.FrameworkSession) Using overridden system date: 2016-10-25
USER01 - 666353-11336-1 2017-01-16 15:37:58,545 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] INFO (support.context.FrameworkSession) Using overridden system date: 2016-10-25
USER01 - 666353-11336-1 2017-01-16 15:37:58,651 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR (domain.integration.RealtimeOutboundMessage) Problems invoking sender : XXXXXXXXXXXXXXXXXX
java.net.UnknownHostException: XXXXXXXXXXXXXXXXXXXx
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
So I'm trying to get them with following pattern:
"%{DATA:user} - %{UUID:uuid} %{TIMESTAMP\_ISO8601:timestamp} [%{DATA:information}] %{LOGLEVEL:loglevel} (%{JAVACLASS:java}) %{GREEDYDATA:line}"
but I'm not able to handle all lines properly. The problemes I've identified are the follows:
1. the number of spaces ' ' after LOGLEVEL is different : 2 for INFO and 1 for ERROR, so the pattern doesn't works for both cases . I've trient to add ( )? before [{JAVACLASS , but it doesn't help
2. I would like to handle multiple lines for the errors (not only) , so I've found multiline codes, but I need to define the pattern which will allow to recognize multiline, but in my case the lines are different , user and uuid are optional , so sometimes they are present and sometimes not. I have no idea how to address this problem.
I will appreciate any hint , Thanks
Przemek
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 17, 2017, 10:33am UTC](https://discuss.elastic.co/t/different-lines-structure-for-same-log/71779/2 "2017-01-17T10:33:49Z")
General comments on your grok expression:
- Don't use DATA in multiple places like that. Prefer more exact patterns, like NOTSPACE if the text you want to match can't contain spaces.
- I don't see why the UUID pattern would work since "666353-11336-1" isn't a UUID.
> the number of spaces ' ' after LOGLEVEL is different : 2 for INFO and 1 for ERROR, so the pattern doesn't works for both cases
Use `\s+` to match one or more whitespace characters.
---
### Author:  [@przecie](https://discuss.elastic.co/u/przecie)
#### Post date: [January 19, 2017, 10:16am UTC](https://discuss.elastic.co/t/different-lines-structure-for-same-log/71779/3 "2017-01-19T10:16:11Z")
Hi Magnus
Thank yo ufor your help.
\s+ and (?:%{PATTERN:field} ) solved most of the issues with parsing moste of the lines .
Nevertheless I have all the time the tag \_grokparsefailure
What is strange that I have it also when I'm simplifying to maximum my grok :
filter {
if [type]=="ccblog" {
grok {
match=\> ["message","%{GREEDYDATA:line}"]
}
}
}
the results is the following :
"\_source": {
"message": " - 2017-01-19 11:19:58,957 [Thread-25:SpaceChangePollerTask(SpaceChangePollerTask-1414964-Space:MAIN)] INFO (support.context.FrameworkSession) Using overridden system date: 2016-10-25",
"@version": "1",
"@timestamp": "2017-01-19T10:19:59.457Z",
"type": "ccblog",
"input\_type": "log",
"beat": {
"name": "vmA18DA9B",
"hostname": "vmA18DA9B",
"version": "5.1.1"
},
"source": "C:\ouaf\CCB2501\logs\system\spl\_service.log",
"offset": 13720622,
"host": "vmA18DA9B",
"tags": [
"beats\_input\_codec\_plain\_applied",
**"\_grokparsefailure"**
],
"line": " - 2017-01-19 11:19:58,957 [Thread-25:SpaceChangePollerTask(SpaceChangePollerTask-1414964-Space:MAIN)] INFO (support.context.FrameworkSession) Using overridden system date: 2016-10-25",
"syslog\_severity\_code": 5,
"syslog\_facility\_code": 1,
"syslog\_facility": "user-level",
"syslog\_severity": "notice"
},
How to avoid \_grokparsefailure ?
Regards
Przemek
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 19, 2017, 11:49am UTC](https://discuss.elastic.co/t/different-lines-structure-for-same-log/71779/4 "2017-01-19T11:49:04Z")
Perhaps there's another grok filter in your configuration. Do you have any extra files in /etc/logstash/conf.d? Logstash reads _all_ files there.
---
### Author:  [@przecie](https://discuss.elastic.co/u/przecie)
#### Post date: [January 20, 2017, 9:27am UTC](https://discuss.elastic.co/t/different-lines-structure-for-same-log/71779/5 "2017-01-20T09:27:50Z")
That's it.
I had a backup file \*.bak and I was not aware that Logstash will take it into account .
Now it works like a charm!
Thanks a lot!
Przemek
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 17, 2017, 9:28am UTC](https://discuss.elastic.co/t/different-lines-structure-for-same-log/71779/6 "2017-02-17T09:28:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.