# Any Help Un-structure log message to map structure message in Logstash
**URL:** https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258
**Category:** Logstash
**Created:** [December 3, 2015, 8:38am UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258 "2015-12-03T08:38:38Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@jayaram](https://discuss.elastic.co/u/jayaram)
#### Post date: [December 3, 2015, 8:38am UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/1 "2015-12-03T08:38:39Z")
Hello, I have the log message
2015-11-18 21:11:38,693 [WARN] [xx.web.common.filter.RequestFilter] NDC[UserPrincipal(ABCDF22602)] request (/member/control/loginAction) exceeded threshold; elapsed milliseconds since start: 188814
Can any one help me. How to tag or map the each same stored/moved to elasticsearch or any output resource
dose GROK plugin help for this mapping? any other plugin available to map/associate the each value to some TAG
I am trying like this in grok plugin but not working
match =\> { "timestamp" =\> "%{TOMCAT\_DATESTAMP:timestamp}"}
match =\> { "level" =\> "[%{LOGLEVEL:level}]" }
match =\> { "class" =\> "[%{JAVACLASS:class}]" }
match =\> { "logmessage" =\> "%{JAVALOGMESSAGE:logmessage}" }
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 3, 2015, 8:46am UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/2 "2015-12-03T08:46:10Z")
Yes, use the grok filter but correctly. It should look similar to this:
```auto
grok {
match => {
"message" => "%{TOMCAT_DATESTAMP:timestamp} \[%{LOGLEVEL:level}\] ..."
}
}
```
(Note the escaping of the square brackets.)
---
### Author:  [@jayaram](https://discuss.elastic.co/u/jayaram)
#### Post date: [December 3, 2015, 9:34am UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/3 "2015-12-03T09:34:12Z")
Hi Magnus,
can we map this NDC[UserPrincipal(ABCDEF22602) with Tag, Subtags Like below, Please give me syntax
MainTag : NDC[SubTag: UserPrincipal(UserTag: ABCDEF22602)
---
### Author:  [@jayaram](https://discuss.elastic.co/u/jayaram)
#### Post date: [December 3, 2015, 9:48am UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/4 "2015-12-03T09:48:15Z")
Hi Magnus,
I have tried as per your suggestion but its not mapping... I am seeing below output
## { "message" =\> "2015-12-01 23:58:00,330 [INFO] [xx.cbm.framework.engine.actionFrame.ActionFrameMonitor] NDC[] Completed checking for expired ActionFrames (there are now 0 ActionFrame references being monitored).\r", "@version" =\> "1", "@timestamp" =\> "2015-12-03T09:38:37.097Z", "host" =\> "MYVDI-XXX", "path" =\> "D:\basefarm\logs\myserver1.log", "tags" =\> [[0] "\_grokparsefailure" ] }
I am expecting below response
{
"message" =\> "2015-12-01 23:58:00,330 [INFO] [xx.cbm.framework.engine.actionFrame.ActionFrameMonitor] NDC[UserPrincipal(abcdef12121)] Completed checking
for expired ActionFrames (there are now 0 ActionFrame references being monitored).\r",
"timestamp" =\> ""2015-12-01 23:58:00",
"level" =\> "INFO",
"javaclass" =\> "xx.bac.framework.engine.actionFrame.ActionFrameMonitor",
"userid" =\> "abcdef12121",
"logmessage" =\> "Completed checking for expired ActionFrames (there are now 0 ActionFrame references being monitored).\r"
"@version" =\> "1",
"@timestamp" =\> "2015-12-03T09:38:37.097Z",
"host" =\> "MYVDI-2033",
"path" =\> "D:\basefarm\logs\myserver1.log",
"tags" =\> [
[0] "\_grokparsefailure"
]
}
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 3, 2015, 9:50am UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/5 "2015-12-03T09:50:00Z")
Without seeing your configuration it's impossible to know what's wrong. I suggest you use [http://grokconstructor.appspot.com/](http://grokconstructor.appspot.com/) as a help to create a grok expression that matches your data.
---
### Author:  [@jayaram](https://discuss.elastic.co/u/jayaram)
#### Post date: [December 3, 2015, 9:55am UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/6 "2015-12-03T09:55:18Z")
Please see my input configurations
input {
file {
path =\> "D:\basefarm\logs\myserver1.log"
start\_position =\> "beginning"
}
}
filter {
grok {
match =\> { "message" =\> "%{TOMCAT\_DATESTAMP:timestamp} [%{LOGLEVEL:level}] [%{JAVACLASS:class}] %{JAVALOGMESSAGE:logmessage}"}
}
grok{
match =\> { "exceptions" =\> "%{JAVASTACKTRACEPART}" }
}
date {
match =\> ["timestamp" , "yyyy-mm-dd HH:mm:ss,SSS Z"]
}
}
output {
elasticsearch {
hosts =\> ["127.0.0.1:9200"]
}
stdout { codec =\> rubydebug }
}
Please help me now
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 3, 2015, 9:58am UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/7 "2015-12-03T09:58:43Z")
One immediate problem is that you're not escaping the square brackets as I instructed you to do in a previous post. If it still doesn't work, use [http://grokconstructor.appspot.com/](http://grokconstructor.appspot.com/). Over and out.
---
### Author:  [@jayaram](https://discuss.elastic.co/u/jayaram)
#### Post date: [December 3, 2015, 10:03am UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/8 "2015-12-03T10:03:18Z")
no it is editor issue not showing single escape "" charactor in preview or in posted message i have added escaping the square brackets as you instructed..
---
### Author:  [@jayaram](https://discuss.elastic.co/u/jayaram)
#### Post date: [December 3, 2015, 4:28pm UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/9 "2015-12-03T16:28:17Z")
Hi Magnus,
I have checked grokconstructor but not able to find or convert the log message to required pattern maching please help me for the below log message
2015-11-18 21:11:38,693 [WARN] [xx.web.common.filter.RequestFilter] NDC[UserPrincipal(ABCDF22602)] request (/member/control/loginAction) exceeded threshold; elapsed milliseconds since start: 188814
This is not working please help me
{ "message" =\> "%{TOMCAT\_DATESTAMP:timestamp} \[%{LOGLEVEL:level}\] \[%{JAVACLASS:class}\] %{JAVALOGMESSAGE:logmessage}"}
This is the log4j pattern we using %d [%-5p] [%c] NDC[%x] %m%n
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 3, 2015, 7:32pm UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/10 "2015-12-03T19:32:32Z")
To debug this, start with the simplest possible expression:
```
%{TOMCAT_DATESTAMP:timestamp}.*
```
Does this work? If yes, add the next part:
```
%{TOMCAT_DATESTAMP:timestamp} \[%{LOGLEVEL:level}\].*
```
And so on. When it stops working you have found the problematic part of the expression.
Looking at the definition of TOMCAT\_DATESTAMP,
>
it ends with ISO8601\_TIMEZONE. There's no timezone in your log. Replacing TOMCAT\_DATESTAMP with TIMESTAMP\_ISO8601 could help.
---
### Author:  [@jayaram](https://discuss.elastic.co/u/jayaram)
#### Post date: [December 4, 2015, 2:03pm UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/11 "2015-12-04T14:03:12Z")
Thank you Magnus,
I have a question, I have give grok filter with following expression for message... in each event there may be no track trace still this is valid exprestion?
grok {
match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:logtimestamp} \[%{LOGLEVEL:level}\] \[%{JAVACLASS:class}\] %{JAVALOGMESSAGE:logmessage} "%{JAVASTACKTRACEPART:exceptions}"}
}
one more question if log message contains curling brackets { } the message is not parsing... grok filter failing to parse... is there any thing we need to add in grok filter to parse the message which contain { }
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 4, 2015, 6:28pm UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/12 "2015-12-04T18:28:11Z")
> I have a question, I have give grok filter with following expression for message... in each event there may be no track trace still this is valid exprestion?
> grok {
> match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:logtimestamp} [%{LOGLEVEL:level}\] [%{JAVACLASS:class}] %{JAVALOGMESSAGE:logmessage} "%{JAVASTACKTRACEPART:exceptions}"}
> }
There's an extra double quote just before the JAVASTACKTRACEPART token so Logstash won't accept it as it stands.
Given the definition of JAVASTACKTRACEPART,
>
it's clear that there are no optional elements, i.e. a grok expression that ends with a reference to JAVASTACKTRACEPART really must end with a stacktrace. You could make it optional by changing the end of your expression like this (note addition of parenthesis and question mark):
```
... %{JAVALOGMESSAGE:logmessage}( %{JAVASTACKTRACEPART:exceptions})?
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:19am UTC](https://discuss.elastic.co/t/any-help-un-structure-log-message-to-map-structure-message-in-logstash/36258/13 "2017-07-06T05:19:56Z")