# New fields not being created and grok help
**URL:** https://discuss.elastic.co/t/new-fields-not-being-created-and-grok-help/37895
**Category:** Logstash
**Created:** [December 23, 2015, 10:30pm UTC](https://discuss.elastic.co/t/new-fields-not-being-created-and-grok-help/37895 "2015-12-23T22:30:16Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@tweetybird](https://discuss.elastic.co/u/tweetybird)
#### Post date: [December 23, 2015, 10:30pm UTC](https://discuss.elastic.co/t/new-fields-not-being-created-and-grok-help/37895/1 "2015-12-23T22:30:17Z")
I'm very new to all this ELK stuff and I'm having a lot of trouble getting logstash to parse my file. I followed a tutorial and managed to get filebeat to send my logs to logstash and I can see them in kibana. Now I'm trying to use grok to split out a bunch of items into their own fields but nothing seems to be working.
The lines from my logs have spaces after the words and then a tab character just before the next entry (and sometimes there may be nothing in a given field). I've also got a few lines at the top of the file which I need to exclude completely but I have no clue how to do that (unless the grok thing will handle it although the very first line in the file may be a match because it's a date and time)
Below is a sample line from the log along with the relevant .conf files for logstash. I can't figure out what I'm doing wrong. Can anyone help?
```
Time Category Severity Entry type Local time UTC time Machine Application type User Entity Entity type Entity guid Details
3:14:34 PM Security Information Entity created 5/28/2015 3:14:34 PM 5/28/2015 7:14:34 PM MYMACHINE MyApp 1.1.1.1 - myentity myentitytype {00000000-0000-0000-0007-0050F9100E7F} My detials
```
in the "mypatterns" file
TIMESTAMP\_12HOUR %{TIME} (AM|PM)
DATETIME\_12HOUR %{DATE\_US} %{TIME} (AM|PM)
MULTIWORD .[^\t]+
input {
beats {
port =\> 5046
type =\> "mylog"
ssl =\> true
ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
if [type] == "mylog" {
grok {
patterns\_dir =\> "/etc/logstash/mypatterns"
match =\> { "message" =\> "%{TIMESTAMP\_12HOUR:logtime} %{WORD:category} %{WORD:severity} %{MULTIWORD:type} %{DATETIME\_12HOUR:localtime} %{DATETIME\_12HOUR:utctime} %{HOSTNAME:hostname} %{MULTIWORD:apptype} %{USER:user} %{MULTIWORD:entity} %{MULTIWORD:entitytype} %{MULTIWORD:guid} %{GREEDYDATA:details}" }
add\_field =\> ["received\_at", "%{@timestamp}"]
add\_field =\> ["received\_from", "%{hostname}"]
```
}
date {
match => ["logtime", "HH:mm:ss a"]
}
date {
match => ["localtime", "MM/dd/YYYY HH:mm:ss a"]
}
date {
match => ["utctime", "MM/dd/YYYY HH:mm:ss a"]
}
}
```
}
output {
elasticsearch { hosts =\> ["localhost:9200"] }
stdout { codec =\> rubydebug }
}
---
### Author:  [@Ralph\_Lo](https://discuss.elastic.co/u/Ralph_Lo)
#### Post date: [December 28, 2015, 11:34am UTC](https://discuss.elastic.co/t/new-fields-not-being-created-and-grok-help/37895/2 "2015-12-28T11:34:14Z")
Hi tweetybird,
first I ran also into the same problem when started with logstash. So far your match- function looks fine, except using blanks (" ") between the predefined patterns.
If you use "\s" instead of blanks " ", this should work. Remember that the everything in the message field must exactly match with your patterns.
So for example if your message- field looks like this:
Time: 3:14:34 PM Category:Security Severity:Information Entry type:Entity created
and your patterns file looks like the one you've mentioned, the match field in your filter should look like:
(Time: 3:14:34 PM Category:Security Severity:Information Entry type:Entity created)
match =\> { "message" =\> "Time:\s%{TIMESTAMP\_12HOUR:localtime}\sCategory:%(WORD:category)\sSeverity:%(WORD:severity)\sEntry\stype:%(WORD:entry\_type) " }
So your matching pattern starts with the word time, since you wanna filter this word, you just write "Time:". The "\s" is a replacement for the withespace/blank character. Then you save the time in the "localtime" field. This is followed by a withespace again ("\s"). Your pattern continues with the word Severity, so you just type "Severity:" and save the value in the "severity" field. Again a withespace character ("\s") followed by the "Entry type" pattern...and so on.
Hope this helps!
Cheers,
Ralph
---
### Author:  [@tweetybird](https://discuss.elastic.co/u/tweetybird)
#### Post date: [December 28, 2015, 2:57pm UTC](https://discuss.elastic.co/t/new-fields-not-being-created-and-grok-help/37895/3 "2015-12-28T14:57:59Z")
Hi Ralph,
Thanks for your help. Using your suggestions and the online grok debugger tools, I've managed to get part of my log line with following command:
%{TIMESTAMP\_12HOUR:logtime}\s\t%{WORD:category}\t%{WORD:severity}\t%{MULTIWORD:type}\s\t%{DATETIME\_12HOUR:localtime}\s\t%{DATETIME\_12HOUR:utctime}\s\t%{HOSTNAME:hostname}
However the logs entry after that aren't being picked up because it's a lot of white space (spaces) and tabs between between the hostname and the next field with real data. Is there a trick to pickup/skip all that whitespace?
I also saw my MULTIWORD:type, may not be correct. It picks up the Start Logging but it also gets all the spaces after it and I can't figure out the correct regex to just get the two works and skip all the whitespace after it. Do you know the correct syntax for that?
Thanks
---
### Author:  [@Ralph\_Lo](https://discuss.elastic.co/u/Ralph_Lo)
#### Post date: [December 28, 2015, 3:56pm UTC](https://discuss.elastic.co/t/new-fields-not-being-created-and-grok-help/37895/4 "2015-12-28T15:56:23Z")
Hi tweetybird,
> However the logs entry after that aren't being picked up because it's a lot of white space (spaces) and tabs between between the hostname and the next field with real data. Is there a trick to pickup/skip all that whitespace?
The only solution which comes into my mind at the moment is to use the **mutate** function. I've never used it before, I just read that you can replace characters , maybe it's a got approach to get rid of all the withspaces.
> However the logs entry after that aren't being picked up because it's a lot of white space (spaces) and tabs between between the hostname and the next field with real data. Is there a trick to pickup/skip all that whitespace?
Check out the following links:
>
and
[http://grokdebug.herokuapp.com/](http://grokdebug.herokuapp.com/)
You could include an example of the line with the white spaces and the tabs if you don't get any further.
Cheers,
Ralph
---
### Author:  [@tweetybird](https://discuss.elastic.co/u/tweetybird)
#### Post date: [December 30, 2015, 5:30pm UTC](https://discuss.elastic.co/t/new-fields-not-being-created-and-grok-help/37895/5 "2015-12-30T17:30:01Z")
The good news is I think I figured out a grok pattern that works, the bad news is I don't see any new fields in Kibana.
In Kibana, under the discover tab, I see the new log entries but when I expand one, all the extra fields i defined in my grok pattern are not there. Is there any way I can debug this to see what i'm not getting all the extra fields?
---
### Author:  [@tweetybird](https://discuss.elastic.co/u/tweetybird)
#### Post date: [December 30, 2015, 7:58pm UTC](https://discuss.elastic.co/t/new-fields-not-being-created-and-grok-help/37895/6 "2015-12-30T19:58:47Z")
Edit: removed. figured out the fields and duplication stuff. think i'm ok now
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:17am UTC](https://discuss.elastic.co/t/new-fields-not-being-created-and-grok-help/37895/7 "2017-07-06T05:17:07Z")