# Logstash not getting "message" from log file input **URL:** \ **Category:** Logstash\ **Created:** [September 15, 2015, 4:31pm UTC](https://discuss.elastic.co/t/logstash-not-getting-message-from-log-file-input/29333 "2015-09-15T16:31:24Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Jtc](https://avatars.discourse-cdn.com/v4/letter/j/eb8c5e/32.png) [@Jtc](https://discuss.elastic.co/u/Jtc)\ **Post date:** [September 15, 2015, 4:31pm UTC](https://discuss.elastic.co/t/logstash-not-getting-message-from-log-file-input/29333/1 "2015-09-15T16:31:24Z")
If I start Logstash with -f /etc/logstash/conf.d then one of my conf files (windowslogs.conf; whose job it is to read and parse Windows log files forwarded by NXlog on Windows hosts) does not work. Logstash recognizes that lines are being added to the log files because I get \_grokParseFail tags, but this is because there is no "message" portion of the entry. But if I start with -f /etc/logstash/conf.d/windowslogs.conf then it reads those Windows log files fine and parses them without problem. Any ideas? I'm pretty much a Logstash NOOB. Thanks in advance!!! Here's an example: Line in log: Sep 15 10:25:25 144.17.53.5 {"EventTime": "2015-09-15 10:25:25","Hostname":"BBTSSNOW","Keywords":-9214364837600034816,"EventType":"AUDIT\_SUCCESS","SeverityValue":2,"Severity":"INFO","EventID":5156,"SourceName":"Microsoft-Windows-Security-Auditing","ProviderGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","Version":1,"Task":12810,"OpcodeValue":0,"RecordNumber":75370997,"ProcessID":4,"ThreadID":88,"Channel":"Security","Message":"The Windows Filtering Platform has permitted a connection.\r\n\r\nApplication Information:\r\n\tProcess ID:\t\t976\r\n\tApplication Name:\t\device\harddiskvolume2\windows\system32\svchost.exe\r\n\r\nNetwork Information:\r\n\tDirection:\t\tOutbound\r\n\tSource Address:\t\t144.17.53.5\r\n\tSource Port:\t\t55170\r\n\tDestination Address:\t144.17.3.82\r\n\tDestination Port:\t\t53\r\n\tProtocol:\t\t17\r\n\r\nFilter Information:\r\n\tFilter Run-Time ID:\t0\r\n\tLayer Name:\t\tConnect\r\n\tLayer Run-Time ID:\t48","Category":"Filtering Platform Connection","Opcode":"Info","Application":"\device\harddiskvolume2\windows\system32\svchost.exe","Direction":"%%14593","SourceAddress":"144.17.53.5","SourcePort":"55170","DestAddress":"144.17.3.82","DestPort":"53","Protocol":"17","FilterRTID":"0","LayerName":"%%14611","LayerRTID":"48","RemoteUserID":"S-1-0-0","RemoteMachineID":"S-1-0-0","EventReceivedTime":1442334325,"SourceModuleName":"in","SourceModuleType":"im\_msvistalog"}#015 What I get from Logstash (using file output, rubydebug) I realize this does not correspond to the log entry as above, just an example: {"@version":"1","@timestamp":"2015-09-14T22:18:17.316Z","host":"elk.snow.edu","path":"/var/log/hosts/windows/144.17.53.5/user-2015-09-14.log","type":"windows","tags":["windows","\_grokParseFailWin1"]} Here's my grok: grok { tag\_on\_failure =\> ["\_grokParseFailWin1"] match =\> ["message", "%{SYSLOGTIMESTAMP:@timestamp} %{SYSLOGHOST:syslog\_host} %{JSON:raw\_message}%{GREEDYDATA}"] } After the grok I do: json { source =\> "raw\_message" } Here's JSON in custom patterns (since it's greedy, it will read to the last closing brace) JSON {.\*} (The curly braces are escaped, but this post is stripping them off) ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 15, 2015, 4:38pm UTC](https://discuss.elastic.co/t/logstash-not-getting-message-from-log-file-input/29333/2 "2015-09-15T16:38:39Z")
Do you have any other files in /etc/logstash/conf.d besides windowslogs.conf? ---
**Author:** ![Jtc](https://avatars.discourse-cdn.com/v4/letter/j/eb8c5e/32.png) [@Jtc](https://discuss.elastic.co/u/Jtc)\ **Post date:** [September 15, 2015, 4:43pm UTC](https://discuss.elastic.co/t/logstash-not-getting-message-from-log-file-input/29333/3 "2015-09-15T16:43:00Z")
Yes, there are conf files for parsing other types of log files. Specifically, I have an authpriv.conf, firewall-outside.conf, linuxsyslogs.conf and of course windowslogs.conf. The others seem to work fine. Within each of those, when I read an input file I mark it with a type and tags so that I can, for example, in the filter and output sections use: if "windows" in [tags] { ........ ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 15, 2015, 5:00pm UTC](https://discuss.elastic.co/t/logstash-not-getting-message-from-log-file-input/29333/4 "2015-09-15T17:00:25Z")
It seems something's off with those conditions in the other files. Your example output record is completely lacking a `message` field. How is that possible? Other comments: - Don't grok directly to `@timestamp`, at least not when the input string isn't ISO8601. Use the date filter to populate that field. - To avoid surprises when posting configuration snippets etc here, format it as preformatted text (Ctrl/Cmd+K). - The %{GREEDYDATA} at the end of the grok expression serves no purpose. ---
**Author:** ![Jtc](https://avatars.discourse-cdn.com/v4/letter/j/eb8c5e/32.png) [@Jtc](https://discuss.elastic.co/u/Jtc)\ **Post date:** [September 15, 2015, 8:24pm UTC](https://discuss.elastic.co/t/logstash-not-getting-message-from-log-file-input/29333/5 "2015-09-15T20:24:58Z")
Hi, your mention of something being off with the other config files led me to go through them and I indeed did find a problem. I had a closing brace in a wrong place which put a snippet of code (a mutate that deletes the "message" if grok'ed properly) outside of the if statement thereby always deleting the "message" portion. I imagine it only caused a problem with the windowslogs.conf because the problem was in the weblogs.conf which is alphabetically before windowslogs since I believe they are processed lexigraphically in order. Thanks for the tip off!! And by the way, I'lve implemented the date filter and removed unecessary %{GREEDYDATA}. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 16, 2015, 3:41am UTC](https://discuss.elastic.co/t/logstash-not-getting-message-from-log-file-input/29333/6 "2015-09-16T03:41:15Z")
It sounds like the place where the brace was wrong looked something like ``` grok { ... } if "_grokparsefailure" not in [tags] { mutate { remove_field => ["message"] } } ``` but that can be simplified like this: ``` grok { ... remove_field => ["message"] } ``` ---
**Author:** ![Jtc](https://avatars.discourse-cdn.com/v4/letter/j/eb8c5e/32.png) [@Jtc](https://discuss.elastic.co/u/Jtc)\ **Post date:** [September 16, 2015, 3:52pm UTC](https://discuss.elastic.co/t/logstash-not-getting-message-from-log-file-input/29333/7 "2015-09-16T15:52:01Z")
Oh cool. Yep, that's exactly what I had going on. You have been most helpful, thank you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:28am UTC](https://discuss.elastic.co/t/logstash-not-getting-message-from-log-file-input/29333/8 "2017-07-06T05:28:53Z")