It looks like my formatting got blown out so all my config changes are now in pastebin in a hopefully easier to read format at https://pastebin.com/DJ0FawPp
I'm trying to use logstash to parse scrapy logs. My logstash config is as follows:
see first config in pastebin link
This works great and outputs (among other lines) this: (See pastebin)
Now, I also want to get the response code, url and the json entry contained in the new 'message' in the above, so I change my conf filter to the new one in the pastebin
and add a file in /vagrant/patterns that contains this:
I have checked the above in grok debugger, and it yields the expected results, but running it in logstash does not display the new response code, url, or jitem fields. Is there something wrong with the way I'm doing things? Is there something I have to do to get the new entries to show up?