Repeating field name seems to silently fail


(Caleb Cullen) #1

For the last 6 weeks or so I have been developing Logstash filters for the various logs our network produces, and I have been on a steep learning curve. It's possible that I am simply ignorant of some basic fact, but I've looked at various examples online and I've seen for myself that if a field name is repeated, Logstash ends up making that field into an array and stores all of the values captured to that fieldname in that array.

I have logs produced by AMaViS which have some fields which may have one or sometimes more than one value, usually separated by commas, sometimes comma-space. So I'd like to use a regular expression (snippet) such as:

(?:%{WORD:data}, )*%{WORD:data}

or its reverse:

%{WORD:data}(?:, %{WORD:data})*

don't seem to work at all.

These patterns work in the Grok debugger but they do not function at all inside Logstash. When I put these patterns into the logstash config, all the entries which ought to match instead get marked as _grokparsefailure

In the rare occasion that something like

(?:%{WORD:data},)+

will work, it seems to work as expected.

That seems mysterious to me. Can anyone shed any light on why patterns using * should fail when a similar pattern using + succeeds?

It's clear to me that I can just capture the list portions and use kv or probably in my case Ruby to parse them into the lists that I want. But it seems like the grok parser should handle this, and that would certainly be simpler.

I am new to the forum and I'm not presently certain how to attach files, so I will look into that, in order to provide a concrete example. (It looks like attachments aren't supported.)


(Magnus B├Ąck) #2

Could you give an example of the kind of data you want to parse (one representative line is enough) as well as the full grok expression you're attempting to use?


(Caleb Cullen) #3

In testing one last time the scenario I complained about, I found that today I was able to get that construct to function properly. So I must retract my complaint.

I believe there may have been a trailing space at the end of the pattern which wasn't working properly.

The patterns I've been working with are extremely hairy, so at a certain point I kinda go crosseyed and start missing details.

Please accept my apologies for projecting my own errors onto grok.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.