Parsing repeated patterns

Elastic Stack Logstash
1

Hi,

I have a log line that has patterns repeating multiple times and the number of times could be varying from line to line. Is there a recipe for this ?

For example,

abc.com 300 IN CNAME xyz.com
xyz.com 300 IN CNAME dgh.com
dgh.com 300 IN CNAME jkl.com
jkl.com 300 IN A 1.2.3.4

The number of CNAME lines is arbitrary. But the structure is same.

thanks
mohan

2

Do you want to parse those four lines as a single message, or what are you trying to do? What is the desired output from Logstash given that input?

3

Hi,

Actually it appears like this..in one line like this:

10-Jan-2011 12:30:42.462 client 1.2.3.4#12345: view 1: UDP: query: abc.com IN A response: NOERROR +E abc.com 300 IN CNAME xyz.com; xyz.com 300 IN CNAME dgh.com; dgh.com 300 IN CNAME jkl.com; kl.com 300 IN A 1.2.3.4

I am trying to parse the individual components so that I can reconstruct this to a different format using ruby. In the 'query' above, I need to extract "abc.com" and the query type (A) and in the 'response' NOERROR, and then the series of fields broken down by . I guess this will go into an array so that I can extract them and mutate appropriately.

thanks
mohan

4

Sorry.. the line got cut..I meant "line broken down by . Note that there is no limit for the repetition. Any help would be appreciated ..

thanks
mohan

5

Use grok to extract the input query, the result of the query (e.g. NOERROR), and the long string with the resulting records into discrete fields. Then use the mutate filter's split option to split the list of records into an array. Each element of that array can then be processed further but I suspect you'll need a ruby filter for that.

6

Would you be able to provide an example on how to capture the multiple occurrences of URL in every line of a log.

7

@mparthas Were you able to solve the issue ??? I am also stuck on the similar issue.