Please tell me how to get message as well as the content of delimiter and AIR name in output as i am getting only message as output

abhi210797 · February 9, 2018, 7:06am

input {
file {
path => "C:\PROJECT\ELK_STACK\logstash-6.1.2\bin\files\tcp_1_logs.log"
}
}
filter {
grok {
match => [ "message", "(?^ Error in sending batch file )(?.)" ]
match => ["message", "(? to AIR server: ) (?.) (?permanently. Retry attempts maxed out)" ]
}
}
output {
stdout {
codec => rubydebug
}
}

magnusbaeck · February 9, 2018, 7:37am

Please

format your configuration snippet as preformatted text using standard Markdown notation or the </> toolbar button,
show what's produced by your stdout output and
show an example line of input.

abhi210797 · February 9, 2018, 8:33am

sample code:

input {
file {
path => "C:\PROJECT\ELK_STACK\logstash-6.1.2\bin\files\tcp_1_logs.log"
}
}
filter {
grok {
match => [ "message", "(?^Error in sending batch file )(?.) (?to AIR server: ) (?.)
(?permanently. Retry attempts maxed out )" ]
}
}
output {
stdout {
codec => rubydebug
}
}

expected output: message + content in delimiter , batchname and AIR name.

magnusbaeck · February 9, 2018, 8:47am

Please have another look at the request in my first bullet.

abhi210797 · February 9, 2018, 8:56am

format your configuration snippet as preformatted text using standard Markdown notation or the </> toolbar button.
Can you please elaborate this?

abhi210797 · February 12, 2018, 5:17am

input {
file {
path => "C:\PROJECT\ELK_STACK\logstash-6.1.2\bin\files\tcp_1_logs.log"
}
}
filter {
grok {
match => [ "message", "(?^Outbound FTP failed for the batch )(?.*) (?Trying again....)" ]
}

grok {
	match => [ "message", "(?<delimeter>^Error in sending batch file )(?<batchname>.*) (?<delimeter>to AIR server: ) (?<AIR name>.*) 
		 (?<delimeter>permanently. Retry attempts maxed out )" ]
}
grok {
	match => [ "message","(?<delimeter>^File(s) matching )(?<fileNamePattern>.*) (?<delimeter> not found on remote site. Attempt)(?<attemptCount>.*)     			 	 (?<delimeter>out of ) (?<downloadFileAttempts>.*)" ]
}
grok {
	match => [ "message", "(?<delimeter>^could not find remote file(s) matching ) (?<fileNamePattern>.*) (?<delimeter>after) 
		 (<downloadFileAttempts>.*) (?<delimeter>attempts)" ] 
}
grok {
	match => [ "message", "(?<delimeter>^Failed to receive RPT file corresponding to )(?<dat file>.*)" ] 
}

grok {
	match => [ "message","(?<delimeter>^Inbound FTP for batch:) (?<fileNamePattern>.*) (?<delimeter>permanently. Retry attempts maxed out)" ] 
}

grok {
	match => [ "message", "(?<fileNamePattern>.*) (?<delimeter> not yet received from AIR & therefore excluding from consolidation report:)  			                 (?<inputFile>.*)" ]
}

}
output {
stdout {
codec => rubydebug
}
}

please see the formatted code above and issue is same as i am not getting other contents except message in the output.

magnusbaeck · February 12, 2018, 7:35am

Don't use multiple grok filters. Use a single filter with multiple expressions. Then the expressions will be tried in order until there's a match. There's an example in the grok filter documentation.

In the screenshot you posted earlier (please don't post screenshots if it's possible to copy/paste plain text) the message field begins with "ror" instead of "Error". That looks very suspicious.

abhi210797 · February 12, 2018, 8:47am

filter {
grok {
match => [ "message", "(?^Error in sending batch file )(?.) (?to AIR server: ) (?.?)
(?permanently. Retry attempts maxed out )" ]
match => [ "message", "(?^Error in sending batch file )(?.) (?to AIR server: ) (?.)
(?permanently. Retry attempts maxed out )" ]
match => [ "message","(?^File(s) matching )(?.) (? not found on remote site. Attempt)(?.)" ]
match => [ "message", "(?^could not find remote file(s) matching ) (?.) (?after)
(.) (?attempts)" ]
match => [ "message", "(?^Failed to receive RPT file corresponding to )(?.)" ]
match => [ "message","(?^Inbound FTP for batch:) (?.) (?permanently. Retry attempts maxed out)" ]
match => [ "message", "(?.*) (? not yet received from AIR & therefore excluding from consolidation report:)" ]
}
}

Now also output is not coming as desired. Is there any restriction on the use of regex
(?.*) multiple times using match in grok filter?

magnusbaeck · February 12, 2018, 1:01pm

Here's the documentation example I was thinking of:

filter {
  grok { match => { "message" => [ "Duration: %{NUMBER:duration}", "Speed: %{NUMBER:speed}" ] } }
}

Now also output is not coming as desired.

So what do you get?

Is there any restriction on the use of regex
(?.*) multiple times using match in grok filter?

Technically no, but multiple .* in the same expression is slow and could easily become ambiguous. Avoid it.

abhi210797 · February 12, 2018, 6:03pm

filter {
grok {
match => { "message" => [ "data: %{delimiter:Error in sending batch file}", (?.), "data : %{delimiter:to AIR server:}", (?.),
"data : %{delimiter: permanently. Retry attempts maxed out}" ] }
}
}

please check what is wrong in this syntax?

magnusbaeck · February 12, 2018, 9:02pm

Unless you format the configuration as preformatted it'll get mangled and I won't see exactly what it looks like.

abhi210797 · February 13, 2018, 5:05am

filter {
	grok {
		match => { "message" => [ "data: %{delimiter:Error in sending batch file}",  (?<batchname>.*), "data : %{delimiter:to AIR server:}", (?<AIR name>.*),
				   "data : %{delimiter: permanently. Retry attempts maxed out}" ] }
	}
}

here i want output to give content of delimiter + content of AIR name + content of batch name for log like "Error in sending batch file BATCH_ADJUST_20180206_jj005 to AIR server: jjjj permanently. Retry attempts maxed out".
please suggest me how to get BATCH_ADJUST_20180206_jj005 in batchname, jjjj in AIR name and other information of log in delimiter.
I hope you get my point.

magnusbaeck · February 13, 2018, 7:10am

What's up with "data:" and "data :" at the beginning of the expression? I don't see it in the log example you posted earlier. Please post it again, as text.

abhi210797 · February 13, 2018, 8:23am

grok {
		match => [ "message", "(?<delimeter>^Error in sending batch file )(?<batchname>.*) (?<delimeter>to AIR server: ) (?<AIR name>.*) 
			     (?<delimeter>permanently. Retry attempts maxed out )" ]
	}

here, i want content of delimiter, batchname and AIR Server as output having log with content: "Error in sending batch file BATCH_ADJUST_20180206_jj005 to AIR server: jjjj permanently. Retry attempts maxed out".
After running this, i am getting only whole message as output but i want seperate batchname, AIR name and delimiter content along with message as ouput.
please suggest me how to go about it?

magnusbaeck · February 13, 2018, 8:33am

Follow the request I made in my previous post.

abhi210797 · February 14, 2018, 5:57am

log: Error in sending batch file BATCH_ADJUST_20180206_jj005 to AIR server: jjjj permanently. Retry attempts maxed out

magnusbaeck · February 14, 2018, 7:15am

And still you don't manage to format the log entry as I requested. Stuff like that matters! If the log you post don't match what you're actually trying to parse my help might be of no use to you and then everyone's time is wasted.

Now, there appears to be two problems with your current expression:

If your log really begins with "log: " you need to drop the ^.
You're being sloppy with the whitespace.

This works:

(?<delimeter>Error in sending batch file )(?<batchname>.*) (?<delimeter>to AIR server: )(?<AIR name>.*) (?<delimeter>permanently. Retry attempts maxed out)

Notes:

I have no idea why you're capturing strings into the delimiter field in multiple places.
The repeated use of .* is very inefficient and could under some circumstances backfire, resulting in incorrect and surprising matches. Any grok expression with more than one .*, DATA, or GREEDYDATA should be heavily scrutinized.

Over and out.

abhi210797 · February 14, 2018, 8:07am

thank you @magnusbaeck for your valuable suggestions and really good advices.

system · March 14, 2018, 8:07am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.