Please tell me how to get message as well as the content of delimiter and AIR name in output as i am getting only message as output

Elastic Stack Logstash
1

input {
file {
path => "C:\PROJECT\ELK_STACK\logstash-6.1.2\bin\files\tcp_1_logs.log"
}
}
filter {
grok {
match => [ "message", "(?^ Error in sending batch file )(?.)" ]
match => ["message", "(? to AIR server: ) (?.) (?permanently. Retry attempts maxed out)" ]
}
}
output {
stdout {
codec => rubydebug
}
}

2

Please

  • format your configuration snippet as preformatted text using standard Markdown notation or the </> toolbar button,
  • show what's produced by your stdout output and
  • show an example line of input.
3

sample%20input_and_output
sample input_and_output.png1366×768 58.2 KB

sample code:

input {
file {
path => "C:\PROJECT\ELK_STACK\logstash-6.1.2\bin\files\tcp_1_logs.log"
}
}
filter {
grok {
match => [ "message", "(?^Error in sending batch file )(?.) (?to AIR server: ) (?.)
(?permanently. Retry attempts maxed out )" ]
}
}
output {
stdout {
codec => rubydebug
}
}

expected output: message + content in delimiter , batchname and AIR name.

4

Please have another look at the request in my first bullet.

5

format your configuration snippet as preformatted text using standard Markdown notation or the </> toolbar button.
Can you please elaborate this?

6

input {
file {
path => "C:\PROJECT\ELK_STACK\logstash-6.1.2\bin\files\tcp_1_logs.log"
}
}
filter {
grok {
match => [ "message", "(?^Outbound FTP failed for the batch )(?.*) (?Trying again....)" ]
}

grok {
	match => [ "message", "(?<delimeter>^Error in sending batch file )(?<batchname>.*) (?<delimeter>to AIR server: ) (?<AIR name>.*) 
		 (?<delimeter>permanently. Retry attempts maxed out )" ]
}
grok {
	match => [ "message","(?<delimeter>^File(s) matching )(?<fileNamePattern>.*) (?<delimeter> not found on remote site. Attempt)(?<attemptCount>.*)     			 	 (?<delimeter>out of ) (?<downloadFileAttempts>.*)" ]
}
grok {
	match => [ "message", "(?<delimeter>^could not find remote file(s) matching ) (?<fileNamePattern>.*) (?<delimeter>after) 
		 (<downloadFileAttempts>.*) (?<delimeter>attempts)" ] 
}
grok {
	match => [ "message", "(?<delimeter>^Failed to receive RPT file corresponding to )(?<dat file>.*)" ] 
}

grok {
	match => [ "message","(?<delimeter>^Inbound FTP for batch:) (?<fileNamePattern>.*) (?<delimeter>permanently. Retry attempts maxed out)" ] 
}

grok {
	match => [ "message", "(?<fileNamePattern>.*) (?<delimeter> not yet received from AIR & therefore excluding from consolidation report:)  			                 (?<inputFile>.*)" ]
}

}
output {
stdout {
codec => rubydebug
}
}

please see the formatted code above and issue is same as i am not getting other contents except message in the output.

7

Don't use multiple grok filters. Use a single filter with multiple expressions. Then the expressions will be tried in order until there's a match. There's an example in the grok filter documentation.

In the screenshot you posted earlier (please don't post screenshots if it's possible to copy/paste plain text) the message field begins with "ror" instead of "Error". That looks very suspicious.

8

filter {
grok {
match => [ "message", "(?^Error in sending batch file )(?.) (?to AIR server: ) (?.?)
(?permanently. Retry attempts maxed out )" ]
match => [ "message", "(?^Error in sending batch file )(?.) (?to AIR server: ) (?.)
(?permanently. Retry attempts maxed out )" ]
match => [ "message","(?^File(s) matching )(?.) (? not found on remote site. Attempt)(?.)" ]
match => [ "message", "(?^could not find remote file(s) matching ) (?.) (?after)
(.) (?attempts)" ]
match => [ "message", "(?^Failed to receive RPT file corresponding to )(?.)" ]
match => [ "message","(?^Inbound FTP for batch:) (?.) (?permanently. Retry attempts maxed out)" ]
match => [ "message", "(?.*) (? not yet received from AIR & therefore excluding from consolidation report:)" ]
}
}

Now also output is not coming as desired. Is there any restriction on the use of regex
(?.*) multiple times using match in grok filter?

9

Here's the documentation example I was thinking of:

filter {
  grok { match => { "message" => [ "Duration: %{NUMBER:duration}", "Speed: %{NUMBER:speed}" ] } }
}

Now also output is not coming as desired.

So what do you get?

Is there any restriction on the use of regex
(?.*) multiple times using match in grok filter?

Technically no, but multiple .* in the same expression is slow and could easily become ambiguous. Avoid it.

10

filter {
grok {
match => { "message" => [ "data: %{delimiter:Error in sending batch file}", (?.), "data : %{delimiter:to AIR server:}", (?.),
"data : %{delimiter: permanently. Retry attempts maxed out}" ] }
}
}

please check what is wrong in this syntax?

11

Unless you format the configuration as preformatted it'll get mangled and I won't see exactly what it looks like.

12 
filter {
	grok {
		match => { "message" => [ "data: %{delimiter:Error in sending batch file}",  (?<batchname>.*), "data : %{delimiter:to AIR server:}", (?<AIR name>.*),
				   "data : %{delimiter: permanently. Retry attempts maxed out}" ] }
	}
}

here i want output to give content of delimiter + content of AIR name + content of batch name for log like "Error in sending batch file BATCH_ADJUST_20180206_jj005 to AIR server: jjjj permanently. Retry attempts maxed out".
please suggest me how to get BATCH_ADJUST_20180206_jj005 in batchname, jjjj in AIR name and other information of log in delimiter.
I hope you get my point.

13

What's up with "data:" and "data :" at the beginning of the expression? I don't see it in the log example you posted earlier. Please post it again, as text.

14 
grok {
		match => [ "message", "(?<delimeter>^Error in sending batch file )(?<batchname>.*) (?<delimeter>to AIR server: ) (?<AIR name>.*) 
			     (?<delimeter>permanently. Retry attempts maxed out )" ]
	}

here, i want content of delimiter, batchname and AIR Server as output having log with content: "Error in sending batch file BATCH_ADJUST_20180206_jj005 to AIR server: jjjj permanently. Retry attempts maxed out".
After running this, i am getting only whole message as output but i want seperate batchname, AIR name and delimiter content along with message as ouput.
please suggest me how to go about it?

15

Follow the request I made in my previous post.

16

log: Error in sending batch file BATCH_ADJUST_20180206_jj005 to AIR server: jjjj permanently. Retry attempts maxed out

17

And still you don't manage to format the log entry as I requested. Stuff like that matters! If the log you post don't match what you're actually trying to parse my help might be of no use to you and then everyone's time is wasted.

Now, there appears to be two problems with your current expression:

  • If your log really begins with "log: " you need to drop the ^.
  • You're being sloppy with the whitespace.

This works:

(?<delimeter>Error in sending batch file )(?<batchname>.*) (?<delimeter>to AIR server: )(?<AIR name>.*) (?<delimeter>permanently. Retry attempts maxed out)

Notes:

  • I have no idea why you're capturing strings into the delimiter field in multiple places.
  • The repeated use of .* is very inefficient and could under some circumstances backfire, resulting in incorrect and surprising matches. Any grok expression with more than one .*, DATA, or GREEDYDATA should be heavily scrutinized.

Over and out.

18

thank you @magnusbaeck for your valuable suggestions and really good advices.:slightly_smiling_face:

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.