How to check the tags values in logstash

Elastic Stack Logstash
1

Dear All,

I'm trying to read my access log but i'm getting below error. Please help

Log lines:
[1047:3a80:421:319:7e91:ac89:eda5:8c9d], 10.10.10.10:10 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/query HTTP/1.1" 401 166 - 3414 - abc-rst-rst5:9080 -
[1047:204:949e:259e:45cd:7d72:1397:1bdc], 10.10.10.10:12 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/init HTTP/1.1" 401 166 - 4427 - abc-xyz-rst4:9080 -
10.10.10.10:98 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/init HTTP/1.1" 401 166 - 3205 - abc-rst-rst4:9080 -
11.11.11.11:42 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/init HTTP/1.1" 401 166 - 3019 - abc-xyz-rst4:9080 -
[1047:204:959d:61f8:e32a:9236:feaa:7457], 10.10.10.10:18 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/query HTTP/1.1" 401 166 - 2794 - abc-xyz-rst4:9080 -
223.189.182.67:22759 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/init HTTP/1.1" 401 166 - 5504 - abc-rst-rst4:9080 -
[1047:204:8085:95bd:90db:8817:e7b:ff3d], 10.10.10.10:35 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/init HTTP/1.1" 401 166 - 2272 - abc-xyz-rst4:9080 -
11.11.11.11:68 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/query HTTP/1.1" 401 166 - 2971 - abc-rst-rst5:9080 -
12.12.12.12:45 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/init HTTP/1.1" 401 166 - 2905 - abc-rst-rst4:9080 -

Log type:

1. 11.11.11.11:68 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/query HTTP/1.1" 401 166 - 2971 - abc-rst-rst5:9080 -

2. [1047:204:8085:95bd:90db:8817:e7b:ff3d], 10.10.10.10:35 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/init HTTP/1.1" 401 166 - 2272 - abc-xyz-rst4:9080 -

my conf file is

filter {
grok {
match => {
"message" => '%{SYSLOG5424SD:cliip}, %{URIHOST:clientip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} [%{HTTPDATE:logtime}] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:webserver} %{NOTSPACE:id5}'
}
}

if [tags] == "_grokparsefailure" {
grok {
match => {
"message" => '%{URIHOST:clientip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} [%{HTTPDATE:logtime}] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:webserver} %{NOTSPACE:id5}'
}
}

}

date {
match => [ "logtime", "dd'/'MMM'/'yyyy:HH:mm:ss ZZ" ]

	}

mutate {
convert => {
"responsecode" => "integer"
"bitstransfer" => "integer"
}
}
}

2

Logstash output

{
"tags" => [
[0] "_grokparsefailure"
],
"message" => "11.11.11.11:68 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/query HTTP/1.1" 401 166 - 2971 - abc-rst-rst5:9080 -",
"host" => "myhost",
"@timestamp" => 2018-12-31T06:19:37.372Z,
"@version" => "1",
"path" => "/my/log/filepath/logfilename"
}
{
"bitstransfer" => 166,
"@version" => "1",
"webserver" => "abc-xyz-rst4:9080",
"request" => "/my/app/path/android/init",
"path" => "/my/log/filepath/logfilename",
"logtime" => "26/Dec/2018:02:31:14 +0530",
"port" => "22211",
"cliip" => "[1047:204:8085:95bd:90db:8817:e7b:ff3d]",
"responsecode" => 200,
"user" => "-",
"role" => "-",
"id2" => "30921",
"method" => "POST",
"pass" => "-",
"message" => "[1047:204:8085:95bd:90db:8817:e7b:ff3d], 10.10.10.10:35 - - - [26/Dec/2018:02:31:14 +0530] "POST /my/app/path/android/init HTTP/1.1" 401 166 - 2272 - abc-xyz-rst4:9080 -",
"host" => "myhost",
"@timestamp" => 2018-12-25T21:29:40.000Z,
"id3" => "-",
"clientip" => "10.10.10.10:35",
"id5" => "-",
"id1" => "-",
"httpversion" => "1.1"
}

3

Could anyone help me how to read the two types of log line.

How to parse the "tags" value in IF condition. I did anything wrong?

4

I also tried another way to achieve this

grok {
match => {
"message" => '%{SYSLOG5424SD:cliip}, %{URIHOST:} %{NOTSPACE:} %{NOTSPACE:} %{NOTSPACE:} [%{HTTPDATE:}] "%{WORD:} %{DATA:} HTTP/%{NUMBER:}" %{NUMBER:} %{NUMBER:} %{NOTSPACE:id1} %{NOTSPACE:} %{NOTSPACE: %{NOTSPACE:} %{NOTSPACE:}'
}
}

if [cliip] == "[.*" {
grok {
match => {
"message" => '%{SYSLOG5424SD:cliip}, %{URIHOST:clientip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} [%{HTTPDATE:logtime}] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:webserver} %{NOTSPACE:id5}'
}
}
}

else {
grok {
match => {
"message" => '%{URIHOST:clientip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} [%{HTTPDATE:logtime}] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:webserver} %{NOTSPACE:id5}'
}
}

}

but no luck

Please guide how to pass the value with .*
my cliip field contains the value as "[1047:204:8085:95bd:90db:8817:e7b:ff3d]" I am parsing this value in IF condition as if [cliip] == "[.*"

5

try this?

if [cliip] =~ [.*

6

if you wan to use regular expression,use =~ instead of =

7

Thank you for the reply @wolfman .

I'm getting error while running the conf file. Please check

my Conf file

filter {

grok {
match =>
{
"message" => '%{SYSLOG5424SD:cliip}.....'
}
}

line 20 if [cliip] =~ ^[.*$ {

	grok {
		match => 
			{
				"message" => '%{SYSLOG5424SD:cliip},......'
			}
		}  
}

else {
grok {
match =>
{
"message" => '%{URIHOST:clientip} ........'
}
}
}

Logstash error

Sending Logstash logs to /app/install/logstash-6.4.2/logs which is now configured via log4j2.properties
[2019-01-02T11:40:28,954][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-02T11:40:29,797][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2019-01-02T11:40:31,145][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, ", ', / at line 20, column 17 (byte 470) after filter {\r\n\r\n grok {\r\n match => \r\n\t {\r\n\t\t\t"message" => '%{SYSLOG5424SD:cliip}, %{URIHOST:} %{NOTSPACE:} %{NOTSPACE:} %{NOTSPACE:} \[%{HTTPDATE:}\] "%{WORD:} %{DATA:} HTTP/%{NUMBER:}" %{NUMBER:} %{NUMBER:} %{NOTSPACE:id1} %{NOTSPACE:} %{NOTSPACE:} %{NOTSPACE:} %{NOTSPACE:}'\r\n\t\t}\r\n\t}\r\n \r\n \r\n\tif [cliip] =~ ", :backtrace=>["/app/install/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:ininitialize'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:22:in initialize'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:90:ininitialize'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline_action/create.rb:38:in execute'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/agent.rb:309:inblock in converge_state'"]}

8

my conf file

filter {

grok {
match =>
{
"message" => '%{SYSLOG5424SD:cliip}, %{URIHOST:} %{NOTSPACE:} %{NOTSPACE:} %{NOTSPACE:} [%{HTTPDATE:}] "%{WORD:} %{DATA:} HTTP/%{NUMBER:}" %{NUMBER:} %{NUMBER:} %{NOTSPACE:id1} %{NOTSPACE:} %{NOTSPACE:} %{NOTSPACE:} %{NOTSPACE:}'
}
}

if  [cliip] =~ ^\[.*$ {
	
	grok {
		match => 
			{
				"message" => '%{SYSLOG5424SD:cliip}, %{URIHOST:clientip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} \[%{HTTPDATE:logtime}\] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:webserver} %{NOTSPACE:id5}'
			}
		}  
}

else {
grok {
match =>
{
"message" => '%{URIHOST:clientip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} [%{HTTPDATE:logtime}] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:webserver} %{NOTSPACE:id5}'
}
}

}


date {
	match => [ "logtime" , "dd/MMM/yyyy:HH:mm:ss ZZ" ]
	}

mutate {
	convert => {
		"responsecode" => "integer"
		"bitstransfer" => "integer"
		}
	}

}

10

try this ???

11

@wolfman just now tried the same

getting below error
Sending Logstash logs to /app/install/logstash-6.4.2/logs which is now configured via log4j2.properties
[2019-01-02T15:27:09,806][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-01-02T15:27:10,670][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2019-01-02T15:27:11,821][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, and, or, xor, nand, { at line 25, column 175 (byte 654) after filter {\r\n\r\n grok {\r\n match => \r\n\t {\r\n\t\t\t"message" => '%{SYSLOG5424SD:cliip}, %{URIHOST} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE}\] "%{WORD} %{DATA} HTTP/%{NUMBER}" %{NUMBER} %{NUMBER} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE}'\r\n\t\t}\r\n\t}\r\n \r\n \r\n\tif [cliip] =~ /^\[.*$ {\r\n\t\t\r\n\t\tgrok {\r\n\t\t\tmatch => \r\n\t\t\t\t{\r\n\t\t\t\t\t"message" => '%{SYSLOG5424SD:cliip}, %{URIHOST:clientip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} \[%{HTTPDATE:logtime}\] "%{WORD:method} %{DATA:request} HTTP/", :backtrace=>["/app/install/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:ininitialize'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:22:in initialize'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:90:ininitialize'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline_action/create.rb:38:in execute'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/agent.rb:309:inblock in converge_state'"]}

12

line no 24{
line no 25 "message" => '%{SYSLOG5424SD:cliip}, %{URIHOST:clientip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} [%{HTTPDATE:logtime}] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:webserver} %{NOTSPACE:id5}'
line no 26 }

13

@wolfman any idea? how to achieve this

14

How about formatting your config?
I was trying to take a look at it but then I just gave up...

15

@admlko Thanks much.

Hi all,

Anyone help me how to read these two types of log lines without missing any data. Tried a lot. But unable to read all data perfectly. Some grok parsing error occurred intermittently.

16

And what have you tried, what is working and what is not working?

19

@admlko

There are two types of logline. both are getting read by the config file but however intermittently we are getting "_grokparsefailure" for the same log pattern which read earlier.

Here is my output:
{
"role" => "-",
"responsecode" => 200,
"id1" => "-",
"path" => "/my/log/file/path/access_log.2018-12-26-02_15_00",
"host" => "myhost",
"@version" => "1",
"@timestamp" => 2018-12-25T21:01:06.000Z,
"clientip" => "10.10.10.10:4883",
"bitstransfer" => 446,
"id5" => "-",
"pass" => "-",
"id3" => "-",
"webserver" => "server54",
"user" => "-",
"logtime" => "26/Dec/2018:02:31:06 +0530",
"request" => "/my/app/file/path/android/query",
"message" => "10.10.10.10:4883 - - - [26/Dec/2018:02:31:06 +0530] "POST /my/app/file/path/android/query HTTP/1.1" 200 446 - 18817 - server54 -",
"httpversion" => "1.1",
"id2" => "18817",
"method" => "POST"
}
{
"role" => "-",
"responsecode" => 200,
"id1" => "-",
"path" => "/my/log/file/path/access_log.2018-12-26-02_15_00",
"host" => "myhost",
"@version" => "1",
"@timestamp" => 2018-12-25T21:01:06.000Z,
"clientip" => "11.11.11.11:49803",
"bitstransfer" => 550,
"id5" => "-",
"pass" => "-",
"cliip" => "[2405:204:e68b:858b:69be:5181:8fc2:e61e]",
"id3" => "-",
"webserver" => "server54",
"user" => "-",
"logtime" => "26/Dec/2018:02:31:06 +0530",
"request" => "/my/app/file/path/android/query",
"message" => "[2405:204:e68b:858b:69be:5181:8fc2:e61e], 11.11.11.11:49803 - - - [26/Dec/2018:02:31:06 +0530] "POST /my/app/file/path/android/query HTTP/1.1" 200 550 - 34935 - server54 -",
"httpversion" => "1.1",
"id2" => "34935",
"method" => "POST"
}
{
"message" => "157.49.157.151:26594 - - - [26/Dec/2018:02:31:06 +0530] "POST /my/server/local/pathprofile HTTP/1.1" 204 - - 2413 - server54 -",
"tags" => [
[0] "_grokparsefailure"
],
"path" => "/my/log/file/path/access_log.2018-12-26-02_15_00",
"host" => "myhost",
"@version" => "1",
"@timestamp" => 2019-01-02T10:29:40.707Z
}

20

myConfig file

filter {

grok {
match =>
{
"message" => '%{SYSLOG5424SD:cliip}, %{URIHOST} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} [%{HTTPDATE}] "%{WORD} %{DATA} HTTP/%{NUMBER}" %{NUMBER} %{NUMBER} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE}'
}
remove_field => [ "port", "tags" ]
}

if [cliip] == "[" {
	
	grok {
		match => 
			{
				"message" => '%{SYSLOG5424SD:cliip}, %{URIHOST:clientip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} \[%{HTTPDATE:logtime}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:webserver} %{NOTSPACE:id5}'
			}
		remove_field => [ "port", "tags" ]
		}  
}

else {
grok {
match =>
{
"message" => '%{URIHOST:clientip} %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} [%{HTTPDATE:logtime}] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:bitstransfer} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:webserver} %{NOTSPACE:id5}'
}
remove_field => [ "port", "tags" ]
}

}


date {
        match => [ "logtime" , "dd/MMM/yyyy:HH:mm:ss Z" ]
        timezone => "Asia/Kolkata"
        target => "@timestamp"
        
    }
	
mutate {
	convert => {
		"responsecode" => "integer"
		"bitstransfer" => "integer"
		}
	}

}

21

" [ this kind of character need to be Escaped by \

22

I will not take a look at these before you successfully format the configs and examples :slight_smile:

23

Hi @wolfman

I tried your sugg but getting below error. What could cause this?

[2019-01-02T16:34:11,185][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"premature end of char-class: /^[./", "backtrace"=>["org/jruby/RubyRegexp.java:918:in initialize'", "(eval):138:inblock in initialize'", "org/jruby/RubyArray.java:1734:in each'", "(eval):136:inblock in initialize'", "(eval):121:in block in filter_func'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:341:infilter_batch'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:320:in worker_loop'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:286:inblock in start_workers'"], :thread=>"#<Thread:0x57cc6ad1 sleep>"}
[2019-01-02T16:34:11,374][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<RegexpError: premature end of char-class: /^[./>, :backtrace=>["org/jruby/RubyRegexp.java:918:in initialize'", "(eval):138:inblock in initialize'", "org/jruby/RubyArray.java:1734:in each'", "(eval):136:inblock in initialize'", "(eval):121:in block in filter_func'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:341:infilter_batch'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:320:in worker_loop'", "/app/install/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:286:inblock in start_workers'"]}
[2019-01-02T16:34:11,445][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit