Turn messages into fields - querying in kibana

Elastic Stack Logstash
#1

Hello,
I am trying to analyze some access and report logs in kibana..I want to be able to query on the message I receive on kibana...Basically I want to turn the messages onto fields to be able to query on them. How can I do that in kibana? modify the existing log-stash file?

The contents of access log looks like the following:

access.log
00:42:10,411 INFO [ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' access:14 - |peka|searchCustomer|https://32.90.12:253/service/soap/vhandset/CustomerManagement/CustomerLookupService

report.log
and the report logs like the following:
2016-03-22 00:4110|mayan|searchVendor|2509

The configuration for the above files is the following:

logstash-shipper.conf
input {
file {
path => [ "/path/to/logs/report.log", "/path/to/logs/access.log" ]
type => "syslog"
start_position => "beginning"
sincedb_path => "/dev/null"
ignore_older => 0
}
}

output {
stdout { codec => rubydebug }
redis { host => "13.22.10.141" data_type => "list" key => "newlog" }
}

#2

I tried the following in logstash config file...but I am getting a grokparsefailure...

filter {
grok {
match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
}

#3

I also tried to the following for the log:
2016-03-22 00:4110|mayan|searchVendor|2509

filter {
grok {
match => [ "message", "%{DATE:todaysdate} %{SECOND:seconds} %{WORD:Username} %{WORD:Vendorname} % {BASE16NUM:bytes}" ]
}
}

#4

match => [ "message", "%{DATE:todaysdate} %{SECOND:seconds} %{WORD:Username} %{WORD:Vendorname} % {BASE16NUM:bytes}" ]

This expression clearly doesn't match the log message. For example, you're ignoring the "|" characters.

Perhaps http://grokdebug.herokuapp.com/ or http://grokconstructor.appspot.com/ can help you build the expression.

As a general comment, you probably don't want to assign these files the type "syslog" since I assume they're not syslog messages. Secondly, since the log messages look very differently I suggest you assign different types to them. Use two file inputs instead of one.

#5

Hi,
Thanks for your reply.
How do i take a | character into account?
Also, what do you mean when you say to use two file inputs?

#6

So I did the following: it's still giving me grokparsewrror:

filter {
grok {
match => [ "message", "%{DATE:todaysdate} %{SPACE} %{SECOND:seconds} %{NOTSPACE} %{WORD:Username} %{NOTSPACE} %{WORD:Vendorname} %{NOTSPACE} %{BASE16NUM:bytes}" ]
}
}

#7

I fixed it! it works!

#8

kibanaerrr.PNG548×569 18.4 KB

Why is there a question mark next to the fields I extracted from the logs shown in Kibana ? Does it indicate it has not been properly extracted? (username, vendorname, bytes, seconds, todaysdate)

#9