Log stash with file-beat log file

Elastic Stack Logstash
1

Hi,

I am trying to load the log file with some filter condition using filebeat plugin, for example please is the raw log file content entry
"2016-01-13 01:39:34, Info DPX Started DPX phase: Inventory
2016-01-13 01:39:36, Info DPX Ended DPX phase: Inventory
2016-01-13 01:39:36, Info DPX CJob::Resume completed with status: 0x8000000a
2016-01-13 01:39:36, Info DPX Started DPX phase: Apply Deltas Provided In File
2016-01-13 01:39:36, Info DPX Ended DPX phase: Apply Deltas Provided In File
2016-01-13 01:39:36, Info DPX CJob::Resume completed with status: 0x0
2016-01-13 01:39:37, Info DPX CreateFileW failed, FileName:\?\C:\Windows\SoftwareDistribution\Download\172ca809639fa1a7f503a9d88144506e$dpx$.tmp\job.xml, Error:0x80070002"

I just want the content need to be loaded over logstash "DPX CJob::Resume completed with status: 0x0" from that log entry file,

so I have created one "conf" file and add those log file path in "filebeat.yml" file,

below is the "conf" file which i was created, but i am not sure whether is that right (or) wrong, even didn't get the filter content data in kibana,

input {
beats {
port => "5044"
}
}

filter {
grok { #parses the common
bits match => [ "DPX CJob::Resume completed with status: 0x0" ] }
}
output {
elasticsearch {
host => [ "aaaaaaaa:bbbb" ]
user => "yyyyyy"
password => "xxxxxxxx"
index => "winlog_elk"
document_type => "winlog_elk"
}
stdout { }
}

Please give your answer to get the right config file,

Thanks,
Nagaraj,

2

Hi,

You could use this filter instead:
match => { 'message' => "(?DPX CJob::Resume completed with status: 0x0)" }

Or (imho a better option), this other:
match => { 'message' => "^(?\d{4}-\d{2}-\d{2} %{TIME}),\s%{LOGLEVEL:loglevel} (?DPX CJob::Resume completed with status: 0x0)$" }

Also, in the output side, you are using a depricated option (document_type). Maybe you don't need to use that and the default "doc" is just ok. As you are using a particular index anyway, removing this seems like a good option to me.

Also, your "index" option in your elasticsearch output, is using only 1 index, I think it is better to have different indexes, one per day, and the way I know to do this is to do something like this:
index => "winlog_elk-%{+YYYY.MM.dd}"

These 2 last changes, would render your output like this:

 output {
   elasticsearch {
     host => [ "aaaaaaaa:bbbb" ]
     user => "yyyyyy"
     password => "xxxxxxxx"
     index => "winlog_elk-%{+YYYY.MM.dd}"

So, one possible conf file could be:

input {
  beats {
    port => "5044"
  }
}

filter {
  grok {
    match => { 'message' => "(?<entry>DPX CJob::Resume completed with status: 0x0)" }

output {
  elasticsearch {
    host => [ "aaaaaaaa:bbbb" ]
    user => "yyyyyy"
    password => "xxxxxxxx"
    index => "winlog_elk-%{+YYYY.MM.dd}"
3

Thanks @pup_seba

If I want to add condition like the file source come from this location mean then the filter should work, so for this can I add the below line for source path,

filter {
if[source]=~"C:\xxx\yyyyy\testlog.log" {
grok {
match => { 'message' => "(?<entry>DPX CJob::Resume completed with status: 0x0)" }
}
}
}

Thanks,
Nagaraj,

4

Hi @pup_seba

Shall I add the filter line like below ?

match => { 'message' => "(?<entry>DPX CJob::Resume completed with status: 0x0)" }

or

match => { 'message' => "(?<?>DPX CJob::Resume completed with status: 0x0)" }

I am just added the snip below, if I run this grok pattern getting error like "pattern is not correct"

ELK
ELK.png1410×346 22.4 KB

Thanks,
Nagaraj,

5

Hi again mate!

The name between "<>" could be what you want it to be. This format "(?expression) is just a way to create a field with name "label" (or whatever you want), with the value that matches the expression. I would reccomend you to take a look at thishttps://github.com/kkos/oniguruma/blob/master/doc/RE

Imgur

That "match" thing, will try to match line per line. In that grok debugger, you should only use that line.

6

I am new to this so still can't able to get the result when I am trying in grok tool,

I was tried your message only like below

ELK

but still getting error like "pattern not match"

did we need to give any custom patterns ?

7

Is that a tab between DPX and CJob?

8

it is only "single space"

9

Your sample data clearly has more than one space, so I would not expect that to match. Try changing it to

match => { 'message' => "(?<entry>DPX\sCJob::Resume completed with status: 0x0)" }
10

excuse me that was 4 space there in original log file

11

Hi,

I just copied/pasted the strings you gave, so I created the filter by the exact same amount of spaces as in the first example you provided. I guess we could make it so it does no matter how many whitespaces chars there are, if you go with something like this :
(?<entry>DPX\s*CJob::Resume\s*completed\s*with\s*status:\s*0x0)

Imgur

As you can see in this example, I changed your original sample and added some extra spaces and tabs in between words...still have a match with this new filter. I'm not really fun of using this form, I think it would be better to try to get the exact same amount and kind of whitespace characters.

I'm also new to this (I've been autolearning for about a month), and trying to help and collaborate with communities is a way of learning for me, so, I'm glad to help. Try to take a look at this, https://github.com/kkos/oniguruma/blob/master/doc/RE
It really is a good guide to understand how these regex work. As you can see the filter you need is really easy, you just write the words you need, and instead of spaces you write "\s*" between the words. Then, you only need to put that inside and extended group (look at the link I gave you), that has this form "(?<your_label>your_match)". Nothing fancy, but really easy and effective.

1 Like
12

it is strange for me, I am not getting the results even If I tried very basic content also,

below is the content:

agent service started with exit 0
execution failed to start service
agent service started with exit 0
failed to start service for getting error with wim

my grok pattern --> match => { 'message' => "(?agent service started with exit 0)" }

but the result showing like --> No Matches

I am trying on grok debugger tool --> https://grokdebug.herokuapp.com/
I don't know where I am exactly standing ?

Thanks,
Nagaraj,

13

any help on this please

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.