# Log stash with file-beat log file

**URL:** <https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807>\
**Category:** Logstash\
**Created:** [March 18, 2019, 3:24pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807 "2019-03-18T15:24:09Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![nagr](https://avatars.discourse-cdn.com/v4/letter/n/aeb1de/32.png) [@nagr](https://discuss.elastic.co/u/nagr)\
**Post date:** [March 18, 2019, 3:24pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/1 "2019-03-18T15:24:10Z")

</div>

Hi,

I am trying to load the log file with some filter condition using filebeat plugin, for example please is the raw log file content entry  
"2016-01-13 01:39:34, Info DPX Started DPX phase: Inventory  
2016-01-13 01:39:36, Info DPX Ended DPX phase: Inventory  
2016-01-13 01:39:36, Info DPX CJob::Resume completed with status: 0x8000000a  
2016-01-13 01:39:36, Info DPX Started DPX phase: Apply Deltas Provided In File  
2016-01-13 01:39:36, Info DPX Ended DPX phase: Apply Deltas Provided In File  
2016-01-13 01:39:36, Info DPX CJob::Resume completed with status: 0x0  
2016-01-13 01:39:37, Info DPX CreateFileW failed, FileName:\?\C:\Windows\SoftwareDistribution\Download\172ca809639fa1a7f503a9d88144506e$dpx$.tmp\job.xml, Error:0x80070002"

I just want the content need to be loaded over logstash " **DPX CJob::Resume completed with status: 0x0**" from that log entry file,

so I have created one "conf" file and add those log file path in "filebeat.yml" file,

below is the "conf" file which i was created, but i am not sure whether is that right (or) wrong, even didn't get the filter content data in kibana,

input {  
beats {  
port =\> "5044"  
}  
}

filter {  
grok { #parses the common  
bits match =\> [" **DPX CJob::Resume completed with status: 0x0**"] }  
}  
output {  
elasticsearch {  
host =\> ["aaaaaaaa:bbbb"]  
user =\> "yyyyyy"  
password =\> "xxxxxxxx"  
index =\> "winlog\_elk"  
document\_type =\> "winlog\_elk"  
}  
stdout { }  
}

Please give your answer to get the right config file,

Thanks,  
Nagaraj,

---

<div class="post-metadata">

**Author:** ![pup\_seba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pup_seba/32/42988_2.png) [@pup\_seba](https://discuss.elastic.co/u/pup_seba)\
**Post date:** [March 19, 2019, 9:18pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/2 "2019-03-19T21:18:45Z")

</div>

Hi,

You could use this filter instead:  
match =\> { 'message' =\> "(?DPX CJob::Resume completed with status: 0x0)" }

Or (imho a better option), this other:  
match =\> { 'message' =\> "^(?\d{4}-\d{2}-\d{2} %{TIME}),\s%{LOGLEVEL:loglevel} (?DPX CJob::Resume completed with status: 0x0)$" }

Also, in the output side, you are using a depricated option (document\_type). Maybe you don't need to use that and the default "doc" is just ok. As you are using a particular index anyway, removing this seems like a good option to me.

Also, your "index" option in your elasticsearch output, is using only 1 index, I think it is better to have different indexes, one per day, and the way I know to do this is to do something like this:  
index =\> "winlog\_elk-%{+YYYY.MM.dd}"

These 2 last changes, would render your output like this:

```
 output {
   elasticsearch {
     host => ["aaaaaaaa:bbbb"]
     user => "yyyyyy"
     password => "xxxxxxxx"
     index => "winlog_elk-%{+YYYY.MM.dd}"

```

So, one possible conf file could be:

```
input {
  beats {
    port => "5044"
  }
}

filter {
  grok {
    match => { 'message' => "(?<entry>DPX CJob::Resume completed with status: 0x0)" }

output {
  elasticsearch {
    host => ["aaaaaaaa:bbbb"]
    user => "yyyyyy"
    password => "xxxxxxxx"
    index => "winlog_elk-%{+YYYY.MM.dd}"
```

---

<div class="post-metadata">

**Author:** ![nagr](https://avatars.discourse-cdn.com/v4/letter/n/aeb1de/32.png) [@nagr](https://discuss.elastic.co/u/nagr)\
**Post date:** [March 20, 2019, 10:00am UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/3 "2019-03-20T10:00:36Z")

</div>

Thanks @pup_seba

If I want to add condition like the file source come from this location mean then the filter should work, so for this can I add the below line for source path,

filter {  
**if[source]=~"C:\xxx\yyyyy\testlog.log" {**  
grok {  
match =\> { 'message' =\> "(?\<entry\>DPX CJob::Resume completed with status: 0x0)" }  
}  
**}**  
}

Thanks,  
Nagaraj,

---

<div class="post-metadata">

**Author:** ![nagr](https://avatars.discourse-cdn.com/v4/letter/n/aeb1de/32.png) [@nagr](https://discuss.elastic.co/u/nagr)\
**Post date:** [March 22, 2019, 12:12pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/4 "2019-03-22T12:12:31Z")

</div>

Hi @pup_seba

Shall I add the filter line like below ?

match =\> { 'message' =\> "(?\<entry\>DPX CJob::Resume completed with status: 0x0)" }

or

match =\> { 'message' =\> "(?\<?\>DPX CJob::Resume completed with status: 0x0)" }

I am just added the snip below, if I run this grok pattern getting error like "pattern is not correct"

 ![ELK](https://us1.discourse-cdn.com/elastic/original/3X/6/f/6f1beec705d39ee78577f2d120bbd688effdadaa.png)

Thanks,  
Nagaraj,

---

<div class="post-metadata">

**Author:** ![pup\_seba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pup_seba/32/42988_2.png) [@pup\_seba](https://discuss.elastic.co/u/pup_seba)\
**Post date:** [March 22, 2019, 2:01pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/5 "2019-03-22T14:01:38Z")

</div>

Hi again mate!

The name between "\<\>" could be what you want it to be. This format "(?expression) is just a way to create a field with name "label" (or whatever you want), with the value that matches the expression. I would reccomend you to take a look at thishttps://github.com/kkos/oniguruma/blob/master/doc/RE

[![Imgur](https://i.imgur.com/yLEZQBO.png?fb "Imgur") ](https://imgur.com/yLEZQBO)

That "match" thing, will try to match line per line. In that grok debugger, you should only use that line.

---

<div class="post-metadata">

**Author:** ![nagr](https://avatars.discourse-cdn.com/v4/letter/n/aeb1de/32.png) [@nagr](https://discuss.elastic.co/u/nagr)\
**Post date:** [March 22, 2019, 2:48pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/6 "2019-03-22T14:48:51Z")

</div>

I am new to this so still can't able to get the result when I am trying in grok tool,

I was tried your message only like below

![ELK](https://us1.discourse-cdn.com/elastic/original/3X/d/8/d89edc430191a83238413df5a065147b49011630.png)

but still getting error like "pattern not match"

did we need to give any custom patterns ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 22, 2019, 2:56pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/7 "2019-03-22T14:56:46Z")

</div>

Is that a tab between DPX and CJob?

---

<div class="post-metadata">

**Author:** ![nagr](https://avatars.discourse-cdn.com/v4/letter/n/aeb1de/32.png) [@nagr](https://discuss.elastic.co/u/nagr)\
**Post date:** [March 22, 2019, 4:24pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/8 "2019-03-22T16:24:03Z")

</div>

it is only "single space"

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 22, 2019, 4:28pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/9 "2019-03-22T16:28:49Z")

</div>

Your sample data clearly has more than one space, so I would not expect that to match. Try changing it to

```
match => { 'message' => "(?<entry>DPX\sCJob::Resume completed with status: 0x0)" }
```

---

<div class="post-metadata">

**Author:** ![nagr](https://avatars.discourse-cdn.com/v4/letter/n/aeb1de/32.png) [@nagr](https://discuss.elastic.co/u/nagr)\
**Post date:** [March 22, 2019, 4:38pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/10 "2019-03-22T16:38:05Z")

</div>

excuse me that was 4 space there in original log file

---

<div class="post-metadata">

**Author:** ![pup\_seba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pup_seba/32/42988_2.png) [@pup\_seba](https://discuss.elastic.co/u/pup_seba)\
**Post date:** [March 22, 2019, 5:23pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/11 "2019-03-22T17:23:10Z")

</div>

Hi,

I just copied/pasted the strings you gave, so I created the filter by the exact same amount of spaces as in the first example you provided. I guess we could make it so it does no matter how many whitespaces chars there are, if you go with something like this :  
`(?<entry>DPX\s*CJob::Resume\s*completed\s*with\s*status:\s*0x0)`

[![Imgur](https://i.imgur.com/N8kjFJJ.png?fb "Imgur") ](https://imgur.com/N8kjFJJ)

As you can see in this example, I changed your original sample and added some extra spaces and tabs in between words...still have a match with this new filter. I'm not really fun of using this form, I think it would be better to try to get the exact same amount and kind of whitespace characters.

I'm also new to this (I've been autolearning for about a month), and trying to help and collaborate with communities is a way of learning for me, so, I'm glad to help. Try to take a look at this, [https://github.com/kkos/oniguruma/blob/master/doc/RE](https://github.com/kkos/oniguruma/blob/master/doc/RE)  
It really is a good guide to understand how these regex work. As you can see the filter you need is really easy, you just write the words you need, and instead of spaces you write "\s\*" between the words. Then, you only need to put that inside and extended group (look at the link I gave you), that has this form "(?\<your\_label\>your\_match)". Nothing fancy, but really easy and effective.

---

<div class="post-metadata">

**Author:** ![nagr](https://avatars.discourse-cdn.com/v4/letter/n/aeb1de/32.png) [@nagr](https://discuss.elastic.co/u/nagr)\
**Post date:** [March 25, 2019, 2:37pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/12 "2019-03-25T14:37:31Z")

</div>

it is strange for me, I am not getting the results even If I tried very basic content also,

below is the content:

**agent service started with exit 0**  
**execution failed to start service**  
**agent service started with exit 0**  
**failed to start service for getting error with wim**

my grok pattern --\> match =\> { 'message' =\> "(?agent service started with exit 0)" }

but the result showing like --\> No Matches

I am trying on grok debugger tool --\> [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/)  
I don't know where I am exactly standing ?

Thanks,  
Nagaraj,

---

<div class="post-metadata">

**Author:** ![nagr](https://avatars.discourse-cdn.com/v4/letter/n/aeb1de/32.png) [@nagr](https://discuss.elastic.co/u/nagr)\
**Post date:** [March 26, 2019, 2:54pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/13 "2019-03-26T14:54:21Z")

</div>

any help on this please

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 23, 2019, 2:54pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807/14 "2019-04-23T14:54:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.