3 grok filter in same file

Elastic Stack Logstash
1

Hi,
I have installed filebeat on a windows server to send logs of filezilla server to logstash.
At this moment, it works, I see my logs.
But it seems, only one filter works.

Here my configuration

   grok {
     match => { "message" => "\(%{GREEDYDATA:id_filezilla}\) %{DATESTAMP:date_filezilla} - %{GREEDYDATA:compte_filezilla} \(%{IPV4:adresseip_filezilla}\)> %{GREEDYDATA:action_filezilla}" }
     remove_field => "message"
    }

    grok {
     match => { "message" => "\(%{GREEDYDATA:id_filezilla}\) %{DATESTAMP:date_filezilla} - \(%{GREEDYDATA:compte_filezilla}\) \(%{IPV4:adresseip_filezilla}\)> %{GREEDYDATA:action_filezilla}" }
     remove_field => "message"
    }

    grok {
     match => { "message" => "\(%{GREEDYDATA:id_filezilla}\) %{DATESTAMP:date_filezilla} - %{GREEDYDATA:compte_filezilla} %{IPV6:adresseip_filezilla}> %{GREEDYDATA:action_filezilla}" }
     remove_field => "message"
    }

The first groks works, I have my fields with information.

The 2 others, they are no working. Specially the third, it's the same as the first but for ipv6 address.

Is my config fine ?

Or how i can improve it to have my 3 filters ?

thank you for your help.

2

Comment out the remove_field lines and show us an example of an event that wasn't processed correctly. Use a stdout { codec => rubydebug } output to dump the raw event.

3

Thnak you for your reply !

My output is the following :

output {
elasticsearch {
hosts => "server:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}

Can i add the output you give me with mine ?
And how i can read the stdout ouput ? I don't know to see this with command line.

I see all my results in kibana, it's here where I have seen my results with the tag "_grokparsefailure"

4

Can i add the output you give me with mine ?

Yes.

I see all my results in kibana, it's here where I have seen my results with the tag "_grokparsefailure"

You can copy/paste the raw event from Kibana's JSON tab, that's fine too.

5

Thank you.

I have added the new output.

Here the results :

it's OK with the following (first grok filter for an ipv4 address :
I have my fields

{
"_index": "filebeat-6.3.0-2018.08.21",
"_type": "doc",
"_id": "CAbdW2UB-4GQqcKpWxkJ",
"_version": 1,
"_score": null,
"_source": {
"message": "(011418) 21/08/2018 11:44:07 - account_fz (IP_SERVER)> RETR BL_2951519.PCL",
"host": {
"name": "SERVERNAME"
},
"prospector": {
"type": "log"
},
"input": {
"type": "log"
},
"offset": 2365486,
"@timestamp": "2018-08-21T09:44:07.938Z",
"@version": "1",
"id_filezilla": "011418",
"date_filezilla": "21/08/2018 11:44:07",
"source": "C:\Program Files (x86)\FileZilla Server\Logs\fzs-2018-08-21.log",
"tags": [
"beats_input_codec_plain_applied",
"_grokparsefailure"
],
"compte_filezilla": "account_fz",
"beat": {
"version": "6.3.0",
"name": "SERVERNAME",
"hostname": "SERVERNAME"
},
"action_filezilla": "RETR BL_2951519.PCL",
"adresseip_filezilla": "172.21.5.28"
},
"fields": {
"@timestamp": [
"2018-08-21T09:44:07.938Z"
]
},
"highlight": {
"adresseip_filezilla": [
"@kibana-highlighted-field@172.21.5.28@/kibana-highlighted-field@"
],
"action_filezilla": [
"@kibana-highlighted-field@RETR BL_2951519.PCL@/kibana-highlighted-field@"
],
"compte_filezilla": [
"@kibana-highlighted-field@ftp_penta_esker@/kibana-highlighted-field@"
]
},
"sort": [
1534844647938
]
}

For the following, it's not ok, I don't have my fields, because, it's a ipv6 address (::1, localhost address) :
It seems my grok filter for this don't work.

{
"_index": "filebeat-6.3.0-2018.08.20",
"_type": "doc",
"_id": "k8i0WGUB-4GQqcKpgfeC",
"_version": 1,
"_score": null,
"_source": {
"tags": [
"beats_input_codec_plain_applied",
"_grokparsefailure"
],
"message": "(009668) 20/08/2018 21:00:29 - account_fz (::1)> STOR TRF_CCOLLECT.004900",
"offset": 5923185,
"input": {
"type": "log"
},
"host": {
"name": "SERVERNAME"
},
"@version": "1",
"source": "C:\Program Files (x86)\FileZilla Server\Logs\fzs-2018-08-20.log",
"prospector": {
"type": "log"
},
"beat": {
"name": "SERVERNAME",
"hostname": "SERVERNAME",
"version": "6.3.0"
},
"@timestamp": "2018-08-20T19:00:38.913Z"
},
"fields": {
"@timestamp": [
"2018-08-20T19:00:38.913Z"
]
},
"highlight": {
"message": [
"(009668) 20/08/2018 21:00:29 - account_fz (::1)> STOR @kibana-highlighted-field@TRF_CCOLLECT@/kibana-highlighted-field@.004900"
]
},
"sort": [
1534791638913
]
}

So, what's wrong with my filters ?

I have tested with https://grokdebug.herokuapp.com/, it's OK.

Do I have to put my grok filter for ipv6 in another file ?

6

That's right, there's something wrong with the grok filter. Debug it by reducing it to the smallest possible expression and make sure that works, then continue building the expression until it breaks.

It might be unrelated in this case, but you're using GREEDYDATA excessively. There are very few cases where it makes sense to have more than one DATA or GREEDYDATA pattern in the same grok expression.

1 Like
7

Thank you for the idea !

I have found a solution.
The filter for ipv4 and ipv6 address is now combined in 1 filter.

filter {
grok {
match => { "message" => "(%{GREEDYDATA:id_filezilla}) %{DATESTAMP:date_filezilla} - %{GREEDYDATA:compte_filezilla} (%{IP:adresseip_filezilla})> %{GREEDYDATA:action_filezilla}" }
}
}

It's ok with ipv4 and ipv6 addresse :slight_smile:

If I put just this filter, it's OK, no error like "grokparsefailure".
But if I add the other filter to not take the "(" character, I hace the error "grokparsefailure"/

So, How I can filter the information with the "(" character ?
Sometimes, it appears, sometime no.

8

(\()?matches ( zero or one times, i.e. it an optional left parenthesis.

9

I have done this :

\(%{NUMBER:id_filezilla}\) %{DATESTAMP:date_filezilla} - ?(\()%{GREEDYDATA:compte_filezilla}?(\)) \(%{IP:adresseip_filezilla}\)> %{GREEDYDATA:action_filezilla}

With this one filter, I have my 3 filters (one for ipv4, one for ipv6, one for the parenthesis

One last question
you give this : (\()?

when I have test on grok debugger, it works only for left parenthesis, if I change for right parenthesis, it don't work.

What's exactly the differrence between (\()? and ?(\()

thank you for your answers, you help me a a lot !

10

when I have test on grok debugger, it works only for left parenthesis, if I change for right parenthesis, it don't work.

Use (\))?.

11

it's strange

(\))? does'nt work, I always see the right parenthesis when I have a information with this caracter.

No error grokparsefailure in kibana.

12

What do you mean by "see the right parenthesis"?

13

Hi,
I have this information sent by filebeat :

(011948) 21/08/2018 16:06:20 - (not logged in) (192.168.1.1)> USER ftp_user

I would like to escape the 2 parenthesis before and after "not logged in" but the parenthesis dont' appears all the sime.

my filter is at this moment the following (and it works only for the left parenthesis)
grok { match => { "message" => "\(%{NUMBER:id_filezilla}\) %{DATESTAMP:date_filezilla} - (\()?%{GREEDYDATA:compte_filezilla}(\))? \(%{IP:adresseip_filezilla}\)> %{GREEDYDATA:action_filezilla}" } }

I have this part (\()?%{GREEDYDATA:compte_filezilla}(\))?

so, for the left parenthesis, it works, it always escape when the filter find a left parenthesis

but for the right parenthesis at the end, it's not working and I don't understand.

14

The problem is probably that you're using GREEDYDATA. Can you give an example of a message without parentheses?

15

here an example

(011982) 21/08/2018 16:23:31 - ftp_user (172.16.16.20)> 226 Successfully transferred "/Folder2/"

Actually, when a user is connecter, we don't have parentheses. Before the user is connected, we have

(not logged in)

16

I suggest you match that by looking for a parenthesized string (\((?<fieldname>[^)]*)\)) or a regular space-delimited string (%{NOTSPACE:fieldname}):

(\((?<fieldname>[^)]*)\)|%{NOTSPACE:fieldname})
17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.