Mutate filter specific word

Elastic Stack Logstash
1

Hi guys,

I'm receiving information from specific webservers.
I wish to filter out lines with errors from my log files.

To start I want to add a field for PHP fatal errors.
Right now, I have this:

input {
        beats {
                port => 5044
        }
}

filter {
        grok {
                match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
        date {
                match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
        }
        if "PHP Fatal error" in [message] {
                mutate { add_field => { "fatal" => "%{[message]}" } } 
        }
}

output {
        elasticsearch { hosts => ["localhost:9200"] }
        stdout { codec => rubydebug }
}

Unfortunately I don't see the field.
How can I create a field for my fatal errors?

2

@Jeffreyshoptrader It would really help if you can attach few lines (10-20) of sample log that you are trying to process.

3

Hi @tamilsweet

Here are some example lines:

[Mon Mar 30 06:31:52.306901 2020] [proxy_fcgi:error] [pid 31437:tid 139917183072000] [client 66.249.66.33:55419] AH01071: Got error 'PHP message: PHP Fatal error: Cannot declare class shoppingCart, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/shopping_cart.php on line 13PHP message: PHP Fatal error: Cannot declare class phpmailerException, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/PHPMailer/class.phpmailer.php on line 3454'
[Mon Mar 30 06:31:52.811663 2020] [proxy_fcgi:error] [pid 31437:tid 139917250213632] [client 66.249.66.33:58258] AH01071: Got error 'PHP message: PHP Fatal error: Cannot declare class shoppingCart, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/shopping_cart.php on line 13PHP message: PHP Fatal error: Cannot declare class phpmailerException, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/PHPMailer/class.phpmailer.php on line 3454'
[Mon Mar 30 06:32:05.306637 2020] [proxy_fcgi:error] [pid 30657:tid 139917225035520] [client 66.249.66.39:44078] AH01071: Got error 'PHP message: PHP Fatal error: Cannot declare class shoppingCart, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/shopping_cart.php on line 13PHP message: PHP Fatal error: Cannot declare class phpmailerException, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/PHPMailer/class.phpmailer.php on line 3454'
[Mon Mar 30 06:32:05.506206 2020] [proxy_fcgi:error] [pid 31437:tid 139917317355264] [client 66.249.66.36:61171] AH01071: Got error 'PHP message: PHP Fatal error: Cannot declare class shoppingCart, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/shopping_cart.php on line 13PHP message: PHP Fatal error: Cannot declare class phpmailerException, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/PHPMailer/class.phpmailer.php on line 3454'
[Mon Mar 30 06:32:05.889129 2020] [proxy_fcgi:error] [pid 30659:tid 139917208250112] [client 66.249.66.33:59998] AH01071: Got error 'PHP message: PHP Fatal error: Cannot declare class shoppingCart, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/shopping_cart.php on line 13PHP message: PHP Fatal error: Cannot declare class phpmailerException, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/PHPMailer/class.phpmailer.php on line 3454'
[Mon Mar 30 06:32:06.089885 2020] [proxy_fcgi:error] [pid 31437:tid 139917444241152] [client 66.249.66.33:55071] AH01071: Got error 'PHP message: PHP Fatal error: Cannot declare class shoppingCart, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/shopping_cart.php on line 13PHP message: PHP Fatal error: Cannot declare class phpmailerException, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/PHPMailer/class.phpmailer.php on line 3454'
[Mon Mar 30 06:32:06.565154 2020] [proxy_fcgi:error] [pid 31437:tid 139917427455744] [client 66.249.66.36:62776] AH01071: Got error 'PHP message: PHP Fatal error: Cannot declare class shoppingCart, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/shopping_cart.php on line 13PHP message: PHP Fatal error: Cannot declare class phpmailerException, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/PHPMailer/class.phpmailer.php on line 3454'
[Mon Mar 30 06:32:06.739932 2020] [proxy_fcgi:error] [pid 30659:tid 139917199857408] [client 66.249.66.36:37391] AH01071: Got error 'PHP message: PHP Fatal error: Cannot declare class shoppingCart, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/shopping_cart.php on line 13PHP message: PHP Fatal error: Cannot declare class phpmailerException, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/PHPMailer/class.phpmailer.php on line 3454'
[Mon Mar 30 06:32:07.157340 2020] [proxy_fcgi:error] [pid 30658:tid 139917342533376] [client 66.249.66.39:64431] AH01071: Got error 'PHP message: PHP Fatal error: Cannot declare class shoppingCart, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/shopping_cart.php on line 13PHP message: PHP Fatal error: Cannot declare class phpmailerException, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/PHPMailer/class.phpmailer.php on line 3454'
[Mon Mar 30 06:32:07.351127 2020] [proxy_fcgi:error] [pid 30658:tid 139917334140672] [client 66.249.66.33:60666] AH01071: Got error 'PHP message: PHP Fatal error: Cannot declare class shoppingCart, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/shopping_cart.php on line 13PHP message: PHP Fatal error: Cannot declare class phpmailerException, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/PHPMailer/class.phpmailer.php on line 3454'
[Mon Mar 30 06:32:07.754167 2020] [proxy_fcgi:error] [pid 30659:tid 139917191464704] [client 66.249.66.33:37407] AH01071: Got error 'PHP message: PHP Fatal error: Cannot declare class shoppingCart, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/shopping_cart.php on line 13PHP message: PHP Fatal error: Cannot declare class phpmailerException, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/PHPMailer/class.phpmailer.php on line 3454'

4

you will need to write regex to capture the PHP Fatal Error. “in” operator will not work

5

I'm not really sure on how to do that.
I have 2 different log files right now.

Is it possible to just make 2 if statements to get the information I need?
I have one error log and one access log.

6

You can use %{HTTPD24_ERRORLOG} to match your error log. It will result in data like

{
  "module": "proxy_fcgi",
  "pid": "31437",
  "message": "Got error 'PHP message: PHP Fatal error: Cannot declare class shoppingCart, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/shopping_cart.php on line 13PHP message: PHP Fatal error: Cannot declare class phpmailerException, because the name is already in use in /var/www/vhosts/schaduwdoekkeuze.nl/httpdocs/bernini/includes/classes/PHPMailer/class.phpmailer.php on line 3454'",
  "tid": "139917250213632",
  "clientport": "58258",
  "loglevel": "error",
  "client": "66.249.66.33",
  "errorcode": "AH01071",
  "timestamp": "Mon Mar 30 06:31:52.811663 2020"
}

Can you try this grok pattern match ?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.