How to define a proper grok pattern for php error logs

[Ritikapawar]

Hello,
I'm creating a connection for elasticsearch and want to upload PHP error logs by creating a script using logstash

PHP error logs format--

[16-Jan-2024 13:39:08 Asia/Calcutta] PHP Warning:  Module 'mcrypt' already loaded in Unknown on line 0
[23-Jan-2024 12:03:18 Asia/Calcutta] PHP Warning:  mysqli::__construct(): (HY000/1049): Unknown database 'labc_v2_mig_1' in /var/www/v17_test/db_connection.php on line 19
[23-Jan-2024 12:03:18 Asia/Calcutta] PHP Warning:  mysqli::query(): Couldn't fetch mysqli in /var/www/v17_test/api/functions.php on line 2180
[23-Jan-2024 12:03:18 Asia/Calcutta] PHP Fatal error:  Uncaught Error: Call to a member function fetch_array() on bool in /var/www/v17_test/api/functions.php:2181
Stack trace:
#0 /var/www/v17_test/common_top.php(8): getVersionDetailsAPI()
#1 /var/www/v17_test/login.php(21): include_once('/var/www/v17_te...')
#2 {main}
  thrown in /var/www/v17_test/api/functions.php on line 2181
[24-Jan-2024 14:54:39 Asia/Calcutta] PHP Warning:  session_destroy(): Trying to destroy uninitialized session in /var/www/commandcenter/logout.php on line 11
[24-Jan-2024 15:09:01 Asia/Calcutta] PHP Warning:  Module 'mcrypt' already loaded in Unknown on line 0
[25-Jan-2024 17:41:33 Asia/Kolkata] PHP Warning:  Use of undefined constant UAT - assumed 'UAT' (this will throw an Error in a future version of PHP) in /var/www/moh_citizen_portal/appFooterBase.php on line 12
[25-Jan-2024 22:29:26 Asia/Calcutta] PHP Warning:  Module 'mcrypt' already loaded in Unknown on line 0

These are my php error logs which i want to upload by using below script

input {
  file {
    path => "/home/plus91/Documents/php_log_DirectUpload.log"
    start_position => "beginning"
    codec => multiline {
      pattern => "^\[%{DATA:timestamp} %{TZ:timezone}\] "
      negate => true
      what => "previous"
    }
  }
}

filter {
  json {
    source => "message"
    remove_field => ["message"]
  }

  grok {
    match => { "message" => "\[%{DATA:timestamp} %{TZ:timezone}\] %{DATA:log_level}: %{GREEDYDATA:log_message}" }
  }

  date {
    match => [ "timestamp", "dd-MMM-yyyy HH:mm:ss", "dd-MMM-yyyy HH:mm:ss z" ]
    target => "@timestamp"
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    user => "elastic"
    password => "plus91"
    index => "php_error_logs_1"
    action => "index"
    document_type => "_doc"
  }
  stdout {
    codec => rubydebug
  }
}

when i run this logs.conf it does not create any index on my elasticsearch index and script takes to much time to load on my terminal
please let me know if i'm missing something.

[Rios]

input {
  file {
   path => "/home/plus91/Documents/php_log_DirectUpload.log"
   start_position => beginning
   codec => multiline
      {
         pattern => '^\[\d{2}-[a-zA-Z]{3}-\d{4} \d{2}:\d{2}:\d{2} '
         negate => true
         what => previous
		 auto_flush_interval => 1
		 multiline_tag => ""
      }
   }
} # input
filter {

 grok {
    match => { "message" => "\[%{LOGTIME:[@metadata][timestamp]} %{DATA:[@metadata][tz]}\] PHP %{DATA:log_level}:%{SPACE}%{GREEDYDATA:log_message}" }
	pattern_definitions => { "LOGTIME" => "%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME}" }
  }

  date {
    match => [ "[@metadata][timestamp]", "dd-MMM-yyyy HH:mm:ss" ]
    target => "@timestamp"
	timezone => "%{[@metadata][tz]}"
  }
  
 # mutate{   remove_field => [ "log","message", "event", "host"] } 
}

output {
    stdout { codec => rubydebug{} }
}

Result:

{
      "log_level" => "Warning",
    "log_message" => "Module 'mcrypt' already loaded in Unknown on line 0\r",
     "@timestamp" => 2024-01-16T08:09:08.000Z,
       "@version" => "1"
}
{
      "log_level" => "Fatal error",
    "log_message" => "Uncaught Error: Call to a member function fetch_array() on bool in /var/www/v17_test/api/functions.php:2181\r\nStack trace:\r\n#0 /var/www/v17_test/common_top.php(8): getVersionDetailsAPI()\r\n#1 /var/www/v17_test/login.php(21): include_once('/var/www/v17_te...')\r\n#2 {main}\r\n  thrown in /var/www/v17_test/api/functions.php on line 2181\r",
     "@timestamp" => 2024-01-23T06:33:18.000Z,
       "@version" => "1"
}

[Ritikapawar]

Hi @Rios as per your suggestion i've changed the script but it is taking a lot of time to run and not creating any index. I'm attaching a screen shot below please have look

[dadoonet (David Pilato)]

Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

[Rios]

Most likely your file had been read, and LS cannot read the same file twice.
Add sincedb_path => "/dev/null" like this:

input {
  file {
   path => "/home/plus91/Documents/php_log_DirectUpload.log"
   start_position => beginning
   sincedb_path => "/dev/null"
   codec => multiline
      {
         pattern => '^\[\d{2}-[a-zA-Z]{3}-\d{4} \d{2}:\d{2}:\d{2} '
         negate => true
         what => previous
		 auto_flush_interval => 1
		 multiline_tag => ""
      }
   }
} 

Or delete the file in sincedb_path every time, before run. The name is on 4th line from the bottom ends with the hash value.

[Ritikapawar]

Hi @Rios Script ran successfully for this type of logs
[23-Jan-2024 12:03:18 Asia/Calcutta] PHP Warning: mysqli::query(): Couldn't fetch mysqli in /var/www/v17_test/api/functions.php on line 2180.
but got one issue in this
[23-Jan-2024 12:03:18 Asia/Calcutta] PHP Fatal error: Uncaught Error: Call to a member function fetch_array() on bool in /var/www/v17_test/api/functions.php:2181 Stack trace: #0 /var/www/v17_test/common_top.php(8): getVersionDetailsAPI() #1 /var/www/v17_test/login.php(21): include_once('/var/www/v17_te...') #2 {main} thrown in /var/www/v17_test/api/functions.php on line 2181
for this blank entry has been added in index like this

{
        "_index" : "php_error_logs_1",
        "_type" : "_doc",
        "_id" : "Y9O8WY0B2AzWmE8ib4Ae",
        "_score" : 1.0,
        "_source" : {
          "path" : "/home/plus91/Documents/php_log_DirectUpload.log",
          "@timestamp" : "2024-01-30T09:38:57.596Z",
          "tags" : [
            "_grokparsefailure"
          ],
          "@version" : "1"
        }
      },

what changes are required in my script to accept all types of logs please suggest

input {
  file {
    path => "/home/plus91/Documents/php_log_DirectUpload.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => '^\[\d{2}-[a-zA-Z]{3}-\d{4} \d{2}:\d{2}:\d{2} '
      negate => true
      what => "previous"
      auto_flush_interval => 1
      multiline_tag => ""
    }
  }
}

filter {
  grok {
    match => { "message" => "\[%{LOGTIME:[@metadata][timestamp]} %{DATA:[@metadata][tz]}\] PHP %{WORD:log_level}:%{SPACE}%{GREEDYDATA:log_message}" }
    pattern_definitions => { "LOGTIME" => "%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME}" }
  }

  date {
    match => [ "[@metadata][timestamp]", "dd-MMM-yyyy HH:mm:ss" ]
    target => "@timestamp"
    timezone => "%{[@metadata][tz]}"
  }

  # You can remove unnecessary fields if needed
  mutate {
    remove_field => ["log", "message", "event", "host"]
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    user => "elastic"
    password => "plus91"
    index => "php_error_logs_1"
    action => "index"
    doc_as_upsert => true
  }

  stdout {
    codec => rubydebug
  }
}

[Rios]

@Ritikapawar

%{WORD:log_level}

You cannot use WORD which is \b\w+\b, because "Fatal error". You should use %{DATA:log_level}, it works for me.

[Ritikapawar]

Data added into index but showing as

[Rios]

And what is a problem? You will have 3 fields
"log_level" => "Fatal error",
"log_message" => "bla bla",
"@timestamp" => 2024-01-23T06:33:18.000Z,