# How to define a proper grok pattern for php error logs
**URL:** https://discuss.elastic.co/t/how-to-define-a-proper-grok-pattern-for-php-error-logs/351978
**Category:** Logstash
**Tags:** elastic-stack-monitoring
**Created:** [January 29, 2024, 10:18am UTC](https://discuss.elastic.co/t/how-to-define-a-proper-grok-pattern-for-php-error-logs/351978 "2024-01-29T10:18:34Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Ritikapawar](https://discuss.elastic.co/u/Ritikapawar)
#### Post date: [January 29, 2024, 10:18am UTC](https://discuss.elastic.co/t/how-to-define-a-proper-grok-pattern-for-php-error-logs/351978/1 "2024-01-29T10:18:34Z")
Hello,
I'm creating a connection for elasticsearch and want to upload PHP error logs by creating a script using logstash
PHP error logs format--
```auto
[16-Jan-2024 13:39:08 Asia/Calcutta] PHP Warning: Module 'mcrypt' already loaded in Unknown on line 0
[23-Jan-2024 12:03:18 Asia/Calcutta] PHP Warning: mysqli::__construct(): (HY000/1049): Unknown database 'labc_v2_mig_1' in /var/www/v17_test/db_connection.php on line 19
[23-Jan-2024 12:03:18 Asia/Calcutta] PHP Warning: mysqli::query(): Couldn't fetch mysqli in /var/www/v17_test/api/functions.php on line 2180
[23-Jan-2024 12:03:18 Asia/Calcutta] PHP Fatal error: Uncaught Error: Call to a member function fetch_array() on bool in /var/www/v17_test/api/functions.php:2181
Stack trace:
#0 /var/www/v17_test/common_top.php(8): getVersionDetailsAPI()
#1 /var/www/v17_test/login.php(21): include_once('/var/www/v17_te...')
#2 {main}
thrown in /var/www/v17_test/api/functions.php on line 2181
[24-Jan-2024 14:54:39 Asia/Calcutta] PHP Warning: session_destroy(): Trying to destroy uninitialized session in /var/www/commandcenter/logout.php on line 11
[24-Jan-2024 15:09:01 Asia/Calcutta] PHP Warning: Module 'mcrypt' already loaded in Unknown on line 0
[25-Jan-2024 17:41:33 Asia/Kolkata] PHP Warning: Use of undefined constant UAT - assumed 'UAT' (this will throw an Error in a future version of PHP) in /var/www/moh_citizen_portal/appFooterBase.php on line 12
[25-Jan-2024 22:29:26 Asia/Calcutta] PHP Warning: Module 'mcrypt' already loaded in Unknown on line 0
```
These are my php error logs which i want to upload by using below script
```auto
input {
file {
path => "/home/plus91/Documents/php_log_DirectUpload.log"
start_position => "beginning"
codec => multiline {
pattern => "^\[%{DATA:timestamp} %{TZ:timezone}\] "
negate => true
what => "previous"
}
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
grok {
match => { "message" => "\[%{DATA:timestamp} %{TZ:timezone}\] %{DATA:log_level}: %{GREEDYDATA:log_message}" }
}
date {
match => ["timestamp", "dd-MMM-yyyy HH:mm:ss", "dd-MMM-yyyy HH:mm:ss z"]
target => "@timestamp"
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
user => "elastic"
password => "plus91"
index => "php_error_logs_1"
action => "index"
document_type => "_doc"
}
stdout {
codec => rubydebug
}
}
```
when i run this logs.conf it does not create any index on my elasticsearch index and script takes to much time to load on my terminal
please let me know if i'm missing something.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [January 29, 2024, 12:21pm UTC](https://discuss.elastic.co/t/how-to-define-a-proper-grok-pattern-for-php-error-logs/351978/2 "2024-01-29T12:21:51Z")
```auto
input {
file {
path => "/home/plus91/Documents/php_log_DirectUpload.log"
start_position => beginning
codec => multiline
{
pattern => '^\[\d{2}-[a-zA-Z]{3}-\d{4} \d{2}:\d{2}:\d{2} '
negate => true
what => previous
auto_flush_interval => 1
multiline_tag => ""
}
}
} # input
filter {
grok {
match => { "message" => "\[%{LOGTIME:[@metadata][timestamp]} %{DATA:[@metadata][tz]}\] PHP %{DATA:log_level}:%{SPACE}%{GREEDYDATA:log_message}" }
pattern_definitions => { "LOGTIME" => "%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME}" }
}
date {
match => ["[@metadata][timestamp]", "dd-MMM-yyyy HH:mm:ss" ]
target => "@timestamp"
timezone => "%{[@metadata][tz]}"
}
# mutate{ remove_field => ["log","message", "event", "host"] }
}
output {
stdout { codec => rubydebug{} }
}
```
Result:
```auto
{
"log_level" => "Warning",
"log_message" => "Module 'mcrypt' already loaded in Unknown on line 0\r",
"@timestamp" => 2024-01-16T08:09:08.000Z,
"@version" => "1"
}
{
"log_level" => "Fatal error",
"log_message" => "Uncaught Error: Call to a member function fetch_array() on bool in /var/www/v17_test/api/functions.php:2181\r\nStack trace:\r\n#0 /var/www/v17_test/common_top.php(8): getVersionDetailsAPI()\r\n#1 /var/www/v17_test/login.php(21): include_once('/var/www/v17_te...')\r\n#2 {main}\r\n thrown in /var/www/v17_test/api/functions.php on line 2181\r",
"@timestamp" => 2024-01-23T06:33:18.000Z,
"@version" => "1"
}
```
---
### Author:  [@Ritikapawar](https://discuss.elastic.co/u/Ritikapawar)
#### Post date: [January 30, 2024, 6:32am UTC](https://discuss.elastic.co/t/how-to-define-a-proper-grok-pattern-for-php-error-logs/351978/3 "2024-01-30T06:32:25Z")
Hi @Rios as per your suggestion i've changed the script but it is taking a lot of time to run and not creating any index. I'm attaching a screen shot below please have look
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 30, 2024, 7:21am UTC](https://discuss.elastic.co/t/how-to-define-a-proper-grok-pattern-for-php-error-logs/351978/4 "2024-01-30T07:21:53Z")
Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.
Instead, paste the text and format it with `` icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.
It would be great if you could update your post to solve this.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [January 30, 2024, 7:23am UTC](https://discuss.elastic.co/t/how-to-define-a-proper-grok-pattern-for-php-error-logs/351978/5 "2024-01-30T07:23:13Z")
Most likely your file had been read, and LS cannot read the same file twice.
Add `sincedb_path => "/dev/null"` like this:
```auto
input {
file {
path => "/home/plus91/Documents/php_log_DirectUpload.log"
start_position => beginning
sincedb_path => "/dev/null"
codec => multiline
{
pattern => '^\[\d{2}-[a-zA-Z]{3}-\d{4} \d{2}:\d{2}:\d{2} '
negate => true
what => previous
auto_flush_interval => 1
multiline_tag => ""
}
}
}
```
Or delete the file in sincedb\_path every time, before run. The name is on 4th line from the bottom ends with the hash value.
---
### Author:  [@Ritikapawar](https://discuss.elastic.co/u/Ritikapawar)
#### Post date: [January 30, 2024, 10:02am UTC](https://discuss.elastic.co/t/how-to-define-a-proper-grok-pattern-for-php-error-logs/351978/6 "2024-01-30T10:02:38Z")
Hi @Rios Script ran successfully for this type of logs
`[23-Jan-2024 12:03:18 Asia/Calcutta] PHP Warning: mysqli::query(): Couldn't fetch mysqli in /var/www/v17_test/api/functions.php on line 2180`.
but got one issue in this
`[23-Jan-2024 12:03:18 Asia/Calcutta] PHP Fatal error: Uncaught Error: Call to a member function fetch_array() on bool in /var/www/v17_test/api/functions.php:2181 Stack trace: #0 /var/www/v17_test/common_top.php(8): getVersionDetailsAPI() #1 /var/www/v17_test/login.php(21): include_once('/var/www/v17_te...') #2 {main} thrown in /var/www/v17_test/api/functions.php on line 2181`
for this blank entry has been added in index like this
```auto
{
"_index" : "php_error_logs_1",
"_type" : "_doc",
"_id" : "Y9O8WY0B2AzWmE8ib4Ae",
"_score" : 1.0,
"_source" : {
"path" : "/home/plus91/Documents/php_log_DirectUpload.log",
"@timestamp" : "2024-01-30T09:38:57.596Z",
"tags" : [
"_grokparsefailure"
],
"@version" : "1"
}
},
```
what changes are required in my script to accept all types of logs please suggest
```auto
input {
file {
path => "/home/plus91/Documents/php_log_DirectUpload.log"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => '^\[\d{2}-[a-zA-Z]{3}-\d{4} \d{2}:\d{2}:\d{2} '
negate => true
what => "previous"
auto_flush_interval => 1
multiline_tag => ""
}
}
}
filter {
grok {
match => { "message" => "\[%{LOGTIME:[@metadata][timestamp]} %{DATA:[@metadata][tz]}\] PHP %{WORD:log_level}:%{SPACE}%{GREEDYDATA:log_message}" }
pattern_definitions => { "LOGTIME" => "%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME}" }
}
date {
match => ["[@metadata][timestamp]", "dd-MMM-yyyy HH:mm:ss" ]
target => "@timestamp"
timezone => "%{[@metadata][tz]}"
}
# You can remove unnecessary fields if needed
mutate {
remove_field => ["log", "message", "event", "host"]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
user => "elastic"
password => "plus91"
index => "php_error_logs_1"
action => "index"
doc_as_upsert => true
}
stdout {
codec => rubydebug
}
}
```
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [January 30, 2024, 10:37am UTC](https://discuss.elastic.co/t/how-to-define-a-proper-grok-pattern-for-php-error-logs/351978/7 "2024-01-30T10:37:14Z")
> [@Ritikapawar](#):
>
> `%{WORD:log_level}`
You cannot use WORD which is \b\w+\b, because "Fatal error". You should use `%{DATA:log_level}`, it works for me.
---
### Author:  [@Ritikapawar](https://discuss.elastic.co/u/Ritikapawar)
#### Post date: [January 30, 2024, 11:04am UTC](https://discuss.elastic.co/t/how-to-define-a-proper-grok-pattern-for-php-error-logs/351978/8 "2024-01-30T11:04:55Z")
Data added into index but showing as
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [January 30, 2024, 11:35am UTC](https://discuss.elastic.co/t/how-to-define-a-proper-grok-pattern-for-php-error-logs/351978/9 "2024-01-30T11:35:14Z")
And what is a problem? You will have 3 fields
"log\_level" =\> "Fatal error",
"log\_message" =\> "bla bla",
"@timestamp" =\> 2024-01-23T06:33:18.000Z,
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 27, 2024, 11:36am UTC](https://discuss.elastic.co/t/how-to-define-a-proper-grok-pattern-for-php-error-logs/351978/10 "2024-02-27T11:36:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.