I'm trying to parse an aplication log which is a Symfony application but its not working fine

Elastic Stack Beats
1

hello, please help me to solve the below issues
i have taken a sample log entries from sympony application and stored in elk server itself /crm.log for testing, if it would have done parsing the logs i will would have started installing filebeat in symfony application. But it was not!

here are the steps which i tried:
i have created a logstash pipeline

input {
  file {
    path => "/crm.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
  }
}
output {
  elasticsearch {
    hosts => ["https://ip"]
    user => "elastic"
    password => "passwd"
    index => "crm" 
    ssl_verification_mode => "full"
    ssl_certificate_authorities => "/etc/elasticsearch/certs/http_ca.crt"
    ssl_enabled => true
  }
  
}

and i have created a mapping like

PUT /_index_template/my_log_template
{
  "index_patterns": ["my_log"], 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    },
    "mappings": {
      "properties": {
        "level": {
          "type": "keyword"
        },
        "message": {
          "type": "text"
        },
        "exception": {
          "properties": {
            "type": {
              "type": "keyword"
            },
            "message": {
              "type": "text"
            },
            "stack_trace": {
              "type": "text"
            }
          }
        },
        "processor": {
          "type": "keyword"
        },
        "message_id": {
          "type": "keyword"
        },
        "message_body": {
          "properties": {
            "timestamp": {
              "type": "date"
            },
            "transaction_id": {
              "type": "keyword"
            },
            "entities_updated": {
              "type": "nested",
              "properties": {
                "entity_class": {
                  "type": "keyword"
                },
                "entity_id": {
                  "type": "integer"
                },
                "change_set": {
                  "type": "object"
                }
              }
            },
            "entities_inserted": {
              "type": "nested",
              "properties": {
                "entity_class": {
                  "type": "keyword"
                },
                "entity_id": {
                  "type": "integer"
                },
                "change_set": {
                  "type": "object"
                }
              }
            },
            "entities_deleted": {
              "type": "nested",
              "properties": {
                "entity_class": {
                  "type": "keyword"
                },
                "entity_id": {
                  "type": "integer"
                },
                "change_set": {
                  "type": "object"
                }
              }
            },
            "collections_updated": {
              "type": "nested",
              "properties": {
                "entity_class": {
                  "type": "keyword"
                },
                "entity_id": {
                  "type": "integer"
                },
                "change_set": {
                  "type": "object",
                  "properties": {
                    "user_d41b1c4b": {
                      "type": "object",
                      "properties": {
                        "inserted": {
                          "type": "nested",
                          "properties": {
                            "entity_class": {
                              "type": "keyword"
                            },
                            "entity_id": {
                              "type": "integer"
                            },
                            "change_set": {
                              "type": "object"
                            }
                          }
                        },
                        "deleted": {
                          "type": "nested",
                          "properties": {
                            "entity_class": {
                              "type": "keyword"
                            },
                            "entity_id": {
                              "type": "integer"
                            },
                            "change_set": {
                              "type": "object"
                            }
                          }
                        },
                        "changed": {
                          "type": "object"
                        }
                      }
                    }
                  }
                }
              }
            },
            "properties": {
              "type": "object",
              "properties": {
                "oro.message_queue.client.topic_name": {
                  "type": "keyword"
                },
                "oro.message_queue.client.processor_name": {
                  "type": "keyword"
                },
                "oro.message_queue.client.queue_name": {
                  "type": "keyword"
                }
              }
            },
            "headers": {
              "type": "object",
              "properties": {
                "content_type": {
                  "type": "keyword"
                },
                "message_id": {
                  "type": "keyword"
                }
              }
            },
            "message_properties": {
              "type": "object",
              "properties": {
                "oro.message_queue.client.topic_name": {
                  "type": "keyword"
                },
                "oro.message_queue.client.processor_name": {
                  "type": "keyword"
                },
                "oro.message_queue.client.queue_name": {
                  "type": "keyword"
                }
              }
            },
            "message_headers": {
              "type": "object",
              "properties": {
                "content_type": {
                  "type": "keyword"
                },
                "message_id": {
                  "type": "keyword"
                }
              }
            }
          }
        }
      }
    }
  }
}

here is my sample log entry

[2024-05-28 04:00:05][][] app.ERROR: Consuming interrupted by exception. "Missed server heartbeat" {"exception":"[object] (\Exception\\AMQPHeartbeatMissedException(code: 0): Missed server heartbeat at /websites/crm-application/vendor/AbstractIO.php:167)"} {"processor":"Oro\\Component\\MessageQueue\\Router\\RouteRecipientListProcessor","message_id":"oro.66d0743","message_body":"{\"timestamp\":1716861605,\"transaction_id\":\"010c1af32f5d\",\"entities_updated\":[],\"entities_inserted\":[],\"entities_deleted\":[],\"collections_updated\":[{\"entity_class\":\"Oro\\\\Bundle\\\\EmailBundle\\\\Entity\\\\Email\",\"entity_id\":253,\"change_set\":{\"user_d44b\":[null,{\"inserted\":[{\"entity_class\":\"Oro\\\\Bundle\\\\UserBundle\\\\Entity\\\\User\",\"entity_id\":194,\"change_set\":[]}],\"deleted\":[],\"changed\":[]}]}},{\"entity_class\":\"Oro\\\\Bundle\\\\ActivityListBundle\\\\Entity\\\\ActivityList\",\"entity_id\":6059,\"change_set\":{\"user_1091\":[null,{\"inserted\":[{\"entity_class\":\"Oro\\\\Bundle\\\\UserBundle\\\\Entity\\\\User\",\"entity_id\":194,\"change_set\":[]}],\"deleted\":[],\"changed\":[]}]}}],\"properties\":{\"oro.message_queue.client.topic_name\":\"oro.data_audit.entities_changed\",\"oro.message_queue.client.processor_name\":\"oro_message_queue.client.route_message_processor\",\"oro.message_queue.client.queue_name\":\"oro.default\"},\"headers\":{\"content_type\":\"application\\/json\",\"message_id\":\"oro.66553743\"}}","message_properties"

in kibana im getting tags: _jsonparsefailure and whole logentry was inside the message field
and i have tried with the filter

filter {
    json {
        source => "message"
        target => "parsed_json"
        remove_field => ["message"]
    }
}

and mapping as

PUT /_index_template/my_index_template
{
 "index_patterns": ["crm1"], 
 "template": {
   "settings": {
     "number_of_shards": 1,
     "number_of_replicas": 0
   },
   "mappings": {
     "properties": {
       "parsed_json": {  // Assuming your fields are nested under a `parsed_json` field
         "properties": {

still i'm facing the same issue!

2

please do reply

3

Your Log is clearly not JSON so it will not parse... You will need to parse the log with a GROK or something first then use json filter.

If you json starts at
...Missed server heartbeat" {"exception....
............................^

Then you are going to need to use a GROK filter to extract the correct json and THEN use the json filter

4

thank you! i'm a newbie to ELK
can you provide some information about how can i use the grok filter to extract the correct json.

5

Hi @evangelin

There are a lot of examples / documentation on Logstash / GROK etc.

I would say take a look and dive in! It looks to me that your log like

  • Has some field including timestamps etc in the beginning
  • Then some field / value pairs which some have JSON etc...
  • I am also not sure if that is a full line as it seems to be missing closing } etc
  • This is a non-trvial parsing :slight_smile: but absolutely can be done

If you need help building patterns to match your logs, you will find the http://grokdebug.herokuapp.com and http://grokconstructor.appspot.com/ applications quite useful!

6

thank you! i will try those and let you know if i got any doubts.