# I'm trying to parse an aplication log which is a Symfony application but its not working fine

**URL:** <https://discuss.elastic.co/t/im-trying-to-parse-an-aplication-log-which-is-a-symfony-application-but-its-not-working-fine/360563>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 30, 2024, 12:47pm UTC](https://discuss.elastic.co/t/im-trying-to-parse-an-aplication-log-which-is-a-symfony-application-but-its-not-working-fine/360563 "2024-05-30T12:47:57Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![evangelin](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@evangelin](https://discuss.elastic.co/u/evangelin)\
**Post date:** [May 30, 2024, 12:47pm UTC](https://discuss.elastic.co/t/im-trying-to-parse-an-aplication-log-which-is-a-symfony-application-but-its-not-working-fine/360563/1 "2024-05-30T12:47:57Z")

</div>

hello, please help me to solve the below issues  
i have taken a sample log entries from sympony application and stored in elk server itself /crm.log for testing, if it would have done parsing the logs i will would have started installing filebeat in symfony application. But it was not!

here are the steps which i tried:  
i have created a logstash pipeline

```auto
input {
  file {
    path => "/crm.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
  }
}
output {
  elasticsearch {
    hosts => ["https://ip"]
    user => "elastic"
    password => "passwd"
    index => "crm" 
    ssl_verification_mode => "full"
    ssl_certificate_authorities => "/etc/elasticsearch/certs/http_ca.crt"
    ssl_enabled => true
  }
  
}

```

and i have created a mapping like

```auto
PUT /_index_template/my_log_template
{
  "index_patterns": ["my_log"], 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    },
    "mappings": {
      "properties": {
        "level": {
          "type": "keyword"
        },
        "message": {
          "type": "text"
        },
        "exception": {
          "properties": {
            "type": {
              "type": "keyword"
            },
            "message": {
              "type": "text"
            },
            "stack_trace": {
              "type": "text"
            }
          }
        },
        "processor": {
          "type": "keyword"
        },
        "message_id": {
          "type": "keyword"
        },
        "message_body": {
          "properties": {
            "timestamp": {
              "type": "date"
            },
            "transaction_id": {
              "type": "keyword"
            },
            "entities_updated": {
              "type": "nested",
              "properties": {
                "entity_class": {
                  "type": "keyword"
                },
                "entity_id": {
                  "type": "integer"
                },
                "change_set": {
                  "type": "object"
                }
              }
            },
            "entities_inserted": {
              "type": "nested",
              "properties": {
                "entity_class": {
                  "type": "keyword"
                },
                "entity_id": {
                  "type": "integer"
                },
                "change_set": {
                  "type": "object"
                }
              }
            },
            "entities_deleted": {
              "type": "nested",
              "properties": {
                "entity_class": {
                  "type": "keyword"
                },
                "entity_id": {
                  "type": "integer"
                },
                "change_set": {
                  "type": "object"
                }
              }
            },
            "collections_updated": {
              "type": "nested",
              "properties": {
                "entity_class": {
                  "type": "keyword"
                },
                "entity_id": {
                  "type": "integer"
                },
                "change_set": {
                  "type": "object",
                  "properties": {
                    "user_d41b1c4b": {
                      "type": "object",
                      "properties": {
                        "inserted": {
                          "type": "nested",
                          "properties": {
                            "entity_class": {
                              "type": "keyword"
                            },
                            "entity_id": {
                              "type": "integer"
                            },
                            "change_set": {
                              "type": "object"
                            }
                          }
                        },
                        "deleted": {
                          "type": "nested",
                          "properties": {
                            "entity_class": {
                              "type": "keyword"
                            },
                            "entity_id": {
                              "type": "integer"
                            },
                            "change_set": {
                              "type": "object"
                            }
                          }
                        },
                        "changed": {
                          "type": "object"
                        }
                      }
                    }
                  }
                }
              }
            },
            "properties": {
              "type": "object",
              "properties": {
                "oro.message_queue.client.topic_name": {
                  "type": "keyword"
                },
                "oro.message_queue.client.processor_name": {
                  "type": "keyword"
                },
                "oro.message_queue.client.queue_name": {
                  "type": "keyword"
                }
              }
            },
            "headers": {
              "type": "object",
              "properties": {
                "content_type": {
                  "type": "keyword"
                },
                "message_id": {
                  "type": "keyword"
                }
              }
            },
            "message_properties": {
              "type": "object",
              "properties": {
                "oro.message_queue.client.topic_name": {
                  "type": "keyword"
                },
                "oro.message_queue.client.processor_name": {
                  "type": "keyword"
                },
                "oro.message_queue.client.queue_name": {
                  "type": "keyword"
                }
              }
            },
            "message_headers": {
              "type": "object",
              "properties": {
                "content_type": {
                  "type": "keyword"
                },
                "message_id": {
                  "type": "keyword"
                }
              }
            }
          }
        }
      }
    }
  }
}

```

here is my sample log entry

```auto
[2024-05-28 04:00:05][][] app.ERROR: Consuming interrupted by exception. "Missed server heartbeat" {"exception":"[object] (\Exception\\AMQPHeartbeatMissedException(code: 0): Missed server heartbeat at /websites/crm-application/vendor/AbstractIO.php:167)"} {"processor":"Oro\\Component\\MessageQueue\\Router\\RouteRecipientListProcessor","message_id":"oro.66d0743","message_body":"{\"timestamp\":1716861605,\"transaction_id\":\"010c1af32f5d\",\"entities_updated\":[],\"entities_inserted\":[],\"entities_deleted\":[],\"collections_updated\":[{\"entity_class\":\"Oro\\\\Bundle\\\\EmailBundle\\\\Entity\\\\Email\",\"entity_id\":253,\"change_set\":{\"user_d44b\":[null,{\"inserted\":[{\"entity_class\":\"Oro\\\\Bundle\\\\UserBundle\\\\Entity\\\\User\",\"entity_id\":194,\"change_set\":[]}],\"deleted\":[],\"changed\":[]}]}},{\"entity_class\":\"Oro\\\\Bundle\\\\ActivityListBundle\\\\Entity\\\\ActivityList\",\"entity_id\":6059,\"change_set\":{\"user_1091\":[null,{\"inserted\":[{\"entity_class\":\"Oro\\\\Bundle\\\\UserBundle\\\\Entity\\\\User\",\"entity_id\":194,\"change_set\":[]}],\"deleted\":[],\"changed\":[]}]}}],\"properties\":{\"oro.message_queue.client.topic_name\":\"oro.data_audit.entities_changed\",\"oro.message_queue.client.processor_name\":\"oro_message_queue.client.route_message_processor\",\"oro.message_queue.client.queue_name\":\"oro.default\"},\"headers\":{\"content_type\":\"application\\/json\",\"message_id\":\"oro.66553743\"}}","message_properties"

```

in kibana im getting tags: \_jsonparsefailure and whole logentry was inside the message field  
and i have tried with the filter

```auto
filter {
    json {
        source => "message"
        target => "parsed_json"
        remove_field => ["message"]
    }
}

```

and mapping as

```auto
PUT /_index_template/my_index_template
{
 "index_patterns": ["crm1"], 
 "template": {
   "settings": {
     "number_of_shards": 1,
     "number_of_replicas": 0
   },
   "mappings": {
     "properties": {
       "parsed_json": { // Assuming your fields are nested under a `parsed_json` field
         "properties": {

```

still i'm facing the same issue!

---

<div class="post-metadata">

**Author:** ![evangelin](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@evangelin](https://discuss.elastic.co/u/evangelin)\
**Post date:** [June 4, 2024, 6:51am UTC](https://discuss.elastic.co/t/im-trying-to-parse-an-aplication-log-which-is-a-symfony-application-but-its-not-working-fine/360563/2 "2024-06-04T06:51:30Z")

</div>

please do reply

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 7, 2024, 5:56am UTC](https://discuss.elastic.co/t/im-trying-to-parse-an-aplication-log-which-is-a-symfony-application-but-its-not-working-fine/360563/3 "2024-06-07T05:56:56Z")

</div>

> [@evangelin](#):
>
> `[2024-05-28 04:00:05][][] app.ERROR: Consuming interrupted by exception. "Missed server heartbeat" {"exception":"[object] (\Exception\\AMQPHeartbeatMissedException(code: 0): Missed server heartbeat at /websites/crm-application/vendor/AbstractIO.php:167)"} {"processor":"Oro\\Component\\MessageQueue\\Router\\RouteRecipientListProcessor","message_id":"oro.66d0743","message_body":"{\"timestamp\":1716861605,\"transaction_id\":\"010c1af32f5d\",\"entities_updated\":[],\"entities_inserted\":[],\"entities_deleted\":[],\"collections_updated\":[{\"entity_class\":\"Oro\\\\Bundle\\\\EmailBundle\\\\Entity\\\\Email\",\"entity_id\":253,\"change_set\":{\"user_d44b\":[null,{\"inserted\":[{\"entity_class\":\"Oro\\\\Bundle\\\\UserBundle\\\\Entity\\\\User\",\"entity_id\":194,\"change_set\":[]}],\"deleted\":[],\"changed\":[]}]}},{\"entity_class\":\"Oro\\\\Bundle\\\\ActivityListBundle\\\\Entity\\\\ActivityList\",\"entity_id\":6059,\"change_set\":{\"user_1091\":[null,{\"inserted\":[{\"entity_class\":\"Oro\\\\Bundle\\\\UserBundle\\\\Entity\\\\User\",\"entity_id\":194,\"change_set\":[]}],\"deleted\":[],\"changed\":[]}]}}],\"properties\":{\"oro.message_queue.client.topic_name\":\"oro.data_audit.entities_changed\",\"oro.message_queue.client.processor_name\":\"oro_message_queue.client.route_message_processor\",\"oro.message_queue.client.queue_name\":\"oro.default\"},\"headers\":{\"content_type\":\"application\\/json\",\"message_id\":\"oro.66553743\"}}","message_properties"`

Your Log is clearly not JSON so it will not parse... You will need to parse the log with a GROK or something first then use json filter.

If you json starts at  
`...Missed server heartbeat" {"exception....`  
`............................^`

Then you are going to need to use a GROK filter to extract the correct json and THEN use the json filter

---

<div class="post-metadata">

**Author:** ![evangelin](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@evangelin](https://discuss.elastic.co/u/evangelin)\
**Post date:** [June 7, 2024, 5:59am UTC](https://discuss.elastic.co/t/im-trying-to-parse-an-aplication-log-which-is-a-symfony-application-but-its-not-working-fine/360563/4 "2024-06-07T05:59:30Z")

</div>

thank you! i'm a newbie to ELK  
can you provide some information about how can i use the grok filter to extract the correct json.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 7, 2024, 4:38pm UTC](https://discuss.elastic.co/t/im-trying-to-parse-an-aplication-log-which-is-a-symfony-application-but-its-not-working-fine/360563/5 "2024-06-07T16:38:41Z")

</div>

Hi @evangelin

There are a lot of examples / documentation on Logstash / GROK etc.

I would say take a look and dive in! It looks to me that your log like

- Has some field including timestamps etc in the beginning
- Then some field / value pairs which some have JSON etc...
- I am also not sure if that is a full line as it seems to be missing closing `}` etc
- This is a non-trvial parsing 🙂 but absolutely can be done

> **[Grok filter plugin | Logstash Reference \[8.14\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html)**

> **[What are Grok Patterns? Tutorial with Examples - Coralogix](https://coralogix.com/blog/logstash-grok-tutorial-with-examples/)**
>
> This tutorial will enable you to take full advantage of Elasticsearch’s analysis and querying capabilities by parsing with Logstash Grok

> If you need help building patterns to match your logs, you will find the [http://grokdebug.herokuapp.com](http://grokdebug.herokuapp.com/) and [http://grokconstructor.appspot.com/](http://grokconstructor.appspot.com/) applications quite useful!

---

<div class="post-metadata">

**Author:** ![evangelin](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@evangelin](https://discuss.elastic.co/u/evangelin)\
**Post date:** [June 10, 2024, 4:03am UTC](https://discuss.elastic.co/t/im-trying-to-parse-an-aplication-log-which-is-a-symfony-application-but-its-not-working-fine/360563/6 "2024-06-10T04:03:41Z")

</div>

thank you! i will try those and let you know if i got any doubts.