Help needed with ingesting custom log data

[Sakis]

Hey there!

I'm new to Elastic Stack (currently using the 14-day cloud trial) and I have a few questions about parsing log data. I work at an web development company and we provide customers with shop systems from Shopware (open sourece shop system). Now we plan to analyze and visualize the logs Shopware creates. It is partly based on Symfony, so the log entries are quite complex and also come in different syntaxes.

So I tried to import a log file for the Custom Log integration. Unfortunately it has problems with detecting the correct syntax of different log entries. Here are a few examples of how different those entries can look:

• [2022-03-25 00:05:11] request.CRITICAL: Uncaught PHP Exception LogicException: "Cannot change the name of an active session." at /home/xxxxxx/public_html/vendor/symfony/http-foundation/Session/Storage/Proxy/AbstractProxy.php line 117 {"exception":"[object] (LogicException(code: 0): Cannot change the name of an active session. at /home/xxxxxxx/public_html/vendor/symfony/http-foundation/Session/Storage/Proxy/AbstractProxy.php:117)"} [ ]

• [2022-05-17 07:19:56] request.ERROR: Uncaught PHP Exception Symfony\Component\HttpKernel\Exception\NotFoundHttpException: "No route found for "GET /bundles/administration/static/img/favicon/favicon-32x32.png" (from "https://xxxxxx")" at /home/xxxxxx/public_html/vendor/symfony/http-kernel/EventListener/RouterListener.php line 136 {"exception":"[object] (Symfony\Component\HttpKernel\Exception\NotFoundHttpException(code: 0): No route found for "GET /bundles/administration/static/img/favicon/favicon-32x32.png" (from "https://xxxxxxx") at /home/xxxxx/public_html/vendor/symfony/http-kernel/EventListener/RouterListener.php:136, Symfony\Component\Routing\Exception\ResourceNotFoundException(code: 0): No routes found for "/bundles/administration/static/img/favicon/favicon-32x32.png/". at /home/xxxxxx/public_html/vendor/symfony/routing/Matcher/Dumper/CompiledUrlMatcherTrait.php:70)"} [ ]

• [2022-04-19 00:00:02] request.ERROR: Uncaught PHP Exception Shopware\Core\Content\Product\Exception\ProductNotFoundException: "Product for id 1355b484b3754c52980b340897f51b54 not found." at /home/xxxxxx/public_html/custom/plugins/theme/src/Storefront/Page/Product/ModifiedProductLoader.php line 82 {"exception":"[object] (Shopware\Core\Content\Product\Exception\ProductNotFoundException(code: 0): Product for id 1355b484b3754c52980b340897f51b54 not found. at /home/xxxxxxxx/public_html/custom/plugins/theme/src/Storefront/Page/Product/ModifiedProductLoader.php:82)"} [ ]

The automatic parsing fails to get the job done properly. Thus, I tried to override the automatic settings, but for some reason I'm not able to shorten the grok expression to timestamp, log level, and greedydata:message. It simply won't let me define the whole part after the log level as a text message. I don't know what I'm doing wrong.. We don't need to extract/parse every detail from a log entry; the message itself would suffice.

As I'm a beginner, I'm quite confused by all the settings and features Elastic Stack offers. So I'm looking for a comprehensible tutorial about how to import logs, analyze them by collecting (critical) errors, and receive an email notification in case of (unusual) errors. YouTube offers many tutorials, but I haven't found one yet that answers all of my question.

Any hint would be greatly appreciated!

Thanks in advance,

Sakis

[Rios]

Here is the filter part:

filter {
	
	grok { 
       match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:method}\.%{DATA:level}\: %{GREEDYDATA:msg}" }
	}
	
    date {
       match => ["timestamp", "yyyy-MM-dd HH:mm:ss" ]
       target => "@timestamp"
	   remove_field => ["timestamp", "message"]
	   timezone => "Asia/Singapore"
    }
}