# Not able to grok filebeat's mysql logs output

**URL:** <https://discuss.elastic.co/t/not-able-to-grok-filebeats-mysql-logs-output/117955>\
**Category:** Logstash\
**Created:** [February 1, 2018, 8:00am UTC](https://discuss.elastic.co/t/not-able-to-grok-filebeats-mysql-logs-output/117955 "2018-02-01T08:00:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rajya\_vardhan\_Mishra](https://avatars.discourse-cdn.com/v4/letter/r/bc79bd/32.png) [@rajya\_vardhan\_Mishra](https://discuss.elastic.co/u/rajya_vardhan_Mishra)\
**Post date:** [February 1, 2018, 8:00am UTC](https://discuss.elastic.co/t/not-able-to-grok-filebeats-mysql-logs-output/117955/1 "2018-02-01T08:00:40Z")

</div>

Hi,

I have been trying to grok filebeat messages(in json format), so that i can create kibana dashboard. But not able to crack this , even after spending couple of days!

Step 1:  
My filebeat plugin puts messages in a log file. Example messages:

> {"@timestamp":"2018-01-31T12:50:09.481Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.1.2","topic":"abc"},"offset":1696118,"prospector":{"type":"log"},"grok\_key":"mysql.errorlog","beat":{"name":"host1","hostname":"host1","version":"6.1.2"},"message":"2018-01-31T12:50:08.436179Z 31898 [Note] Aborted connection 123 to db: 'product' user: 'user' host: '0.0.0.0' (Got timeout reading communication packets)","source":"/logs/mysql-error.log"}

I want to grok this message so that i can plot them in Kibana. Referring grok examples from [here](https://www.elastic.co/guide/en/logstash/current/logstash-config-for-filebeat-modules.html#parsing-mysql)

So my logstash.conf looks like:

> ```
> input {
> file {
> path => "/tmp/slow.log"
> type => "mysql.errorlog"
> start_position => "beginning"
> }
> }
> 
> filter {
> json{
> source => "message"
> }
> grok {
> match => { "message" => ["%{LOCALDATETIME:[mysql][error][timestamp]} (\[%{DATA:[mysql][error][level]}\] )?%{GREEDYDATA:[mysql][error][message]}",
> "%{TIMESTAMP_ISO8601:[mysql][error][timestamp]} %{NUMBER:[mysql][error][thread_id]} \[%{DATA:[mysql][error][level]}\] %{GREEDYDATA:[mysql][error][message1]}",
> "%{GREEDYDATA:[mysql][error][message2]}"] }
> pattern_definitions => {
> "LOCALDATETIME" => "[0-9]+ %{TIME}"
> }
> remove_field => "message"
> }
> mutate {
> rename => { "[mysql][error][message1]" => "[mysql][error][message]" }
> }
> mutate {
> rename => { "[mysql][error][message2]" => "[mysql][error][message]" }
> }
> date {
> match => ["[mysql][error][timestamp]", "ISO8601", "YYMMdd H:m:s" ]
> remove_field => "[mysql][error][time]"
> }
> }
> 
> output {
> elasticsearch { hosts => localhost }
> stdout {codec => rubydebug }
> }
> 
> ```

I start logstash as:

> /opt/logstash/bin/logstash -f /usr/share/logstash/pipeline/logstash.conf --debug --verbose

When the log message is parsed, nothing is pushed to ES.  
On stdout: from output it seems that grok is working fine and able to run regex successfully. But, no output goes in ES.

> ```
> {
> "offset" => 1696118,
> "prospector" => {
> "type" => "log"
> },
> "source" => "/logs/mysql-error.log",
> "type" => "mysql.errorlog",
> "path" => "/tmp/slow.log",
> "@timestamp" => 2018-01-31T12:50:08.436Z,
> "@version" => "1",
> "host" => "1111",
> "beat" => {
> "hostname" => "host1",
> "name" => "host1",
> "version" => "6.1.2"
> },
> "mysql" => {
> "error" => {
> "thread_id" => "31898",
> "level" => "Note",
> "message" => "Aborted connection 31898 to db: 'product' user: 'user' host: '0.0.0.0' (Got timeout reading communication packets)",
> "timestamp" => "2018-01-31T12:50:08.436179Z"
> }
> },
> "grok_key" => "mysql.errorlog"
> }
> 
> ```

Weird, that logstash also doesn't give any debug/error messages.

I have tried many changes in this config, and nothing seems to be working.  
If i remove the grok filter and keep the filter simply as below, then messages go to ES:

> ```
> filter {
> json{
> source => "message"
> }
> }
> 
> ```

Plz help.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 1, 2018, 9:52am UTC](https://discuss.elastic.co/t/not-able-to-grok-filebeats-mysql-logs-output/117955/2 "2018-02-01T09:52:15Z")

</div>

If you're not getting any log messages at all from Logstash that's something you should address first.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 1, 2018, 9:52am UTC](https://discuss.elastic.co/t/not-able-to-grok-filebeats-mysql-logs-output/117955/3 "2018-03-01T09:52:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.