Observability Engineer Lab 3.3 - Empty Columns

Elastic Community and Ecosystem Elastic Training
1

Hi,

I'm working on the Observability Engineer lab. In Module 3.3, task 12, the document shows that the messages from /var/lib/mysql/mysql.log are parsed correctly, where as in my setup, the messages are not parsed.

the message field is showing up as

2021-09-28T13:38:05.815137Z	   12 Query	select owner0_.id as id1_0_0_, owner0_.first_name as first_na2_0_0_, owner0_.last_name as last_nam3_0_0_, owner0_.address as address4_0_0_, owner0_.city as city5_0_0_, owner0_.state as state6_0_0_, owner0_.telephone as telephon7_0_0_, owner0_.zip_code as zip_code8_0_0_, pets1_.owner_id as owner_id4_1_1_, pets1_.id as id1_1_1_, pets1_.id as id1_1_2_, pets1_.name as name2_1_2_, pets1_.birth_date as birth_da3_1_2_, pets1_.owner_id as owner_id4_1_2_, pets1_.type_id as type_id5_1_2_, pettype2_.id as id1_4_3_, pettype2_.name as name2_4_3_, visits3_.pet_id as pet_id4_8_4_, visits3_.id as id1_8_4_, visits3_.id as id1_8_5_, visits3_.visit_date as visit_da2_8_5_, visits3_.description as descript3_8_5_, visits3_.pet_id as pet_id4_8_5_ from owners owner0_ left outer join pets pets1_ on owner0_.id=pets1_.owner_id left outer join types pettype2_ on pets1_.type_id=pettype2_.id left outer join visits visits3_ on pets1_.id=visits3_.pet_id where owner0_.id=11

Should this be parsed as shown in the document ?

Thanks
Murali

2

I think you're asking about the exam prep lab, yes? If so, the issue is that there is a tab stop character after the date field. This is a tricky one to get working! Here's a solution to try (you'll have to paste it in as code directly, but the GUI will parse it):

{
"processors": [
  {
    "dissect": {
      "field": "message",
      "pattern": "%{@timestamp}\t%{->} %{process.thread.id} %{event.action}\t%{message}"
    }
  },
  {
    "lowercase": {
      "field": "event.action"
    }
  },
  {
    "set": {
      "field": "event.category",
      "value": "database"
    }
  }
]
}
3

@dinyes

Thanks for your quick response. It's strange that the screen shot shown in the documentation for the mysql logs (lab 3.2 -> task 13) is same as what I see in my setup. There is no pipeline to process mysql logs that's in /var/lib/mysql/mysql.log folder.

I will try to add the log to the mysql module instead of adding it in the filebeat.yml to see if it makes a difference.

Thanks
Murali

4

Ah, I may be thinking of a different task. Yes, try putting the log location in the mysql module.

5

No luck, I got the following error

ERROR   instance/beat.go:951    Exiting: Failed to start crawler: creating module reloader failed: fileset mysql/log is configured but doesn't existerror.log*"]

This is my entry in mysql.yml file (same as the one in filebeat.reference file)

- module: mysql
  # MySQL Logs
  log:
    enabled: true
    #var.paths:

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Slow logs
  slowlog:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
6

Hi

I got this to work. I had to do this to process the mysql logs (probabaly there is another work around for this as well).

Copied mysql.yml as mysql1.yml and enabled just the error log as below.

- module: mysql

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: [ "/var/lib/mysql/mysql.log" ]

Thanks
Murali

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.