Filebeat breaks log entries when there is a colon in the value of a field

Elastic Stack Beats
1

Dear Community,

I would like to ask for an advice from you. I tried several methods to resolve this problem, but without a success.

My problem is: If a log entry (generated by my application) contains a colon (":"), filebeat breaks the value of the field (the field that contains the colon) and places it into seperate fields.

My filebeat.yml:

# ---------------------------- Log files Input ------------------------------
filebeat.inputs:

  - type: log
    enabled: true
    json.keys_under_root: true
    json.add_error_key: true
    fields:
      log_type: log_1
    paths:
      - path/to/log/file

# ---------------------------- Elasticsearch Output --------------------------
output:
  elasticsearch:
    hosts: ...
    index: "%{[fields.log_type]}-%{+yyyy.MM.dd}"

# ---------------------------- Elasticsearch settings ------------------------
setup:
  template:
    name: "filebeat"
    pattern: "*"
    settings:
      index.number_of_shards: 3
      index.number_of_replicas: 1
  ilm:
    enabled: false
# ---------------------------- Processors ------------------------
 processors:
  - convert:
      fields:
        - from: "message"
          to: "message"
          type: "string"

Example log entry:

{"reqId":"t45rtdfg2zd9d74bbd4d6df100befa56","level":3,"time":"2024-06-26T08:44:03.234+00:00","remoteAddr":"xx.xx.xx.xx","user":"user1","app":"no app in context","method":"GET","url":"/index.php/one.jpg","message":"Requested file with invalid file id: -1","userAgent":"Mozilla/5.0 (Android) myApp/9.10.9","version":"9.10.9","data":[]}

or the same in a readable format:

{
"reqId":"t45rtdfg2zd9d74bbd4d6df100befa56",
"level":3,
"time":"2024-06-26T08:44:03.234+00:00",
"remoteAddr":"xx.xx.xx.xx",
"user":"user1",
"app":"no app in context",
"method":"GET",
"url":"/index.php/one.jpg",
"message":"Requested file with invalid file id: -1",
"userAgent":"Mozilla/5.0 (Android) myApp/9.10.9",
"version":"9.10.9",
"data":[]
}

The rest of the log entries (that does not contain a colon in the "message" field) are processed fine, and displayed in Kibana as it should.

Is there any method to escape the colons in that field or any other method to solve this problem?

I am using filebeat version 7.12, and Kibana 7.10.2.

2

Hi,

you can try to use the Decode Json Fields

Regards

3

Hi yago82,

Thanks for your answer, however I already tried it without success. This is what I've tried:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/your/logfile.log
  processors:
    - decode_json_fields:
        fields: ["message"]
        process_array: false
        max_depth: 1
        target: ""
        overwrite_keys: true

and tried with dissect as well, but there were no effect:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/your/logfile.log
  processors:
      - dissect:
          tokenizer: "%{message}"
          field: "message"
          trim_values: "none"
          ignore_failure: true
          overwrite_keys: true
          target_prefix: ""