Log Parsing in filebeat

Hi,

I need some help to setup different configuration on filebeat.

I would like add a new grok pattern in /usr/share/filebeat/module/system/syslog/ingest/pipeline.json for this pattern of log :

[test0][pression][3.10.2][vpfr][ERR] 2019/07/11 07:42:19 MASTER-PRESSION-VTOM.go:122: <MSG MASTER> > [ 2019-07-11 07:42:19.824120057 +0200 CEST m=+650.565875227 ]

So I just added the following :

  grok": {
                 "field": "message",
                 "patterns": [
                     .....
                     "\\[%{DATA:agent.hostname}\\]\\[%{DATA:process.name}\\]\\[%{DATA:process.version}\\]\\[%{DATA:process.customer}\\]\\[%{DATA:log.level}\\]  %{GREEDYDATA:system.syslog.message}"
                 ],
 ...

I have created a new template so setup :

PUT _template\logs-software
{
    "order" : 1,
    "index_patterns" : [
      "logs-software-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "logs-software",
          "rollover_alias" : "logs-software"
        },
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_routing_shards" : "30",
        "number_of_shards": "1",
        "number_of_replicas": "0",
        "query" : {
          "default_field" : [
            "message",
            "agent.name",
            "log.level",
            "process.name",
            "process.version",
            "process.customer",
            "log.file.path",
            "input.type",
            "fields.*"
          ]
        }
      }
    },
    "mappings" : {
      "dynamic_templates" : [
        {
        "string_as_keyword": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword",
            "doc_values": "true"
          }
        }
      }
      ],
      "properties" : {
        "agent" : {
          "properties" : {
            "hostname" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            }
          }
        },
        "system" : {
          "properties" : {
            "syslog" : {
              "properties" : { }
            }
          }
        },
        "log" : {
          "properties" : {
            "original" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "file" : {
              "properties" : {
                "path" : {
                  "ignore_above" : 1024,
                  "type" : "keyword"
                }
              }
            },
            "offset" : {
              "type" : "long"
            },
            "level" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "flags" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "source" : {
              "properties" : {
                "address" : {
                  "ignore_above" : 1024,
                  "type" : "keyword"
                }
              }
            }
          }
        },
        "syslog" : {
          "properties" : {
            "priority" : {
              "type" : "long"
            },
            "facility" : {
              "type" : "long"
            },
            "severity_label" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "facility_label" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            }
          }
        },
        "process" : {
          "properties" : {
            "args" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "customer": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "pid" : {
              "type" : "long"
            }
          }
        },
        "message" : {
          "norms" : false,
          "type" : "text"
        },
        "@timestamp" : {
          "type" : "date"
        }
      }
    }
  }

I did some tests on grok debugger which is in Kibana to be sur taht the format is correct.

But when I check my in visualize in kibana the parsing seems not working, I did many differents things, but nothing is working, so I would like to get some help :slight_smile:

please find below the result in kibana :

{
  "_index": "logs-software-pression-2019.07.11",
  "_type": "_doc",
  "_id": "OTzf4WsB",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-07-11T16:30:45.309Z",
    "message": "[test0][pression][3.10.2][vpfr][ERR] 2019/07/11 07:42:19 MASTER-PRESSION.go:122: <MSG MASTER> > [ 2019-07-11 07:42:19.824120057 +0200 CEST m=+650.565875227 ]",
    "input": {
      "type": "log"
    },
    "fields": {
      "origin": "logs-software-pression"
    },
    "ecs": {
      "version": "1.0.0"
    },
    "host": {
      "name": "test0"
    },
    "agent": {
      "ephemeral_id": "de8b0eb0-5076-432a-942a-65de23632814",
      "hostname": "test0",
      "id": "da7ed90a-f9f5-47bd-89b",
      "version": "7.2.0",
      "type": "filebeat"
    },
    "log": {
      "offset": 166,
      "file": {
        "path": "/home/log_services/pression/test.log"
      }
    }
  },
  "fields": {
    "@timestamp": [
      "2019-07-11T16:30:45.309Z"
    ]
  },
  "sort": [
    1562862645309
  ]
}

As you can see my fields does not appear (process.customer, process.version, etc...)

Have you got any idea ?

Thank you

up :slight_smile:

Hi @Eowin and welcome to discuss :slight_smile:

When you create your own custom pipeline you need to install it and tell filebeat to use it. It seems you already installed it, you may need to add pipeline: 'logs-software' to your input definition so it uses this pipeline.
If it doesn't work, could you share your filebeat configuration?

Hi @jsoriano,

Thank you for you response, but I think I don't have to create a new pipeline, because I already use the default pipeline which is in 'syslog' module, I just added a new pattern in grok.

Please find below my filebeat.yml :

###################### Filebeat Configuration Example #########################
#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /home/log_services/pression/test.log
  fields:
    origin: "logs-software-pression"

- type: log
  enabled: true
  paths:
    - /var/log/*.log
#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

#==================================== ILM =====================================

setup.ilm.enabled: true
setup.ilm.policy_file: /etc/filebeat/ilm.json

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ['elastic.home.com:9200']
  indices:
    - index: '%{[fields.origin]}-%{+yyyy.MM.dd}'
      when:
        equals:
          fields.origin: "logs-software-pression"

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "beats"
  password: "${beats}"


logging.metrics.enabled: false

#============================== Xpack Monitoring ===============================
monitoring.enabled: true
monitoring.elasticsearch:

Nobody can help me ?