Ingest pipeline is not being used in the events sent by filebeat


(Jerald Sabu M) #1

I have an issue where Ingest pipeline is not being used in the events sent by filebeat.

I have created an ingest pipeline with simple grok patterns using put ingest API.
and I use filebeat to sent events to elasticsearch.

Pipeline view from the kibana dev tools is shown below:

GET _ingest/pipeline/console_log

{
"console_log": {
"description": "Ingest pipeline for Console Log Format",
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{TIMESTAMP_ISO8601:jvm_timestamp} [?%{LOGLEVEL:level}]? %{GREEDYDATA:detail}"
]
}
},
{
"date": {
"timezone": "Europe/Berlin",
"field": "jvm_timestamp",
"formats": [
"ISO8601"
]
}
}
]
}
}

My file beat configuration is the following:

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

  • input_type: log
    paths:

    • /ABC/jvm/log/tomcat/console*.log
      document_type: tomcat_console
      fields:
      environment: ABC-DEV
      instance: ABC
      application: tomcat
      fields_under_root: true
      multiline:
      pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}+[0-9]{4})'
      negate: true
      match: after
  • input_type: log
    paths:

    • /ABC/solr/server/logs/solr.log
      document_type: solr_log
      fields:
      environment: ABC-DEV
      instance: ABC
      application: Solr
      fields_under_root: true
      multiline:
      pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}+[0-9]{4})'
      negate: true
      match: after

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
hosts: ["192.168.0.1:9200"]
pipelines:
- pipeline: console_log
when.equals:
fields.application: "Tomcat"
- pipeline: console_log
when.equals:
fields.application: "Solr"

My logs starts in the following format "2018-02-20T10:48:30.085+0100 INFO....."

I am supposed to see the new fields like jvm_timestamp and level in kibana, but I cannot find them.
Any help would be appreciated.


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.