Apply multiple pipelines on logs by configuring filebeat config file (.yml)

Elastic Stack Beats
1

Hello world,

I'm trying to apply ingest pipelines simultaneously on my logs before they get into Elasticsearch.
My logs must go through a specific pipeline (A or B, configured in 'inputs' area) depending on where they are harvested (this step works).
Then my logs must go through a last pipeline (C, configured in output area) with a "when"-condition. But this last step doesn't work, it seems that my logs don't go through this las pipeline (C) according to my kibana devTools results. Moreover, I don't have any error message logged on my Win command :confused:

My current configs :

#=========================== Filebeat inputs =============================
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - C:\My_logs_A\*.log
    pipeline: pipeline_A

  - type: log
    enabled: true
    paths:
      - C:\My_logs_B\*.log
    pipeline: pipeline_B

#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

#============================== Kibana =====================================
setup.kibana:
  host: "localhost:5601"

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
hosts: ["localhost:9200"]
  pipelines:     
    - pipeline: pipeline_C
      when.contains.msg: "{"
  indices:
    - index: "My_logs-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template:
  enabled: false
2

Could you please share your debug logs (output of ./filebeat -e -d "*")?

3

of course,
(debug logs pasted here start form the pipelines processing because of size limitation...)

2020-03-18T00:10:52.196+0100    DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-03-17T23:10:52.196Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.6.0",
    "pipeline": "pipeline_suitsfusion"
  },
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "host": {
    "name": "TAHAA"
  },
  "agent": {
    "ephemeral_id": "27864aef-8e5b-43d2-a5eb-e0d06a6a5a88",
    "hostname": "TAHAA",
    "id": "090429d2-b9eb-4120-84ab-40588ea5b9a7",
    "version": "7.6.0",
    "type": "filebeat"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "C:\\Users\\vernaudon\\Documents\\logs_suit_fusion\\format_suitsFusion.log"
    }
  },
  "message": "INFO ;2020-04-30 09:24:40;[thread-1];{\"Etape\":\"GROUP\",\"Id12c\":123456789453,\"Date\":20200308102542,\"Banc\":\"Banc_1\",\"Moteur\":\"Leap-1A\"} Déplacement des fichiers vers la zone de stockage"
}
2020-03-18T00:10:52.196+0100    DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-03-17T23:10:52.196Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.6.0",
    "pipeline": "pipeline_suitsfusion"
  },
  "log": {
    "offset": 185,
    "file": {
      "path": "C:\\Users\\vernaudon\\Documents\\logs_suit_fusion\\format_suitsFusion.log"
    }
  },
  "message": "WARN ;2019-07-15 05:03:35;[thread-2];Vous n'êtes pas inscris au foot du vendredi",
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "host": {
    "name": "TAHAA"
  },
  "agent": {
    "type": "filebeat",
    "ephemeral_id": "27864aef-8e5b-43d2-a5eb-e0d06a6a5a88",
    "hostname": "TAHAA",
    "id": "090429d2-b9eb-4120-84ab-40588ea5b9a7",
    "version": "7.6.0"
  }
}
2020-03-18T00:10:52.197+0100    DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-03-17T23:10:52.197Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.6.0",
    "pipeline": "pipeline_suitsfusion"
  },
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "host": {
    "name": "TAHAA"
  },
  "agent": {
    "type": "filebeat",
    "ephemeral_id": "27864aef-8e5b-43d2-a5eb-e0d06a6a5a88",
    "hostname": "TAHAA",
    "id": "090429d2-b9eb-4120-84ab-40588ea5b9a7",
    "version": "7.6.0"
  },
  "log": {
    "offset": 268,
    "file": {
      "path": "C:\\Users\\vernaudon\\Documents\\logs_suit_fusion\\format_suitsFusion.log"
    }
  },
  "message": "ERROR;2017-01-27 16:47:12;[thread-3];{\"Etape\":\"CLEAN\",\"Id12c\":789556211476,\"Date\":20200308102542,\"Banc\":\"Banc_3\",\"Moteur\":\"Leap-1A\"} Clément a encore oublié de remplir VP"
}
2020-03-18T00:10:52.197+0100    DEBUG   [harvester]     log/log.go:107  End of file reached: C:\Users\vernaudon\Documents\logs_suit_fusion\format_suitsFusion.log; Backoff now.
2020-03-18T00:10:52.199+0100    DEBUG   [registrar]     registrar/registrar.go:404      Registry file updated. 1 states written.
2020-03-18T00:10:53.197+0100    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://localhost:9200))
2020-03-18T00:10:53.197+0100    DEBUG   [elasticsearch] elasticsearch/client.go:733     ES Ping(url=http://localhost:9200)
2020-03-18T00:10:53.199+0100    DEBUG   [harvester]     log/log.go:107  End of file reached: C:\Users\vernaudon\Documents\logs_suit_fusion\format_suitsFusion.log; Backoff now.
2020-03-18T00:10:53.200+0100    DEBUG   [elasticsearch] elasticsearch/client.go:756     Ping status code: 200
2020-03-18T00:10:53.200+0100    INFO    elasticsearch/client.go:757     Attempting to connect to Elasticsearch version 7.5.2
2020-03-18T00:10:53.201+0100    DEBUG   [elasticsearch] elasticsearch/client.go:775     GET http://localhost:9200/_license?human=false  <nil>
2020-03-18T00:10:53.214+0100    DEBUG   [license]       licenser/check.go:31    Checking that license covers %sBasic
2020-03-18T00:10:53.214+0100    INFO    [license]       licenser/es_callback.go:50      Elasticsearch license: Basic
2020-03-18T00:10:53.214+0100    DEBUG   [elasticsearch] elasticsearch/client.go:775     GET http://localhost:9200/  <nil>
2020-03-18T00:10:53.217+0100    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://localhost:9200)) established
2020-03-18T00:10:53.332+0100    DEBUG   [elasticsearch] elasticsearch/client.go:354     PublishEvents: 3 events have been published to elasticsearch in 114.9419ms.
2020-03-18T00:10:53.332+0100    DEBUG   [publisher]     memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 3]
2020-03-18T00:10:53.333+0100    DEBUG   [publisher]     memqueue/eventloop.go:535       broker ACK events: count=3, start-seq=1, end-seq=3

2020-03-18T00:10:53.333+0100    DEBUG   [publisher]     memqueue/ackloop.go:128 ackloop: return ack to broker loop:3
2020-03-18T00:10:53.333+0100    DEBUG   [acker] beater/acker.go:64      stateful ack    {"count": 3}
2020-03-18T00:10:53.334+0100    DEBUG   [registrar]     registrar/registrar.go:356      Processing 3 events
2020-03-18T00:10:53.333+0100    DEBUG   [publisher]     memqueue/ackloop.go:131 ackloop:  done send ack
2020-03-18T00:10:53.334+0100    DEBUG   [registrar]     registrar/registrar.go:326      Registrar state updates processed. Count: 3
2020-03-18T00:10:53.334+0100    DEBUG   [registrar]     registrar/registrar.go:411      Write registry file: D:\Programs\Elastic_Stack\filebeat-7.6.0-windows-x86_64\data\registry\filebeat\data.json (1)
2020-03-18T00:10:53.337+0100    DEBUG   [registrar]     registrar/registrar.go:404      Registry file updated. 1 states written.
2020-03-18T00:10:55.199+0100    DEBUG   [harvester]     log/log.go:107  End of file reached: C:\Users\vernaudon\Documents\logs_suit_fusion\format_suitsFusion.log; Backoff now.
2020-03-18T00:10:59.199+0100    DEBUG   [harvester]     log/log.go:107  End of file reached: C:\Users\vernaudon\Documents\logs_suit_fusion\format_suitsFusion.log; Backoff now.
2020-03-18T00:11:02.153+0100    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31,"time":{"ms":31}},"total":{"ticks":265,"time":{"ms":265},"value":265},"user":{"ticks":234,"time":{"ms":234}}},"handles":{"open":286},"info":{"ephemeral_id":"27864aef-8e5b-43d2-a5eb-e0d06a6a5a88","uptime":{"ms":30182}},"memstats":{"gc_next":8476592,"memory_alloc":5497656,"memory_total":13930088,"rss":37019648},"runtime":{"goroutines":36}},"filebeat":{"events":{"added":6,"done":6},"harvester":{"files":{"168db991-70ec-4322-b335-2f28559bc22e":{"last_event_published_time":"2020-03-18T00:10:52.197Z","last_event_timestamp":"2020-03-18T00:10:52.197Z","name":"C:\\Users\\vernaudon\\Documents\\logs_suit_fusion\\format_suitsFusion.log","read_offset":442,"size":442,"start_time":"2020-03-18T00:10:52.188Z"}},"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":3,"batches":1,"total":3},"read":{"bytes":1561},"type":"elasticsearch","write":{"bytes":2497}},"pipeline":{"clients":2,"events":{"active":0,"filtered":3,"published":3,"retry":3,"total":6},"queue":{"acked":3}}},"registrar":{"states":{"cleanup":1,"current":1,"update":6},"writes":{"success":4,"total":4}},"system":{"cpu":{"cores":12}}}}}
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.