Apply multiple pipelines on logs by configuring filebeat config file (.yml)

Hello world,

I'm trying to apply ingest pipelines simultaneously on my logs before they get into Elasticsearch.
My logs must go through a specific pipeline (A or B, configured in 'inputs' area) depending on where they are harvested (this step works).
Then my logs must go through a last pipeline (C, configured in output area) with a "when"-condition. But this last step doesn't work, it seems that my logs don't go through this las pipeline (C) according to my kibana devTools results. Moreover, I don't have any error message logged on my Win command :confused:

My current configs :

#=========================== Filebeat inputs =============================
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - C:\My_logs_A\*.log
    pipeline: pipeline_A

  - type: log
    enabled: true
    paths:
      - C:\My_logs_B\*.log
    pipeline: pipeline_B

#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

#============================== Kibana =====================================
setup.kibana:
  host: "localhost:5601"

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
hosts: ["localhost:9200"]
  pipelines:     
    - pipeline: pipeline_C
      when.contains.msg: "{"
  indices:
    - index: "My_logs-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template:
  enabled: false


Could you please share your debug logs (output of ./filebeat -e -d "*")?

of course,
(debug logs pasted here start form the pipelines processing because of size limitation...)

2020-03-18T00:10:52.196+0100    DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-03-17T23:10:52.196Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.6.0",
    "pipeline": "pipeline_suitsfusion"
  },
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "host": {
    "name": "TAHAA"
  },
  "agent": {
    "ephemeral_id": "27864aef-8e5b-43d2-a5eb-e0d06a6a5a88",
    "hostname": "TAHAA",
    "id": "090429d2-b9eb-4120-84ab-40588ea5b9a7",
    "version": "7.6.0",
    "type": "filebeat"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "C:\\Users\\vernaudon\\Documents\\logs_suit_fusion\\format_suitsFusion.log"
    }
  },
  "message": "INFO ;2020-04-30 09:24:40;[thread-1];{\"Etape\":\"GROUP\",\"Id12c\":123456789453,\"Date\":20200308102542,\"Banc\":\"Banc_1\",\"Moteur\":\"Leap-1A\"} Déplacement des fichiers vers la zone de stockage"
}
2020-03-18T00:10:52.196+0100    DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-03-17T23:10:52.196Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.6.0",
    "pipeline": "pipeline_suitsfusion"
  },
  "log": {
    "offset": 185,
    "file": {
      "path": "C:\\Users\\vernaudon\\Documents\\logs_suit_fusion\\format_suitsFusion.log"
    }
  },
  "message": "WARN ;2019-07-15 05:03:35;[thread-2];Vous n'êtes pas inscris au foot du vendredi",
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "host": {
    "name": "TAHAA"
  },
  "agent": {
    "type": "filebeat",
    "ephemeral_id": "27864aef-8e5b-43d2-a5eb-e0d06a6a5a88",
    "hostname": "TAHAA",
    "id": "090429d2-b9eb-4120-84ab-40588ea5b9a7",
    "version": "7.6.0"
  }
}
2020-03-18T00:10:52.197+0100    DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-03-17T23:10:52.197Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.6.0",
    "pipeline": "pipeline_suitsfusion"
  },
  "input": {
    "type": "log"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "host": {
    "name": "TAHAA"
  },
  "agent": {
    "type": "filebeat",
    "ephemeral_id": "27864aef-8e5b-43d2-a5eb-e0d06a6a5a88",
    "hostname": "TAHAA",
    "id": "090429d2-b9eb-4120-84ab-40588ea5b9a7",
    "version": "7.6.0"
  },
  "log": {
    "offset": 268,
    "file": {
      "path": "C:\\Users\\vernaudon\\Documents\\logs_suit_fusion\\format_suitsFusion.log"
    }
  },
  "message": "ERROR;2017-01-27 16:47:12;[thread-3];{\"Etape\":\"CLEAN\",\"Id12c\":789556211476,\"Date\":20200308102542,\"Banc\":\"Banc_3\",\"Moteur\":\"Leap-1A\"} Clément a encore oublié de remplir VP"
}
2020-03-18T00:10:52.197+0100    DEBUG   [harvester]     log/log.go:107  End of file reached: C:\Users\vernaudon\Documents\logs_suit_fusion\format_suitsFusion.log; Backoff now.
2020-03-18T00:10:52.199+0100    DEBUG   [registrar]     registrar/registrar.go:404      Registry file updated. 1 states written.
2020-03-18T00:10:53.197+0100    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://localhost:9200))
2020-03-18T00:10:53.197+0100    DEBUG   [elasticsearch] elasticsearch/client.go:733     ES Ping(url=http://localhost:9200)
2020-03-18T00:10:53.199+0100    DEBUG   [harvester]     log/log.go:107  End of file reached: C:\Users\vernaudon\Documents\logs_suit_fusion\format_suitsFusion.log; Backoff now.
2020-03-18T00:10:53.200+0100    DEBUG   [elasticsearch] elasticsearch/client.go:756     Ping status code: 200
2020-03-18T00:10:53.200+0100    INFO    elasticsearch/client.go:757     Attempting to connect to Elasticsearch version 7.5.2
2020-03-18T00:10:53.201+0100    DEBUG   [elasticsearch] elasticsearch/client.go:775     GET http://localhost:9200/_license?human=false  <nil>
2020-03-18T00:10:53.214+0100    DEBUG   [license]       licenser/check.go:31    Checking that license covers %sBasic
2020-03-18T00:10:53.214+0100    INFO    [license]       licenser/es_callback.go:50      Elasticsearch license: Basic
2020-03-18T00:10:53.214+0100    DEBUG   [elasticsearch] elasticsearch/client.go:775     GET http://localhost:9200/  <nil>
2020-03-18T00:10:53.217+0100    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://localhost:9200)) established
2020-03-18T00:10:53.332+0100    DEBUG   [elasticsearch] elasticsearch/client.go:354     PublishEvents: 3 events have been published to elasticsearch in 114.9419ms.
2020-03-18T00:10:53.332+0100    DEBUG   [publisher]     memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 3]
2020-03-18T00:10:53.333+0100    DEBUG   [publisher]     memqueue/eventloop.go:535       broker ACK events: count=3, start-seq=1, end-seq=3

2020-03-18T00:10:53.333+0100    DEBUG   [publisher]     memqueue/ackloop.go:128 ackloop: return ack to broker loop:3
2020-03-18T00:10:53.333+0100    DEBUG   [acker] beater/acker.go:64      stateful ack    {"count": 3}
2020-03-18T00:10:53.334+0100    DEBUG   [registrar]     registrar/registrar.go:356      Processing 3 events
2020-03-18T00:10:53.333+0100    DEBUG   [publisher]     memqueue/ackloop.go:131 ackloop:  done send ack
2020-03-18T00:10:53.334+0100    DEBUG   [registrar]     registrar/registrar.go:326      Registrar state updates processed. Count: 3
2020-03-18T00:10:53.334+0100    DEBUG   [registrar]     registrar/registrar.go:411      Write registry file: D:\Programs\Elastic_Stack\filebeat-7.6.0-windows-x86_64\data\registry\filebeat\data.json (1)
2020-03-18T00:10:53.337+0100    DEBUG   [registrar]     registrar/registrar.go:404      Registry file updated. 1 states written.
2020-03-18T00:10:55.199+0100    DEBUG   [harvester]     log/log.go:107  End of file reached: C:\Users\vernaudon\Documents\logs_suit_fusion\format_suitsFusion.log; Backoff now.
2020-03-18T00:10:59.199+0100    DEBUG   [harvester]     log/log.go:107  End of file reached: C:\Users\vernaudon\Documents\logs_suit_fusion\format_suitsFusion.log; Backoff now.
2020-03-18T00:11:02.153+0100    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31,"time":{"ms":31}},"total":{"ticks":265,"time":{"ms":265},"value":265},"user":{"ticks":234,"time":{"ms":234}}},"handles":{"open":286},"info":{"ephemeral_id":"27864aef-8e5b-43d2-a5eb-e0d06a6a5a88","uptime":{"ms":30182}},"memstats":{"gc_next":8476592,"memory_alloc":5497656,"memory_total":13930088,"rss":37019648},"runtime":{"goroutines":36}},"filebeat":{"events":{"added":6,"done":6},"harvester":{"files":{"168db991-70ec-4322-b335-2f28559bc22e":{"last_event_published_time":"2020-03-18T00:10:52.197Z","last_event_timestamp":"2020-03-18T00:10:52.197Z","name":"C:\\Users\\vernaudon\\Documents\\logs_suit_fusion\\format_suitsFusion.log","read_offset":442,"size":442,"start_time":"2020-03-18T00:10:52.188Z"}},"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":3,"batches":1,"total":3},"read":{"bytes":1561},"type":"elasticsearch","write":{"bytes":2497}},"pipeline":{"clients":2,"events":{"active":0,"filtered":3,"published":3,"retry":3,"total":6},"queue":{"acked":3}}},"registrar":{"states":{"cleanup":1,"current":1,"update":6},"writes":{"success":4,"total":4}},"system":{"cpu":{"cores":12}}}}}