# Apply multiple pipelines on logs by configuring filebeat config file (.yml)
**URL:** https://discuss.elastic.co/t/apply-multiple-pipelines-on-logs-by-configuring-filebeat-config-file-yml/223944
**Category:** Beats
**Tags:** filebeat
**Created:** [March 17, 2020, 2:05pm UTC](https://discuss.elastic.co/t/apply-multiple-pipelines-on-logs-by-configuring-filebeat-config-file-yml/223944 "2020-03-17T14:05:26Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@oula\_oula](https://discuss.elastic.co/u/oula_oula)
#### Post date: [March 17, 2020, 2:05pm UTC](https://discuss.elastic.co/t/apply-multiple-pipelines-on-logs-by-configuring-filebeat-config-file-yml/223944/1 "2020-03-17T14:05:26Z")
Hello world,
I'm trying to apply ingest pipelines simultaneously on my logs before they get into Elasticsearch.
My logs must go through a specific pipeline (A or B, configured in 'inputs' area) depending on where they are harvested (this step works).
Then my logs must go through a last pipeline (C, configured in output area) with a "when"-condition. But this last step doesn't work, it seems that my logs don't go through this las pipeline (C) according to my kibana devTools results. Moreover, I don't have any error message logged on my Win command 😕
My current configs :
```auto
#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
enabled: true
paths:
- C:\My_logs_A\*.log
pipeline: pipeline_A
- type: log
enabled: true
paths:
- C:\My_logs_B\*.log
pipeline: pipeline_B
#============================= Filebeat modules ===============================
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
#============================== Kibana =====================================
setup.kibana:
host: "localhost:5601"
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
hosts: ["localhost:9200"]
pipelines:
- pipeline: pipeline_C
when.contains.msg: "{"
indices:
- index: "My_logs-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template:
enabled: false
```
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [March 17, 2020, 6:55pm UTC](https://discuss.elastic.co/t/apply-multiple-pipelines-on-logs-by-configuring-filebeat-config-file-yml/223944/2 "2020-03-17T18:55:55Z")
Could you please share your debug logs (output of `./filebeat -e -d "*"`)?
---
### Author:  [@oula\_oula](https://discuss.elastic.co/u/oula_oula)
#### Post date: [March 17, 2020, 11:18pm UTC](https://discuss.elastic.co/t/apply-multiple-pipelines-on-logs-by-configuring-filebeat-config-file-yml/223944/3 "2020-03-17T23:18:01Z")
of course,
(debug logs pasted here start form the pipelines processing because of size limitation...)
```auto
2020-03-18T00:10:52.196+0100 DEBUG [processors] processing/processors.go:186 Publish event: {
"@timestamp": "2020-03-17T23:10:52.196Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.6.0",
"pipeline": "pipeline_suitsfusion"
},
"input": {
"type": "log"
},
"ecs": {
"version": "1.4.0"
},
"host": {
"name": "TAHAA"
},
"agent": {
"ephemeral_id": "27864aef-8e5b-43d2-a5eb-e0d06a6a5a88",
"hostname": "TAHAA",
"id": "090429d2-b9eb-4120-84ab-40588ea5b9a7",
"version": "7.6.0",
"type": "filebeat"
},
"log": {
"offset": 0,
"file": {
"path": "C:\\Users\\vernaudon\\Documents\\logs_suit_fusion\\format_suitsFusion.log"
}
},
"message": "INFO ;2020-04-30 09:24:40;[thread-1];{\"Etape\":\"GROUP\",\"Id12c\":123456789453,\"Date\":20200308102542,\"Banc\":\"Banc_1\",\"Moteur\":\"Leap-1A\"} Déplacement des fichiers vers la zone de stockage"
}
2020-03-18T00:10:52.196+0100 DEBUG [processors] processing/processors.go:186 Publish event: {
"@timestamp": "2020-03-17T23:10:52.196Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.6.0",
"pipeline": "pipeline_suitsfusion"
},
"log": {
"offset": 185,
"file": {
"path": "C:\\Users\\vernaudon\\Documents\\logs_suit_fusion\\format_suitsFusion.log"
}
},
"message": "WARN ;2019-07-15 05:03:35;[thread-2];Vous n'êtes pas inscris au foot du vendredi",
"input": {
"type": "log"
},
"ecs": {
"version": "1.4.0"
},
"host": {
"name": "TAHAA"
},
"agent": {
"type": "filebeat",
"ephemeral_id": "27864aef-8e5b-43d2-a5eb-e0d06a6a5a88",
"hostname": "TAHAA",
"id": "090429d2-b9eb-4120-84ab-40588ea5b9a7",
"version": "7.6.0"
}
}
2020-03-18T00:10:52.197+0100 DEBUG [processors] processing/processors.go:186 Publish event: {
"@timestamp": "2020-03-17T23:10:52.197Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.6.0",
"pipeline": "pipeline_suitsfusion"
},
"input": {
"type": "log"
},
"ecs": {
"version": "1.4.0"
},
"host": {
"name": "TAHAA"
},
"agent": {
"type": "filebeat",
"ephemeral_id": "27864aef-8e5b-43d2-a5eb-e0d06a6a5a88",
"hostname": "TAHAA",
"id": "090429d2-b9eb-4120-84ab-40588ea5b9a7",
"version": "7.6.0"
},
"log": {
"offset": 268,
"file": {
"path": "C:\\Users\\vernaudon\\Documents\\logs_suit_fusion\\format_suitsFusion.log"
}
},
"message": "ERROR;2017-01-27 16:47:12;[thread-3];{\"Etape\":\"CLEAN\",\"Id12c\":789556211476,\"Date\":20200308102542,\"Banc\":\"Banc_3\",\"Moteur\":\"Leap-1A\"} Clément a encore oublié de remplir VP"
}
2020-03-18T00:10:52.197+0100 DEBUG [harvester] log/log.go:107 End of file reached: C:\Users\vernaudon\Documents\logs_suit_fusion\format_suitsFusion.log; Backoff now.
2020-03-18T00:10:52.199+0100 DEBUG [registrar] registrar/registrar.go:404 Registry file updated. 1 states written.
2020-03-18T00:10:53.197+0100 INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://localhost:9200))
2020-03-18T00:10:53.197+0100 DEBUG [elasticsearch] elasticsearch/client.go:733 ES Ping(url=http://localhost:9200)
2020-03-18T00:10:53.199+0100 DEBUG [harvester] log/log.go:107 End of file reached: C:\Users\vernaudon\Documents\logs_suit_fusion\format_suitsFusion.log; Backoff now.
2020-03-18T00:10:53.200+0100 DEBUG [elasticsearch] elasticsearch/client.go:756 Ping status code: 200
2020-03-18T00:10:53.200+0100 INFO elasticsearch/client.go:757 Attempting to connect to Elasticsearch version 7.5.2
2020-03-18T00:10:53.201+0100 DEBUG [elasticsearch] elasticsearch/client.go:775 GET http://localhost:9200/_license?human=false
2020-03-18T00:10:53.214+0100 DEBUG [license] licenser/check.go:31 Checking that license covers %sBasic
2020-03-18T00:10:53.214+0100 INFO [license] licenser/es_callback.go:50 Elasticsearch license: Basic
2020-03-18T00:10:53.214+0100 DEBUG [elasticsearch] elasticsearch/client.go:775 GET http://localhost:9200/
2020-03-18T00:10:53.217+0100 INFO pipeline/output.go:105 Connection to backoff(elasticsearch(http://localhost:9200)) established
2020-03-18T00:10:53.332+0100 DEBUG [elasticsearch] elasticsearch/client.go:354 PublishEvents: 3 events have been published to elasticsearch in 114.9419ms.
2020-03-18T00:10:53.332+0100 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 3]
2020-03-18T00:10:53.333+0100 DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=3, start-seq=1, end-seq=3
2020-03-18T00:10:53.333+0100 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:3
2020-03-18T00:10:53.333+0100 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 3}
2020-03-18T00:10:53.334+0100 DEBUG [registrar] registrar/registrar.go:356 Processing 3 events
2020-03-18T00:10:53.333+0100 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2020-03-18T00:10:53.334+0100 DEBUG [registrar] registrar/registrar.go:326 Registrar state updates processed. Count: 3
2020-03-18T00:10:53.334+0100 DEBUG [registrar] registrar/registrar.go:411 Write registry file: D:\Programs\Elastic_Stack\filebeat-7.6.0-windows-x86_64\data\registry\filebeat\data.json (1)
2020-03-18T00:10:53.337+0100 DEBUG [registrar] registrar/registrar.go:404 Registry file updated. 1 states written.
2020-03-18T00:10:55.199+0100 DEBUG [harvester] log/log.go:107 End of file reached: C:\Users\vernaudon\Documents\logs_suit_fusion\format_suitsFusion.log; Backoff now.
2020-03-18T00:10:59.199+0100 DEBUG [harvester] log/log.go:107 End of file reached: C:\Users\vernaudon\Documents\logs_suit_fusion\format_suitsFusion.log; Backoff now.
2020-03-18T00:11:02.153+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31,"time":{"ms":31}},"total":{"ticks":265,"time":{"ms":265},"value":265},"user":{"ticks":234,"time":{"ms":234}}},"handles":{"open":286},"info":{"ephemeral_id":"27864aef-8e5b-43d2-a5eb-e0d06a6a5a88","uptime":{"ms":30182}},"memstats":{"gc_next":8476592,"memory_alloc":5497656,"memory_total":13930088,"rss":37019648},"runtime":{"goroutines":36}},"filebeat":{"events":{"added":6,"done":6},"harvester":{"files":{"168db991-70ec-4322-b335-2f28559bc22e":{"last_event_published_time":"2020-03-18T00:10:52.197Z","last_event_timestamp":"2020-03-18T00:10:52.197Z","name":"C:\\Users\\vernaudon\\Documents\\logs_suit_fusion\\format_suitsFusion.log","read_offset":442,"size":442,"start_time":"2020-03-18T00:10:52.188Z"}},"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":3,"batches":1,"total":3},"read":{"bytes":1561},"type":"elasticsearch","write":{"bytes":2497}},"pipeline":{"clients":2,"events":{"active":0,"filtered":3,"published":3,"retry":3,"total":6},"queue":{"acked":3}}},"registrar":{"states":{"cleanup":1,"current":1,"update":6},"writes":{"success":4,"total":4}},"system":{"cpu":{"cores":12}}}}}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 14, 2020, 11:18pm UTC](https://discuss.elastic.co/t/apply-multiple-pipelines-on-logs-by-configuring-filebeat-config-file-yml/223944/4 "2020-04-14T23:18:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.