# Hello elastic - Multiple Pipelines **URL:** \ **Category:** Elasticsearch\ **Tags:** docker, ingest-pipeline\ **Created:** [March 1, 2022, 9:04am UTC](https://discuss.elastic.co/t/hello-elastic-multiple-pipelines/298484 "2022-03-01T09:04:20Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![jie](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@jie](https://discuss.elastic.co/u/jie)\ **Post date:** [March 1, 2022, 9:04am UTC](https://discuss.elastic.co/t/hello-elastic-multiple-pipelines/298484/1 "2022-03-01T09:04:20Z")
```auto DELETE index PUT index/_doc/1 { "foo": "bar" } GET index/_search { "query": { "match": { "foo": "bar" } } } ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 1, 2022, 9:04am UTC](https://discuss.elastic.co/t/hello-elastic-multiple-pipelines/298484/2 "2022-03-01T09:04:42Z")
G'Day. What can we help with? ---
**Author:** ![jie](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@jie](https://discuss.elastic.co/u/jie)\ **Post date:** [March 1, 2022, 9:09am UTC](https://discuss.elastic.co/t/hello-elastic-multiple-pipelines/298484/3 "2022-03-01T09:09:07Z")
Hello warkolm Can multiple pipelines be configured for one log message in Filebeat. yml ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 1, 2022, 2:34pm UTC](https://discuss.elastic.co/t/hello-elastic-multiple-pipelines/298484/5 "2022-03-01T14:34:16Z")
No only a single pipeline per input type. Here are some thoughts on a solution. > [@How to configure multiple ingest pipelines on filebeat to load data in elastic cloud](https://discuss.elastic.co/t/how-to-configure-multiple-ingest-pipelines-on-filebeat-to-load-data-in-elastic-cloud/270290/4): > > [Here](https://discuss.elastic.co/t/setting-multiple-pipelines-for-elasticserach-output/269999/2) is a method / suggestion The advantage of doing it this way having a top level common pipeline that then directs and sorts to the correct specific pipeline is that you can control this outside of beat so if you change something you don't need to change the beats config. You can also build modular pipelines so they can be reused. You can also define a pipeline per input ... See [here](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#_pipeline_11)... ---
**Author:** ![jie](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@jie](https://discuss.elastic.co/u/jie)\ **Post date:** [March 2, 2022, 1:47am UTC](https://discuss.elastic.co/t/hello-elastic-multiple-pipelines/298484/6 "2022-03-02T01:47:57Z")
hi How do I write multiple Groks to parse a log file ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 2, 2022, 2:57am UTC](https://discuss.elastic.co/t/hello-elastic-multiple-pipelines/298484/7 "2022-03-02T02:57:28Z")
You should really read this Page / Doc entirely [Here](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) See the Docs [Here](https://www.elastic.co/guide/en/elasticsearch/reference/current/grok-processor.html) You can put multiple Groks in a single grok processor Example ```auto POST _ingest/pipeline/_simulate { "pipeline": { "description": "...", "processors": [ { "grok": { "field": "message", "patterns": [ "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes:int} %{NUMBER:duration:double}", "%{IP:client} - %{WORD:level} - %{NUMBER:bytes:int} %{DATA:message_detail}" ] } } ] }, "docs": [ { "_source": { "message": "55.3.244.1 GET /index.html 15824 0.043" } }, { "_source": { "message": "55.3.244.1 - WARN - 932 - Can not connect to server " } } ] } ``` result ```auto { "docs" : [ { "doc" : { "_index" : "_index", "_id" : "_id", "_source" : { "duration" : 0.043, "request" : "/index.html", "method" : "GET", "bytes" : 15824, "client" : "55.3.244.1", "message" : "55.3.244.1 GET /index.html 15824 0.043" }, "_ingest" : { "timestamp" : "2022-03-02T02:28:54.1600216Z" } } }, { "doc" : { "_index" : "_index", "_id" : "_id", "_source" : { "level" : "WARN", "bytes" : 932, "client" : "55.3.244.1", "message" : "55.3.244.1 - WARN - 932 - Can not connect to server ", "message_detail" : "" }, "_ingest" : { "timestamp" : "2022-03-02T02:28:54.1600448Z" } } } ] } ``` Or you can create something like this if you have a tag or field you can use see [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#conditionally-apply-pipelines) ```auto PUT _ingest/pipeline/log-type1 { "description": "...", "processors": [ { "grok": { "field": "message", "patterns": [ "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes:int} %{NUMBER:duration:double}" ] } } ] } PUT _ingest/pipeline/log-type2 { "description": "...", "processors": [ { "grok": { "field": "message", "patterns": [ "%{IP:client} - %{WORD:level} - %{NUMBER:bytes:int} %{DATA:message_detail}" ] } } ] } PUT _ingest/pipeline/top-level-log-pipeline { "description": "...", "processors": [ { "pipeline": { "description": "If logtype web-log", "if": "ctx.type == 'web-log'", "name": "log-type1" } }, { "pipeline": { "description": "If logtype app-log", "if": "ctx.type == 'app-log'", "name": "log-type2" } } ] } POST _ingest/pipeline/top-level-log-pipeline/_simulate { "docs": [ { "_source": { "type" : "web-log", "message": "55.3.244.1 GET /index.html 15824 0.043" } }, { "_source": { "type" : "app-log", "message": "55.3.244.1 - WARN - 932 - Can not connect to server " } } ] } ``` results ```auto { "docs" : [ { "doc" : { "_index" : "_index", "_id" : "_id", "_source" : { "duration" : 0.043, "request" : "/index.html", "method" : "GET", "bytes" : 15824, "client" : "55.3.244.1", "type" : "web-log", "message" : "55.3.244.1 GET /index.html 15824 0.043" }, "_ingest" : { "timestamp" : "2022-03-02T02:55:50.6633169Z" } } }, { "doc" : { "_index" : "_index", "_id" : "_id", "_source" : { "level" : "WARN", "bytes" : 932, "client" : "55.3.244.1", "type" : "app-log", "message" : "55.3.244.1 - WARN - 932 - Can not connect to server ", "message_detail" : "" }, "_ingest" : { "timestamp" : "2022-03-02T02:55:50.6633322Z" } } } ] } ``` ---
**Author:** ![jie](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@jie](https://discuss.elastic.co/u/jie)\ **Post date:** [March 2, 2022, 3:00am UTC](https://discuss.elastic.co/t/hello-elastic-multiple-pipelines/298484/8 "2022-03-02T03:00:04Z")
Thank you very much ---
**Author:** ![jie](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@jie](https://discuss.elastic.co/u/jie)\ **Post date:** [March 2, 2022, 6:24am UTC](https://discuss.elastic.co/t/hello-elastic-multiple-pipelines/298484/9 "2022-03-02T06:24:35Z")
hi Sorry to bother you [The problem](https://discuss.elastic.co/t/there-is-no-grok-expression-that-can-parse-100-of-the-following-log-information/298462/3) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 2, 2022, 6:29am UTC](https://discuss.elastic.co/t/hello-elastic-multiple-pipelines/298484/10 "2022-03-02T06:29:38Z")
No easy solution you are going to need to write a grok for each pattern...and put them all the grok processor patterns array like in the first example above. You should define the most common first and then next most common and continue to iterate ---
**Author:** ![jie](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@jie](https://discuss.elastic.co/u/jie)\ **Post date:** [March 2, 2022, 6:45am UTC](https://discuss.elastic.co/t/hello-elastic-multiple-pipelines/298484/11 "2022-03-02T06:45:45Z")
Ok, thanks for your help ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 30, 2022, 6:45am UTC](https://discuss.elastic.co/t/hello-elastic-multiple-pipelines/298484/12 "2022-03-30T06:45:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.