# Trouble using ingest pipeline to parse two different log formats

**URL:** https://discuss.elastic.co/t/trouble-using-ingest-pipeline-to-parse-two-different-log-formats/67819
**Category:** Elasticsearch
**Created:** [December 2, 2016, 2:31am UTC](https://discuss.elastic.co/t/trouble-using-ingest-pipeline-to-parse-two-different-log-formats/67819 "2016-12-02T02:31:12Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Sam\_Barham](https://avatars.discourse-cdn.com/v4/letter/s/e495f1/32.png) [@Sam\_Barham](https://discuss.elastic.co/u/Sam_Barham)
#### Post date: [December 2, 2016, 2:31am UTC](https://discuss.elastic.co/t/trouble-using-ingest-pipeline-to-parse-two-different-log-formats/67819/1 "2016-12-02T02:31:13Z")

</div>

As I understand it, ingest pipelines should be able to parse two different formats of log at once, by supplying multiple formats to the grok pattern list. I've done that, but I just can't seem to get it working, even with lots of fiddling.

Two example log lines:

> 2016-12-02T02:14:43.094093+00:00 | \<daemon.err\> | localhost | f343f43a3e6c[392]: | [time="2016-12-02T02:14:43Z" level=info msg="Stuff happening."]

> 2016-12-02T02:17:01.972174+00:00 | \<cron.info\> | localhost | CRON[13747]: | [(root) CMD ( cd / && run-parts --report /etc/cron.hourly)]

my ingest pipeline:

PUT \_ingest/pipeline/mypipeline  
{  
"description" : "Parse logs",  
"processors" : [  
{  
"grok" : {  
"field": "message",  
"patterns" : [  
"%{TIMESTAMP\_ISO8601} \| \<%{DATA:facility}.%{LOGLEVEL}\> \| %{SYSLOGHOST:logsource} \| %{SYSLOGPROG}: \| \[[\s]_time="%{TIMESTAMP\_ISO8601:time}" level=%{LOGLEVEL:loglevel} msg="%{DATA:msg}"]",  
"%{TIMESTAMP\_ISO8601:time} \| \<%{DATA:facility}.%{LOGLEVEL:loglevel}\> \| %{SYSLOGHOST:logsource} \| %{SYSLOGPROG}: \| \[[\s]_%{DATA:msg}]"]  
}  
}  
]  
}

If I use that pipeline, nothing gets through. If I remove the first of the patterns, logs get through, but the first kind have a msg field of 'time="2016-12-02T02:14:43Z" level=info msg="Stuff happening."'. Obviously, I'd prefer to actually parse that out rather than just leaving it as a lump. I've tried lots of things, such as GREEDYDATA or DATA, turning it into one pattern with a big "option1|option2" section, adding a second grok processor to parse the lump etc, and nothing seems to help. Any ideas?

---

<div class="post-metadata">

### Author: ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [December 5, 2016, 1:04pm UTC](https://discuss.elastic.co/t/trouble-using-ingest-pipeline-to-parse-two-different-log-formats/67819/2 "2016-12-05T13:04:08Z")

</div>

Hey,

can you try to put your data and your pipeline together into the [ingest simulate API](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/12.1/api-reference.html#api-ingest-simulate), so we can also see the output?

--Alex

---

<div class="post-metadata">

### Author: ![Sam\_Barham](https://avatars.discourse-cdn.com/v4/letter/s/e495f1/32.png) [@Sam\_Barham](https://discuss.elastic.co/u/Sam_Barham)
#### Post date: [December 6, 2016, 12:36am UTC](https://discuss.elastic.co/t/trouble-using-ingest-pipeline-to-parse-two-different-log-formats/67819/3 "2016-12-06T00:36:39Z")

</div>

Thanks for the hint about ingest simulate. I managed to figure out a solution using that. In the following, I parse out either version, then further parse the one with more values within 'msg'. I've got no idea why that works by my other attempts didn't, but at least it does work

> POST \_ingest/pipeline/\_simulate  
> {  
> "pipeline" : {  
> "processors" : [  
> {  
> "grok" : {  
> "trace\_match": true,  
> "field": "message",  
> "patterns" : [  
> "%{GREEDYDATA:msg}"  
> ]  
> }  
> },  
> {  
> "grok" : {  
> "field": "msg",  
> "patterns" : [  
> "[\s]_time="%{TIMESTAMP\_ISO8601:time}" level=%{LOGLEVEL:loglevel} msg="%{DATA:msg}"",  
> "[\s]_%{DATA:msg}"  
> ]  
> }  
> }  
> ]  
> },  
> "docs" : [  
> {  
> "\_source": {  
> "message" : "(root) CMD ( cd / && run-parts --report /etc/cron.hourly)"  
> }  
> },  
> {  
> "\_source": {  
> "message" : "time="2016-12-02T02:14:43Z" level=info msg="Stuff happening.""  
> }  
> }  
> ]  
> }

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 3, 2017, 12:37am UTC](https://discuss.elastic.co/t/trouble-using-ingest-pipeline-to-parse-two-different-log-formats/67819/4 "2017-01-03T00:37:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.