# How to find error from logs using regex pattern?

**URL:** <https://discuss.elastic.co/t/how-to-find-error-from-logs-using-regex-pattern/128099>\
**Category:** Logstash\
**Created:** [April 16, 2018, 4:18am UTC](https://discuss.elastic.co/t/how-to-find-error-from-logs-using-regex-pattern/128099 "2018-04-16T04:18:33Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![RameshNagargoje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshnagargoje/32/30212_2.png) [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\
**Post date:** [April 16, 2018, 4:18am UTC](https://discuss.elastic.co/t/how-to-find-error-from-logs-using-regex-pattern/128099/1 "2018-04-16T04:18:33Z")

</div>

regex Pattern which is saved in the local drive-in abc.json file and input of logs comes from Kafka server and send only error log snippet to elasticsearch if errors are not matched in abc.json then skip dumping of json in ES

here is abc.json

* * *

```
{
    "tag" : "CPU_Bad_Mode",
    "module": "syssw",
    "regex" : "Bad mode in Error handler detected",
    "regexFlag": 2,
    "file": "dmesg",
    "priority": 0
},
{

    "tag" : "CPU_CLK_Counter_Read_Error",
    "module": "syssw",
    "regex" : "Error in reading cpu clk counters before delay",
    "regexFlag": 2,
    "file": "dmesg",
    "priority": 0
},
{
    "tag" : "CPU_CLK_Set_Failed",
    "module": "syssw",
    "regex" : "failed to set pll parent clock",
    "regexFlag": 2,
    "file": "dmesg",
    "priority": 0
},
{
    "tag" : "PCIe_Bus_Error",
    "module": "syssw",
    "regex" : "PCIe Bus Error",
    "regexFlag": 2,
    "file": "dmesg",
    "priority": 0
}

```

* * *

and Here is input json whch come from kafka

* * *

{

"testSuite": "rts",  
"logSnippet": [  
type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",  
"[Fri Apr 13 18:08:15 2018] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, id=0008(Requester ID)",

```
],
"branch": "abc",
"shim": null,
"timestamp": "2018-04-13 18:40:31",
"feedbackDesc": null,
"hostName": "ramesh-nagargoje"

```

## }

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 16, 2018, 5:56am UTC](https://discuss.elastic.co/t/how-to-find-error-from-logs-using-regex-pattern/128099/2 "2018-04-16T05:56:56Z")

</div>

There's no plugin that does exactly what you want. I suggest you generate a Logstash configuration with abc.json as input. Actually, a translate filter would probably work. Then you can generate e.g. a CSV file with the abc.json data and Logstash will automatically pick up any changes.

---

<div class="post-metadata">

**Author:** ![RameshNagargoje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshnagargoje/32/30212_2.png) [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\
**Post date:** [April 16, 2018, 6:51am UTC](https://discuss.elastic.co/t/how-to-find-error-from-logs-using-regex-pattern/128099/3 "2018-04-16T06:51:41Z")

</div>

Can we get two input using config file one is abc.json and another is logs which comes from kafka?  
If yes then how can we match errors using regex and dumps in ES?.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 16, 2018, 7:05am UTC](https://discuss.elastic.co/t/how-to-find-error-from-logs-using-regex-pattern/128099/4 "2018-04-16T07:05:04Z")

</div>

You can certainly have two inputs in the same Logstash configuration but you can't use it to accomplish what you want in this case.

---

<div class="post-metadata">

**Author:** ![RameshNagargoje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshnagargoje/32/30212_2.png) [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\
**Post date:** [April 16, 2018, 12:37pm UTC](https://discuss.elastic.co/t/how-to-find-error-from-logs-using-regex-pattern/128099/5 "2018-04-16T12:37:37Z")

</div>

Thanks, @magnusbaeck, Can you suggest what should I use to accomplish What I want?  
any help would be appreciated.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 16, 2018, 12:38pm UTC](https://discuss.elastic.co/t/how-to-find-error-from-logs-using-regex-pattern/128099/6 "2018-04-16T12:38:16Z")

</div>

I already made a suggestion in my first reply.

---

<div class="post-metadata">

**Author:** ![RameshNagargoje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshnagargoje/32/30212_2.png) [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\
**Post date:** [April 17, 2018, 4:32am UTC](https://discuss.elastic.co/t/how-to-find-error-from-logs-using-regex-pattern/128099/7 "2018-04-17T04:32:13Z")

</div>

yes, I can get input abc.json and translate it in to .csv format. But I also want to get input from Kafka and use that abc.json as a filter to parse error and dump it in ES.

If I get abc.json and Kafka json input then, how I can differentiate messages from between them?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 17, 2018, 10:12am UTC](https://discuss.elastic.co/t/how-to-find-error-from-logs-using-regex-pattern/128099/8 "2018-04-17T10:12:34Z")

</div>

I'm not saying you should read abc.json with a file input. I'm saying that you should use a translate filter to process your Kafka messages. That translate filter should read a file in a compatible format that you have generated from abc.json. It won't be able to read abc.json in its current format.

---

<div class="post-metadata">

**Author:** ![RameshNagargoje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshnagargoje/32/30212_2.png) [@RameshNagargoje](https://discuss.elastic.co/u/RameshNagargoje)\
**Post date:** [April 18, 2018, 4:25am UTC](https://discuss.elastic.co/t/how-to-find-error-from-logs-using-regex-pattern/128099/9 "2018-04-18T04:25:08Z")

</div>

thanks @magnusbaeck

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 16, 2018, 4:25am UTC](https://discuss.elastic.co/t/how-to-find-error-from-logs-using-regex-pattern/128099/10 "2018-05-16T04:25:09Z")

</div>

Closed on May 16, 2018, 4:25am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.