# Parsing datapower syslog using GROK for latency calculation for given URI

**URL:** <https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118>\
**Category:** Logstash\
**Created:** [March 8, 2018, 5:02pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118 "2018-03-08T17:02:22Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![mguttula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mguttula/32/78256_2.png) [@mguttula](https://discuss.elastic.co/u/mguttula)\
**Post date:** [March 8, 2018, 5:02pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/1 "2018-03-08T17:02:22Z")

</div>

Hi,

I'm new to GROK and Logstash.

## I'm trying to calculate the response time from latency field of a syslog entry of Datapower. Entry looks like following:

## 20180308T211418.088Z [0x80e00073][mpgw][debug] mpgw(MyServices\_MPGW): tid(12380973)[102.29.16.10] gtid(5ae25f9f5a986d770257aec3): Latency: 0 138 0 56 56 56 0 138 138 138 138 138 138 138 56 138 [[http://10.40.59.102:8070/rephierarchy/savedlist/2798](http://10.40.59.102:8070/rephierarchy/savedlist/2798)]

each string of Above entry is represented as following:  
20180301T211418.088Z - Timestamp  
{0x80e00073] - errorcode  
[mpgw]- objectype  
[debug] - loglevel  
mpgw(MyServices\_MPGW): - servicename  
tid(12380973)[102.29.16.10] - transactionid  
gtid(5ae25f9f5a986d770257aec3): - gid  
Latency: 0 138 0 56 56 56 0 138 138 138 138 138 138 138 56 138 --- latency  
[http://10.40.59.102:8070/rephierarchy/savedlist/2798](http://10.40.59.102:8070/rephierarchy/savedlist/2798)] -- URL

I'm trying to write GROK filter in logstash conf file to decode above entry and get the timestamp, latency and URI. I'll calculate response time from latency (response time =subtracting 12 position field from 6th position field)

## my logstash conf entry looks like

input {  
tcp {  
port =\> 5514  
type =\> syslog  
}  
}

filter {  
syslog\_pri {  
add\_field =\> { "[@metadata][type]" =\> "syslog" }  
add\_field =\> { "[@metadata][beat]" =\> "syslog" }  
}  
if [type] == "syslog" {  
grok {  
match =\> { "message" =\> "%{SYSLOGLINE}" }  
}  
date {  
match =\> ["timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]  
}  
}  
}

output {  
if [type] == "syslog" {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
sniffing =\> true  
manage\_template =\> false  
user =\> elastic  
password =\> defaultpassword  
index =\> "datapower-%{+YYYY.MM.dd}"  
}  
}  
}

## A sample JSON Entry in Kibana for this index:

## { "\_index": "datapower-2018.03.08", "\_type": "syslog", "\_id": "AWIGW4Ud24FGqAt6qLue", "\_version": 1, "\_score": null, "\_source": { "@timestamp": "2018-03-08T16:04:43.926Z", "syslog\_severity\_code": 5, "port": 33670, "syslog\_facility": "user-level", "@version": "1", "host": "10.40.137.99", "syslog\_facility\_code": 1, "message": "\<14\>Mar 08 11:03:11 DVI\_MyServices [0x80e00073][latency][info] mpgw(MyServices\_MPGW): trans(12362109)[102.29.16.10] gtid(5ae25f9f5aa15f0704322723): Latency: 0 138 0 56 56 56 0 138 138 138 138 138 138 138 56 138 [[http://10.40.59.102:8070/rephierarchy/savedlist/2798](http://10.40.59.102:8070/rephierarchy/savedlist/2798)]", "type": "syslog", "syslog\_severity": "notice", "tags": ["\_grokparsefailure", "\_jsonparsefailure"] }, "fields": { "@timestamp": [1520525083926] }, "sort": [1520525083926] }

I tried to use Grok debugger but I'm stuck with timestamp entry itself. I tried few options like  
%{DATESTAMP:timestamp}  
%{SYSLOGTIMESTAMP:timestamp}  
%{TIMESTAMP\_ISO8601:timestamp}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 8, 2018, 5:37pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/2 "2018-03-08T17:37:24Z")

</div>

Personally I would use dissect rather than grok.

```auto
  dissect {
    mapping => {
      message => "<%{level}>%{ts} %{+ts} %{+ts} %{unnamed} [%{errorcode}][%{loglevel}] %{servicename}: %{transactiondid} %{gid} Latency: %{latency} [%{url}]"
    }
  }
  ruby {
    # My mind is going blank on how to do this with non-ruby filter
    code => 'event.set("latencystuff", event.get("latency").split())'
  }
  mutate { convert => { "latencystuff" => "integer" } }
  date { match => ["ts", "MMM dd HH:mm:ss"] target => "@timestamp" }

```

---

<div class="post-metadata">

**Author:** ![mguttula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mguttula/32/78256_2.png) [@mguttula](https://discuss.elastic.co/u/mguttula)\
**Post date:** [March 8, 2018, 6:37pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/3 "2018-03-08T18:37:10Z")

</div>

Is it possible to ignore the fields other than timestamp, latency and URL? I'll try your suggestion.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 8, 2018, 6:45pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/4 "2018-03-08T18:45:55Z")

</div>

> [@mguttula](#):
>
> Is it possible to ignore the fields other than timestamp, latency and URL?

Yes, if you do not supply a name then dissect will not store the result.

```
message => "<%{}>%{ts} %{+ts} %{+ts} %{} [%{}][%{}] %{}: %{} %{} Latency: %{latency} [%{url}]"
```

---

<div class="post-metadata">

**Author:** ![mguttula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mguttula/32/78256_2.png) [@mguttula](https://discuss.elastic.co/u/mguttula)\
**Post date:** [March 11, 2018, 2:44pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/5 "2018-03-11T14:44:01Z")

</div>

> [@Badger](#):
>
> message =\> "\<%{}\>%{ts} %{+ts} %{+ts} %{} [%{}][%{}] %{}: %{} %{} Latency: %{latency} [%{url}]"

What does the \<%{level}\> and %{unnamed} stand for from your previous reply?

**\<%{level}\>** %{ts} %{+ts} %{+ts} **%{unnamed}** [%{errorcode}][%{loglevel}] %{servicename}: %{transactiondid} %{gid} Latency: %{latency} [%{url}]"

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 11, 2018, 3:33pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/6 "2018-03-11T15:33:47Z")

</div>

> [@mguttula](#):
>
> What does the \<%{level}\> and %{unnamed} stand for from your previous reply?

Placeholders to be replaced with better field names if you wanted to keep those fields 🙂

---

<div class="post-metadata">

**Author:** ![mguttula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mguttula/32/78256_2.png) [@mguttula](https://discuss.elastic.co/u/mguttula)\
**Post date:** [March 11, 2018, 3:41pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/7 "2018-03-11T15:41:38Z")

</div>

> [@mguttula](#):
>
> 20180308T211418.088Z [0x80e00073][mpgw][debug] mpgw(MyServices\_MPGW): tid(12380973)[102.29.16.10] gtid(5ae25f9f5a986d770257aec3): Latency: 0 138 0 56 56 56 0 138 138 138 138 138 138 138 56 138 [[http://10.40.59.102:8070/rephierarchy/savedlist/2798](http://10.40.59.102:8070/rephierarchy/savedlist/2798)]

I was wonder how the \<%{level}\> and %{unnamed} related to the below log event  
20180308T211418.088Z [0x80e00073][mpgw][debug] mpgw(MyServices\_MPGW): tid(12380973)[102.29.16.10] gtid(5ae25f9f5a986d770257aec3): Latency: 0 138 0 56 56 56 0 138 138 138 138 138 138 138 56 138 [[http://10.40.59.102:8070/rephierarchy/savedlist/2798](http://10.40.59.102:8070/rephierarchy/savedlist/2798)]

As log as the filter works I'm good. I made the changes you have recommend and I would not be able to test them until I restart ELK Stack. I'm not sure if the is expected behavior of the ELK stask to restart all (Logstash, Elasticsearch and Kibana). I have dependence with other team who is testing other functionality. I'll reboot once the other team have done testing their changes. After installing XPack, I have noticed that rebooting ELK stack doesnt reflect logstash config changes.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 11, 2018, 3:59pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/8 "2018-03-11T15:59:14Z")

</div>

```
"message": "<14>Mar 08 11:03:11 DVI_MyServices [0x80e00073][latency][info] mpgw(MyServices_MPGW): trans(12362109)[102.29.16.10] gtid(5ae25f9f5aa15f0704322723): Latency: 0 138 0 56 56 56 0 138 138 138 138 138 138 138 56 138 [http://10.40.59.102:8070/rephierarchy/savedlist/2798]"
```

In that message, level would pick up the 14 at the beginning, and unnamed would have been set to DVI\_MyServices.

If you restart logstash it will pick up the new configuration, no need to restart elasticsearch or kibana.

---

<div class="post-metadata">

**Author:** ![mguttula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mguttula/32/78256_2.png) [@mguttula](https://discuss.elastic.co/u/mguttula)\
**Post date:** [March 11, 2018, 6:05pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/9 "2018-03-11T18:05:45Z")

</div>

Now I understood what you mean by \<%{level}\> and %{unnamed}.

## Question: You still want me to GROK along dissect? or Can we skip GROK in this scenario? Below output I got is using GROK: --- GROK OUTPUT-- "\<14\>Mar 08 11:03:11 DVI\_MyServices [0x80e00073][latency][info] mpgw(MyServices\_MPGW): trans(12362109)[102.29.16.10] gtid(5ae25f9f5aa15f0704322723): Latency: 0 138 0 56 56 56 0 138 138 138 138 138 138 138 56 138 [[http://10.40.59.102:8070/rephierarchy/savedlist/2798](http://10.40.59.102:8070/rephierarchy/savedlist/2798)]"

## Without using GROK, the input from TCP to Logstash would look like the following ---- TCP input ---- 20180308T211418.088Z [0x80e00073][mpgw][debug] mpgw(MyServices\_MPGW): tid(12380973)[102.29.16.10] gtid(5ae25f9f5a986d770257aec3): Latency: 0 138 0 56 56 56 0 138 138 138 138 138 138 138 56 138 [[http://10.40.59.102:8070/rephierarchy/savedlist/2798](http://10.40.59.102:8070/rephierarchy/savedlist/2798)]

## Based our your latest response, may be you wanted me to use filter config as following:

filter {  
syslog\_pri {  
add\_field =\> { "[@metadata][type]" =\> "syslog" }  
add\_field =\> { "[@metadata][beat]" =\> "syslog" }  
}  
if [type] == "syslog" {

```
    grok {
        match => { "message" => "%{SYSLOGLINE}" }
    }
    date {
        match => ["timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
    }
    dissect {
            mapping => {
                    message => "<%{}>%{ts} %{+ts} %{+ts} %{} [%{}][%{}] %{}: %{} %{} Latency: %{latency} [%{url}]"
            }
    }
    ruby {
            code => 'event.set("latencystuff", event.get("latency").split())'
    }
    mutate { convert => { "latencystuff" => "integer" } }
    date { match => ["ts", "MMM dd HH:mm:ss"] target => "@timestamp" }
}

```

## }

I'll test above changes once a request comes from TCP (I cannot manually trigger the TCP request).

Also is it possible to get URI from URL request without query parameters using dissect ?  
For example  
dissect {  
mapping =\> {  
message =\> "\<%{}\>%{ts} %{+ts} %{+ts} %{} [%{}][%{}] %{}: %{} %{} Latency: %{latency} [%{url}]"  
"url" =\> "http://%{}:%{}/%{uri}?%{}"  
}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 12, 2018, 2:50pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/10 "2018-03-12T14:50:55Z")

</div>

> [@mguttula](#):
>
> is it possible to get URI from URL request without query parameters using dissect ?

Yes, but the pattern you gave only works if there is a query string. So you would have to test for the presence of ? to decide whether to include ?%{} in the mapping.

---

<div class="post-metadata">

**Author:** ![mguttula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mguttula/32/78256_2.png) [@mguttula](https://discuss.elastic.co/u/mguttula)\
**Post date:** [March 12, 2018, 6:58pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/11 "2018-03-12T18:58:37Z")

</div>

I made some progress and Removed GROK. I'm getting URL with "]" appended to the end. Below is my latest logstash config for message:  
"message": "\<14\>Mar 08 11:03:11 DVI\_MyServices [0x80e00073][latency][info] mpgw(MyServices\_MPGW): trans(12362109)[102.29.16.10] gtid(5ae25f9f5aa15f0704322723): Latency: 0 186 0 65 65 65 0 186 187 186 187 187 186 186 65 186 [[http://10.40.59.102:8070/appview/details?appID=125635](http://10.40.59.102:8070/appview/details?appID=125635)]"

------- logstash config -----  
input {  
tcp {  
port =\> 5514  
type =\> syslog  
}  
}

filter {  
syslog\_pri {  
add\_field =\> { "[@metadata][type]" =\> "syslog" }  
add\_field =\> { "[@metadata][beat]" =\> "syslog" }  
}  
if [type] == "syslog" {  
dissect {  
mapping =\> {  
message =\> "\<%{}\>%{ts} %{+ts} %{+ts} %{} [%{}][%{}][%{loglevel}] %{}: %{} %{} Latency: %{latency} [%{url}]"  
"url" =\> "%{}://%{}:%{}/%{uri}"  
"latency" =\> " %{?lf1-\>} %{?lf2-\>} %{?lf3-\>} %{?lf4-\>} %{?lf5-\>} %{lfConnectionAttempted-\>} %{?lf7-\>} %{?lf8-\>} %{?lf9-\>} %{?lf10-\>} %{?lf11-\>} %{lfparsingcomplete-\>} %{?lf13-\>} %{?lf14-\>} %{?lf15-\>} %{?lf16}"  
}  
convert\_datatype =\>  
{  
lfConnectionAttempted =\> "int"  
lfparsingcomplete =\> "int"  
}

```
    }
    ruby {
            code => 'event.set("latencystuff", event.get("latency").split())'
    }
    mutate { convert => { "latencystuff" => "integer" } }
    date { match => ["ts", "MMM dd yyyy HH:mm:ss"] target => "@timestamp" }
}

```

}

## output { if [type] == "syslog" { elasticsearch { hosts =\> ["localhost:9200"] sniffing =\> true manage\_template =\> false user =\> elastic password =\> defaultpassword index =\> "test11-%{+YYYY.MM.dd}" } } }

## Output in Kibana:

## "uri": "accountview/details?accountID=125635]", "url": "[http://10.40.59.102:8070/appview/details?appID=125635](http://10.40.59.102:8070/appview/details?appID=125635)]", "latencystuff": [0, 186, 0, 65, 65, 65, 0, 186, 187, 186, 187, 187, 186, 186, 65, 186], "loglevel": "info",

How do I get rid of "]" from URL? Do I need to regex to get rid of "]" and to parse URI when there are query parameters?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 12, 2018, 7:20pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/12 "2018-03-12T19:20:25Z")

</div>

> [@mguttula](#):
>
> How do I get rid of "]" from URL?

Add a trailing %{} to the dissect mapping as shown [here](https://discuss.elastic.co/t/problem-using-trailing-delimiter-in-dissect/119557/2)

---

<div class="post-metadata">

**Author:** ![mguttula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mguttula/32/78256_2.png) [@mguttula](https://discuss.elastic.co/u/mguttula)\
**Post date:** [March 13, 2018, 12:46am UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/13 "2018-03-13T00:46:40Z")

</div>

that worked. Now I need to figure out how out how to parse URL to get URI without query parameters. Thank you Badger.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 13, 2018, 1:10am UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/14 "2018-03-13T01:10:44Z")

</div>

I haven't tested this, but one approach would be to do two dissects with each of

```
"url" => "http://%{}:%{}/%{uri1}?%{}"
"url" => "http://%{}:%{}/%{uri2}"
```

Then pick one or the other based on whether [uri1] == [uri2]

---

<div class="post-metadata">

**Author:** ![mguttula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mguttula/32/78256_2.png) [@mguttula](https://discuss.elastic.co/u/mguttula)\
**Post date:** [March 13, 2018, 1:41pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/15 "2018-03-13T13:41:22Z")

</div>

> [@Badger](#):
>
> \>

After Adding the query parameter URL line along with URL line without query parameter in logstash config, logstash is not even listening on port 5514 after restarting. Removing query parameter URL from config is working as expected.  
Is there a way to check in some log files where the problem is?

## "logstash-plain.log" file size is empty.

## /var/log/logstash/logstash-plain.log -rw-r--r--. 1 logstash logstash 0 Mar 13 09:36 logstash-plain.log

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 13, 2018, 1:58pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/16 "2018-03-13T13:58:29Z")

</div>

> [@mguttula](#):
>
> Is there a way to check in some log files where the problem is?

Did you add both patterns to a single dissect? That results in an incomprehensible error message 🙂 Try this

```
dissect { mapping => { "url" => "%{}://%{}:%{}/%{uri1}?%{}" } }
  dissect { mapping => { "url" => "%{}://%{}:%{}/%{uri2}" } }
  if [uri1] != "" {
    mutate { add_field => { "uri" => "%{uri1}" } }
  } else {
    mutate { add_field => { "uri" => "%{uri2}" } }
  }
```

---

<div class="post-metadata">

**Author:** ![mguttula](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mguttula/32/78256_2.png) [@mguttula](https://discuss.elastic.co/u/mguttula)\
**Post date:** [March 13, 2018, 6:44pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/17 "2018-03-13T18:44:19Z")

</div>

> [@Badger](#):
>
> dissect { mapping =\> { "url" =\> "%{}://%{}:%{}/%{uri1}?%{}" } }  
> dissect { mapping =\> { "url" =\> "%{}://%{}:%{}/%{uri2}" } }  
> if [uri1] != "" {  
> mutate { add\_field =\> { "uri" =\> "%{uri1}" } }  
> } else {  
> mutate { add\_field =\> { "uri" =\> "%{uri2}" } }  
> }

Thank you Sir. It worked.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 10, 2018, 6:45pm UTC](https://discuss.elastic.co/t/parsing-datapower-syslog-using-grok-for-latency-calculation-for-given-uri/123118/18 "2018-04-10T18:45:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.