# New fields

**URL:** https://discuss.elastic.co/t/new-fields/141537
**Category:** Logstash
**Created:** [July 25, 2018, 9:59am UTC](https://discuss.elastic.co/t/new-fields/141537 "2018-07-25T09:59:21Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![mohamed.samir](https://avatars.discourse-cdn.com/v4/letter/m/94ad74/32.png) [@mohamed.samir](https://discuss.elastic.co/u/mohamed.samir)
#### Post date: [July 25, 2018, 9:59am UTC](https://discuss.elastic.co/t/new-fields/141537/1 "2018-07-25T09:59:21Z")

</div>

i have forwarded sys log messages to my ELK.  
i have used grok to filter this logs and abstract the fields, but it didn't work.  
i am sure with my grok.  
when i forward the php logs directly to the ELK, with another grok filter, it works !!!  
any idea, why syslog message with grok, to splitt fields is not working?

# bellow a samble of the syslog message

\<134\>Jul 25 10:51:29 cisco tmm1[13548]: Rule /Common/elastic\_log \<HTTP\_RESPONSE\>: Client:x.x.x.x -- VS:y.y.y.y -- URL:www.aaaa.com/index.aspx -- username: -- Node:a.a.a.a -- RES:600

# and the syslog grok filter is

match =\> { "message" =\> "\<%{BASE10NUM:elastics\_code}\>%{SYSLOGTIMESTAMP:elastics\_time} cisco tmm1[%{BASE10NUM:tmm1\_code}]: Rule /Common/elastic\_log \<HTTP\_RESPONSE\>: Client:%{IPV4:client\_ip} -- VS:%{IPV4:server\_ip} -- URL:%{DATA:uri} -- username:%{DATA:username} -- Node:%{IPV4:site\_ip} -- RES:%{BASE10NUM:response}" }  
}

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 25, 2018, 1:04pm UTC](https://discuss.elastic.co/t/new-fields/141537/2 "2018-07-25T13:04:18Z")

</div>

It's a regexp, so you need to escape characters that have meaning in regexps, such as square brackets.

```
<%{BASE10NUM:elastics_code}>%{SYSLOGTIMESTAMP:elastics_time} cisco tmm1\[%{BASE10NUM:tmm1_code}\]: Rule /Common/elastic_log <HTTP_RESPONSE>: Client:%{IPV4:client_ip} -- VS:%{IPV4:server_ip} -- URL:%{DATA:uri} -- username:%{DATA:username} -- Node:%{IPV4:site_ip} -- RES:%{BASE10NUM:response}
```

---

<div class="post-metadata">

### Author: ![mohamed.samir](https://avatars.discourse-cdn.com/v4/letter/m/94ad74/32.png) [@mohamed.samir](https://discuss.elastic.co/u/mohamed.samir)
#### Post date: [July 26, 2018, 10:39am UTC](https://discuss.elastic.co/t/new-fields/141537/3 "2018-07-26T10:39:14Z")

</div>

thanks a lot badger, it works great

---

<div class="post-metadata">

### Author: ![mohamed.samir](https://avatars.discourse-cdn.com/v4/letter/m/94ad74/32.png) [@mohamed.samir](https://discuss.elastic.co/u/mohamed.samir)
#### Post date: [July 29, 2018, 8:12am UTC](https://discuss.elastic.co/t/new-fields/141537/4 "2018-07-29T08:12:03Z")

</div>

please, i have updated my grok to  
grok {  
match =\> { "message" =\> "\<%{BASE10NUM:elastics\_code}\>%{SYSLOGTIMESTAMP:elastics\_time} cisco %{DATA:tmm1\_code}: Rule \<HTTP\_RESPONSE\>: Client:%{IPV4:client\_ipv4} -- Server:%{IPV4:virtual\_server\_ip} -- URL:%{DATA:request\_uri} -- username:%{DATA:username} -- Node:%{IPV4:node\_ipv4} -- RES:%{BASE10NUM:server\_response\_code} -- country\_area:%{DATA:country}" }

it works fine, and it add all fields except "username" and "country" !!!!  
all other fields appeared fine, like "elastics\_code" , "request\_uri", ................  
any idea please?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 29, 2018, 10:37am UTC](https://discuss.elastic.co/t/new-fields/141537/5 "2018-07-29T10:37:11Z")

</div>

In your example message from your first post, neither of those fields exist, so I am not surprised they do not match. Please show both the message and the grok pattern indented by 4 spaces, so that they look like this

```
<%{BASE10NUM:elastics_code}>%{SYSLOGTIMESTAMP:elastics_time} cisco tmm1\[%{BASE10NUM:tmm1_code}\]: Rule /Common/elastic_log <HTTP_RESPONSE>: Client:%{IPV4:client_ip} -- VS:%{IPV4:server_ip} -- URL:%{DATA:uri} -- username:%{DATA:username} -- Node:%{IPV4:site_ip} -- RES:%{BASE10NUM:response}

```

rather than this

\<%{BASE10NUM:elastics\_code}\>%{SYSLOGTIMESTAMP:elastics\_time} cisco tmm1[%{BASE10NUM:tmm1\_code}]: Rule /Common/elastic\_log \<HTTP\_RESPONSE\>: Client:%{IPV4:client\_ip} -- VS:%{IPV4:server\_ip} -- URL:%{DATA:uri} -- username:%{DATA:username} -- Node:%{IPV4:site\_ip} -- RES:%{BASE10NUM:response}

---

<div class="post-metadata">

### Author: ![mohamed.samir](https://avatars.discourse-cdn.com/v4/letter/m/94ad74/32.png) [@mohamed.samir](https://discuss.elastic.co/u/mohamed.samir)
#### Post date: [July 29, 2018, 11:21am UTC](https://discuss.elastic.co/t/new-fields/141537/6 "2018-07-29T11:21:48Z")

</div>

yes, i have updated the message 🙂

# example of the messsage is:

\<134\>Jul 29 11:06:09 cisco cisco[123]: Rule \<HTTP\_RESPONSE\>: Client:196.153.4.34 -- Server:172.20.38.109 -- URL:cndc.aast.edu/images/s3.gif -- username: -- Node:172.20.39.75 -- RES:304 -- country\_area:US

# and the grok is

grok {  
match =\> { "message" =\> "\<%{BASE10NUM:elastics\_code}\>%{SYSLOGTIMESTAMP:elastics\_time} cisco %{DATA:tmm1\_code}: Rule \<HTTP\_RESPONSE\>: Client:%{IPV4:client\_ipv4} -- Server:%{IPV4:virtual\_server\_ip} -- URL:%{DATA:request\_uri} -- username:%{DATA:username} -- Node:%{IPV4:node\_ipv4} -- RES:%{BASE10NUM:server\_response\_code} -- country\_area:%{DATA:country}" }

again, all fields works fine, except these fields "username: and "country"

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 29, 2018, 11:38am UTC](https://discuss.elastic.co/t/new-fields/141537/7 "2018-07-29T11:38:53Z")

</div>

Please indent the message and the filter so that escaped characters are shown.

---

<div class="post-metadata">

### Author: ![mohamed.samir](https://avatars.discourse-cdn.com/v4/letter/m/94ad74/32.png) [@mohamed.samir](https://discuss.elastic.co/u/mohamed.samir)
#### Post date: [July 29, 2018, 2:10pm UTC](https://discuss.elastic.co/t/new-fields/141537/10 "2018-07-29T14:10:35Z")

</div>

```
<134>Jul 29 11:06:09 cisco cisco[123]: Rule <HTTP_RESPONSE>: Client:196.153.4.34 -- Server:172.20.38.109 -- URL:cndc.aast.edu/images/s3.gif -- username: -- Node:172.20.39.75 -- RES:304 -- country_area:US     

grok { match => { "message" => "<%{BASE10NUM:elastics_code}>%{SYSLOGTIMESTAMP:elastics_time} cisco %{DATA:tmm1_code}: Rule <HTTP_RESPONSE>: Client:%{IPV4:client_ipv4} -- Server:%{IPV4:virtual_server_ip} -- URL:%{DATA:request_uri} -- username:%{DATA:username} -- Node:%{IPV4:node_ipv4} -- RES:%{BASE10NUM:server_response_code} -- country_area:%{DATA:country}" }
```

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 29, 2018, 3:44pm UTC](https://discuss.elastic.co/t/new-fields/141537/11 "2018-07-29T15:44:09Z")

</div>

You don't get a username field because there is no username field in the event.

I do not understand why DATA does not match US. You can use GREEDYDATA to pick up the US and the following whitespace. Alternatively use country\_area:%{NOTSPACE:country}

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 26, 2018, 3:50pm UTC](https://discuss.elastic.co/t/new-fields/141537/12 "2018-08-26T15:50:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.