# When importing Drupal watchdog log file in logstash not getting fields separated from message in elastisearch

**URL:** https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522
**Category:** Logstash
**Created:** [January 9, 2019, 11:47am UTC](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522 "2019-01-09T11:47:01Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![Bdr](https://avatars.discourse-cdn.com/v4/letter/b/7c8e57/32.png) [@Bdr](https://discuss.elastic.co/u/Bdr)
#### Post date: [January 9, 2019, 11:47am UTC](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522/1 "2019-01-09T11:47:01Z")

</div>

This is my watchdog file  
Jan 3 06:28:34 10.81.157.182 shopingstore: https://secure.shopingstore.com|1546496914|product\_rest|23.194.213.4|https://secure.shopingstore.com/api/product/part?\_format=json||1||Update Part request\_id="v-cb88a3a2-0f20-11e9-b27c-22000aee32e6

this is my logstash confg file

""""  
input {  
file {  
path =\> "/home/Desktop/logfiles/drupal-watchdog.log"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
}  
}  
filter {  
if [type] == "drupalsyslog" {  
grok {  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:logdate} %{IPORHOST:logsource} %{SYSLOGHOST:syslogprog}: https?://%{HOSTNAME:drupal\_vhost}|%{NUMBER:drupal\_timestamp}|(?\<drupal\_action\>[^|]_)|%{IP:drupal\_ip}|(?\<drupal\_request\_uri\>[^|]_)|(?\<drupal\_referer\>[^|]_)|(?\<drupal\_uid\>[^|]_)|(?\<drupal\_link\>[^|]_)|(?\<drupal\_message\>._)" }  
}  
date {  
locale =\> "en"  
match =\> ["logdate", "dd/MMM/yyyy:HH:mm:ss Z", "MMM d HH:mm:ss" , "MMM d HH:mm:ss", "dd-MMM-yyyy HH:mm:ss ZZZ"]  
}  
}  
}  
output {  
elasticsearch {  
hosts =\> "localhost:9200"  
manage\_template =\> false  
index =\> "dwl-%{+YYYY.MM.dd}"  
}  
stdout { codec =\> rubydebug }  
}  
"""""

Output  
"""  
\_index: "dwl-2019.01.09",  
\_type: "doc",  
\_id: "vTY8MmgBNTrSTUskwpY9",  
\_score: 1,  
\_source: {  
path: "/home/Desktop/logfiles/drupal-watchdog.log",  
tags: [  
"\_grokparsefailure",  
"\_geoip\_lookup\_failure"  
],  
message: "Jan 3 06:28:34 10.81.157.182 shopingstore: https://secure.shopingstore.com|1546496914|product\_rest|23.194.213.4|https://secure.shopingstore.com/api/product/part?\_format=json||1||Update Part request\_id="v-cb88a3a2-0f20-11e9-b27c-22000aee32e6",  
@timestamp: "2019-01-09T10:50:54.603Z",  
@version: "1",  
type: "drupal"  
""  
I need values in that message separately like drupal\_request\_uri, client ip

Please help me someone.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 9, 2019, 1:52pm UTC](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522/2 "2019-01-09T13:52:31Z")

</div>

You need to escape the | when it is a separator and the character groups (square brackets) require a modifier for zero or more occurrences. Try

```
grok { match => { "message" => "%{SYSLOGTIMESTAMP:logdate} %{IPORHOST:logsource} %{SYSLOGHOST:syslogprog}: https?://%{HOSTNAME:drupal_vhost}\|%{NUMBER:drupal_timestamp}\|(?<drupal_action>[^|]*)\|%{IP:drupal_ip}\|(?<drupal_request_uri>[^|]*)\|(?<drupal_referer>[^|]*)\|(?<drupal_uid>[^|]*)\|(?<drupal_link>[^|]*)\|(?<drupal_message>.*)" } }

```

I added a \* to the . in drupal\_message as well. I assume you do not just want the first character of it.

---

<div class="post-metadata">

### Author: ![Bdr](https://avatars.discourse-cdn.com/v4/letter/b/7c8e57/32.png) [@Bdr](https://discuss.elastic.co/u/Bdr)
#### Post date: [January 10, 2019, 4:17am UTC](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522/3 "2019-01-10T04:17:34Z")

</div>

> [@Bdr](#):
>
> drupalsyslog

Thanks for giving reply.  
I tried with that but still I'm getting same result. I need values separately output should be like

"""  
\_index: "dwl-2019.01.09",  
\_type: "doc",  
\_id: "vTY8MmgBNTrSTUskwpY9",  
\_score: 1,  
\_source: {  
path: "/home/Desktop/logfiles/drupal-watchdog.log",  
logsource: "",  
drupal\_vhost: "",  
drupal\_ip: "",  
message: "Jan 3 06:28:34 10.81.157.182 shopingstore: https://secure.shopingstore.com|1546496914|product\_rest|23.194.213.4|[https://secure.shopingstore.com/api/product/part?\_format=json||1||Update](https://secure.shopingstore.com/api/product/part?_format=json%7C%7C1%7C%7CUpdate) Part request\_id="v-cb88a3a2-0f20-11e9-b27c-22000aee32e6",  
@timestamp: "2019-01-09T10:50:54.603Z",  
@version: "1",  
type: "drupal"  
""

---

<div class="post-metadata">

### Author: ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)
#### Post date: [January 10, 2019, 4:42am UTC](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522/4 "2019-01-10T04:42:12Z")

</div>

@Bdr

Hi Can you post your drupal log file ?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 10, 2019, 2:09pm UTC](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522/5 "2019-01-10T14:09:27Z")

</div>

With the grok I wrote and the message you gave I get

```
           "logdate" => "Jan 3 06:28:34",
      "drupal_vhost" => "secure.shopingstore.com",
     "drupal_action" => "product_rest",
        "syslogprog" => "shopingstore",
  "drupal_timestamp" => "1546496914",
"drupal_request_uri" => "https://secure.shopingstore.com/api/product/part?_format=json",
         "drupal_ip" => "23.194.213.4",
           "message" => "Jan 3 06:28:34 10.81.157.182 shopingstore: https://secure.shopingstore.com|1546496914|product_rest|23.194.213.4|https://secure.shopingstore.com/api/product/part?_format=json||1||Update Part request_id=\"v-cb88a3a2-0f20-11e9-b27c-22000aee32e6",
        "drupal_uid" => "1",
         "logsource" => "10.81.157.182",
    "drupal_message" => "Update Part request_id=\"v-cb88a3a2-0f20-11e9-b27c-22000aee32e6"

```

Do you not get the same?

---

<div class="post-metadata">

### Author: ![Bdr](https://avatars.discourse-cdn.com/v4/letter/b/7c8e57/32.png) [@Bdr](https://discuss.elastic.co/u/Bdr)
#### Post date: [January 21, 2019, 6:21am UTC](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522/6 "2019-01-21T06:21:19Z")

</div>

> [@Badger](#):
>
> grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:logdate} %{IPORHOST:logsource} %{SYSLOGHOST:syslogprog}: https?://%{HOSTNAME:drupal\_vhost}|%{NUMBER:drupal\_timestamp}|(?\<drupal\_action\>[^|]_)|%{IP:drupal\_ip}|(?\<drupal\_request\_uri\>[^|]_)|(?\<drupal\_referer\>[^|]_)|(?\<drupal\_uid\>[^|]_)|(?\<drupal\_link\>[^|]_)|(?\<drupal\_message\>._)" } }

I'm not getting the same.could you post full logstash.conf file.

---

<div class="post-metadata">

### Author: ![Bdr](https://avatars.discourse-cdn.com/v4/letter/b/7c8e57/32.png) [@Bdr](https://discuss.elastic.co/u/Bdr)
#### Post date: [January 21, 2019, 6:51am UTC](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522/7 "2019-01-21T06:51:33Z")

</div>

This is my code in logstash-watchdog.conf file

"""""""""  
input {  
file {  
path =\> "/home/Desktop/logfiles/drupal-watchdog.log"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
}  
}

filter {  
if [type] == "drupalsyslog" {  
grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:logdate} %{IPORHOST:logsource} %{SYSLOGHOST:syslogprog}: https?://%{HOSTNAME:drupal\_vhost}|%{NUMBER:drupal\_timestamp}|(?\<drupal\_action\>[^|]_)|%{IP:drupal\_ip}|(?\<drupal\_request\_uri\>[^|]_)|(?\<drupal\_referer\>[^|]_)|(?\<drupal\_uid\>[^|]_)|(?\<drupal\_link\>[^|]_)|(?\<drupal\_message\>._)" } }

}  
date {  
locale =\> "en"  
match =\> ["logdate", "dd/MMM/yyyy:HH:mm:ss Z", "MMM d HH:mm:ss" , "MMM d HH:mm:ss", "dd-MMM-yyyy HH:mm:ss ZZZ"]  
}

}

output {  
elasticsearch {  
hosts =\> "localhost:9200"  
manage\_template =\> false  
index =\> "watchdg-%{+YYYY.MM.dd}"  
document\_type =\> "watchdg"  
}  
stdout { codec =\> rubydebug }  
}

""""""""""

Is it correct?

---

<div class="post-metadata">

### Author: ![Bdr](https://avatars.discourse-cdn.com/v4/letter/b/7c8e57/32.png) [@Bdr](https://discuss.elastic.co/u/Bdr)
#### Post date: [January 21, 2019, 7:15am UTC](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522/8 "2019-01-21T07:15:17Z")

</div>

This is the log present in drupal log file.  
"""Jan 3 06:28:34 10.81.157.182 shopingstore: https://secure.shopingstore.com|1546496914|product\_rest|23.194.213.4|https://secure.shopingstore.com/api/product/part?\_format=json||1||Update Part request\_id="v-cb88a3a2-0f20-11e9-b27c-22000aee32e6""""""

---

<div class="post-metadata">

### Author: ![Bdr](https://avatars.discourse-cdn.com/v4/letter/b/7c8e57/32.png) [@Bdr](https://discuss.elastic.co/u/Bdr)
#### Post date: [January 21, 2019, 8:00am UTC](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522/9 "2019-01-21T08:00:53Z")

</div>

Thanks @Badger after removing if condition then it's working fine

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 18, 2019, 8:00am UTC](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522/10 "2019-02-18T08:00:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
