When importing Drupal watchdog log file in logstash not getting fields separated from message in elastisearch

[Bdr]

This is my watchdog file
Jan 3 06:28:34 10.81.157.182 shopingstore: https://secure.shopingstore.com|1546496914|product_rest|23.194.213.4|https://secure.shopingstore.com/api/product/part?_format=json||1||Update Part request_id="v-cb88a3a2-0f20-11e9-b27c-22000aee32e6

this is my logstash confg file

""""
input {
file {
path => "/home/Desktop/logfiles/drupal-watchdog.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
if [type] == "drupalsyslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:logdate} %{IPORHOST:logsource} %{SYSLOGHOST:syslogprog}: https?://%{HOSTNAME:drupal_vhost}|%{NUMBER:drupal_timestamp}|(?<drupal_action>[^|])|%{IP:drupal_ip}|(?<drupal_request_uri>[^|])|(?<drupal_referer>[^|])|(?<drupal_uid>[^|])|(?<drupal_link>[^|])|(?<drupal_message>.)" }
}
date {
locale => "en"
match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z", "MMM d HH:mm:ss" , "MMM d HH:mm:ss", "dd-MMM-yyyy HH:mm:ss ZZZ" ]
}
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "dwl-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
"""""

Output
"""
_index: "dwl-2019.01.09",
_type: "doc",
_id: "vTY8MmgBNTrSTUskwpY9",
_score: 1,
_source: {
path: "/home/Desktop/logfiles/drupal-watchdog.log",
tags: [
"_grokparsefailure",
"_geoip_lookup_failure"
],
message: "Jan 3 06:28:34 10.81.157.182 shopingstore: https://secure.shopingstore.com|1546496914|product_rest|23.194.213.4|https://secure.shopingstore.com/api/product/part?_format=json||1||Update Part request_id="v-cb88a3a2-0f20-11e9-b27c-22000aee32e6",
@timestamp: "2019-01-09T10:50:54.603Z",
@version: "1",
type: "drupal"
""
I need values in that message separately like drupal_request_uri, client ip

Please help me someone.

[Badger]

You need to escape the | when it is a separator and the character groups (square brackets) require a modifier for zero or more occurrences. Try

grok { match => { "message" => "%{SYSLOGTIMESTAMP:logdate} %{IPORHOST:logsource} %{SYSLOGHOST:syslogprog}: https?://%{HOSTNAME:drupal_vhost}\|%{NUMBER:drupal_timestamp}\|(?<drupal_action>[^|]*)\|%{IP:drupal_ip}\|(?<drupal_request_uri>[^|]*)\|(?<drupal_referer>[^|]*)\|(?<drupal_uid>[^|]*)\|(?<drupal_link>[^|]*)\|(?<drupal_message>.*)" } }

I added a * to the . in drupal_message as well. I assume you do not just want the first character of it.

[Bdr]

@Bdr

drupalsyslog

Thanks for giving reply.
I tried with that but still I'm getting same result. I need values separately output should be like

"""
_index: "dwl-2019.01.09",
_type: "doc",
_id: "vTY8MmgBNTrSTUskwpY9",
_score: 1,
_source: {
path: "/home/Desktop/logfiles/drupal-watchdog.log",
logsource: "",
drupal_vhost: "",
drupal_ip: "",
message: "Jan 3 06:28:34 10.81.157.182 shopingstore: https://secure.shopingstore.com|1546496914|product_rest|23.194.213.4|https://secure.shopingstore.com/api/product/part?_format=json||1||Update Part request_id="v-cb88a3a2-0f20-11e9-b27c-22000aee32e6",
@timestamp: "2019-01-09T10:50:54.603Z",
@version: "1",
type: "drupal"
""

[Makra (Makara)]

@Bdr

Hi Can you post your drupal log file ?

[Badger]

With the grok I wrote and the message you gave I get

           "logdate" => "Jan 3 06:28:34",
      "drupal_vhost" => "secure.shopingstore.com",
     "drupal_action" => "product_rest",
        "syslogprog" => "shopingstore",
  "drupal_timestamp" => "1546496914",
"drupal_request_uri" => "https://secure.shopingstore.com/api/product/part?_format=json",
         "drupal_ip" => "23.194.213.4",
           "message" => "Jan 3 06:28:34 10.81.157.182 shopingstore: https://secure.shopingstore.com|1546496914|product_rest|23.194.213.4|https://secure.shopingstore.com/api/product/part?_format=json||1||Update Part request_id=\"v-cb88a3a2-0f20-11e9-b27c-22000aee32e6",
        "drupal_uid" => "1",
         "logsource" => "10.81.157.182",
    "drupal_message" => "Update Part request_id=\"v-cb88a3a2-0f20-11e9-b27c-22000aee32e6"

Do you not get the same?

[Bdr]

@Badger

grok { match => { "message" => "%{SYSLOGTIMESTAMP:logdate} %{IPORHOST:logsource} %{SYSLOGHOST:syslogprog}: https?://%{HOSTNAME:drupal_vhost}|%{NUMBER:drupal_timestamp}|(?<drupal_action>[^|])|%{IP:drupal_ip}|(?<drupal_request_uri>[^|])|(?<drupal_referer>[^|])|(?<drupal_uid>[^|])|(?<drupal_link>[^|])|(?<drupal_message>.)" } }

I'm not getting the same.could you post full logstash.conf file.

[Bdr]

This is my code in logstash-watchdog.conf file

"""""""""
input {
file {
path => "/home/Desktop/logfiles/drupal-watchdog.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
if [type] == "drupalsyslog" {
grok { match => { "message" => "%{SYSLOGTIMESTAMP:logdate} %{IPORHOST:logsource} %{SYSLOGHOST:syslogprog}: https?://%{HOSTNAME:drupal_vhost}|%{NUMBER:drupal_timestamp}|(?<drupal_action>[^|])|%{IP:drupal_ip}|(?<drupal_request_uri>[^|])|(?<drupal_referer>[^|])|(?<drupal_uid>[^|])|(?<drupal_link>[^|])|(?<drupal_message>.)" } }

}
date {
locale => "en"
match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z", "MMM d HH:mm:ss" , "MMM d HH:mm:ss", "dd-MMM-yyyy HH:mm:ss ZZZ" ]
}

}

output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "watchdg-%{+YYYY.MM.dd}"
document_type => "watchdg"
}
stdout { codec => rubydebug }
}

""""""""""

Is it correct?

[Bdr]

This is the log present in drupal log file.
"""Jan 3 06:28:34 10.81.157.182 shopingstore: https://secure.shopingstore.com|1546496914|product_rest|23.194.213.4|https://secure.shopingstore.com/api/product/part?_format=json||1||Update Part request_id="v-cb88a3a2-0f20-11e9-b27c-22000aee32e6""""""

[Bdr]

Thanks @Badger after removing if condition then it's working fine