# VMware and syslog

**URL:** <https://discuss.elastic.co/t/vmware-and-syslog/65479>\
**Category:** Elasticsearch\
**Created:** [November 9, 2016, 12:16pm UTC](https://discuss.elastic.co/t/vmware-and-syslog/65479 "2016-11-09T12:16:55Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![dmcn\_es](https://avatars.discourse-cdn.com/v4/letter/d/f19dbf/32.png) [@dmcn\_es](https://discuss.elastic.co/u/dmcn_es)\
**Post date:** [November 9, 2016, 12:16pm UTC](https://discuss.elastic.co/t/vmware-and-syslog/65479/1 "2016-11-09T12:16:56Z")

</div>

Hi,  
we have a VMware infrastructure and have been using loginsight to process and visualize logs. However as we move forward we want to use the functionality of ELK stack to collect, parse and display VMware syslog events in Kibana.

I have installed the ELK stack and can view basic items on kibana.  
We are not using logstash-forwarder or filebeats .

At the moment I am looking at the task of how to configure input/filter/output file in such a way as we can get a valuable kibana dashboard for VMware/ESXi outputs.

Has anyone done this already, can these scripts or procedures be shared. ?

Thanks,  
Daragh

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 13, 2016, 8:17am UTC](https://discuss.elastic.co/t/vmware-and-syslog/65479/2 "2016-11-13T08:17:01Z")

</div>

> [@dmcn\_es](#):
>
> configure input/filter/output file in such a way as we can get a valuable kibana dashboard for VMware/ESXi outputs

What file are you referring to here?

---

<div class="post-metadata">

**Author:** ![dmcn\_es](https://avatars.discourse-cdn.com/v4/letter/d/f19dbf/32.png) [@dmcn\_es](https://discuss.elastic.co/u/dmcn_es)\
**Post date:** [November 14, 2016, 11:25am UTC](https://discuss.elastic.co/t/vmware-and-syslog/65479/3 "2016-11-14T11:25:11Z")

</div>

hi Mark,  
Thanks very much for getting back to us on this.

I didn't put all of the below configuration file in this message since it is repetitive based on the different types of faults.

we have this path :- /etc/logstash/conf.d  
And at that location we have this conf file:-

[root@logstash221 conf.d]# cat logstash-syslog.conf  
input {  
udp {  
port =\> 514  
type =\> esxi  
}

}

filter {  
if [type] == "esxi" or "lumberjack" in [tags] {  
grok {  
break\_on\_match =\> true  
match =\> [  
"message", "\<%{POSINT:syslog\_pri}\>%{TIMESTAMP\_ISO8601:@timestamp} %{SYSLOGHOST:hostname} %{SYSLOGPROG:message\_program}: (?

\<body\_type\_1\>(?\<message\_body\>(?\<message\_system\_info\>(?:[%{DATA:message\_thread\_id} %{DATA:syslog\_level} '%{DATA:message\_service}'\ ?%{DATA:message\_opID}]))

[%{DATA:message\_service\_info}]\ (?\<message\_syslog\>(%{GREEDYDATA}))))",  
"message", "\<%{POSINT:syslog\_pri}\>%{TIMESTAMP\_ISO8601:@timestamp} %{SYSLOGHOST:hostname} %{SYSLOGPROG:message\_program}: (?

\<body\_type\_2\>(?\<message\_body\>(?\<message\_system\_info\>(?:[%{DATA:message\_thread\_id} %{DATA:syslog\_level} '%{DATA:message\_service}'\ ?%{DATA:message\_opID}])) (?\<message\_syslog\>

(%{GREEDYDATA}))))",  
"message", "\<%{POSINT:syslog\_pri}\>%{TIMESTAMP\_ISO8601:@timestamp} %{SYSLOGHOST:hostname} %{SYSLOGPROG:message\_program}: (?

\<body\_type\_3\>(?\<message\_body\>%{GREEDYDATA:message\_syslog}))",  
"message", "\<%{POSINT:syslog\_pri}\>._?\s\r\t[\s]._?(?\<message\_program\>[a-zA-

Z0-9-[]\_]{3,})[:][\s](?\<body\_type\_6\>(?\<message\_body\>(?\<message\_syslog\>._)))",  
"message", "(?\<body\_type\_7\>(?\<message\_body\>(?\<message\_debug\>._)))"  
]  
}  
if [message] =~ /(?i)warning|error|fault|ALERT|busy|Failed|[\s]dead|[\s]space|esx.|vob.|com.vmware|nmp|volume|consolidate|FS3|question|ha-eventmgr|VisorFS|Fil3|DLX|MCE|HBX|MPN|

mpn|p2m|Reset|timeout|msg./ and [message] !~ /(?i)crossdup|hostprofiletrace/{

```
                    if [message] =~ /(?i)vmkwarning:/ and [message] !~ /(?i)crossdup|performance|LinuxCharWrite/{
                    # <181>2014-12-18T18:30:36.400Z esx.vmware.com vmkwarning: cpu29:4317)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:972:Could not select path for device 

```

"naa.60002ac000000000000004b00000d155".  
mutate {  
add\_tag =\> "vmkwarning"  
}  
}  
if [message] =~ /(?i)ALERT:/{  
# \<181\>2014-12-17T07:50:52.629Z [esx.vmware.com](http://esx.vmware.com) vmkernel: cpu9:8942)ALERT: URB timed out - USB device may not respond  
mutate {  
add\_tag =\> "achtung"  
add\_field =\> { "alert" =\> "ALERT" }  
}  
} else if [message] =~ /(?i)[\s]dead/ {  
# \<166\>2014-09-15T14:52:23.782Z [esx.vmware.com](http://esx.vmware.com) Hostd: [77381B90 error 'Default'] Unable to build Durable Name dependent properties: Unable to query VPD pages

repetitive edited/removed section since can only put in too big a reply here...

output {  
elasticsearch {

if [type] == "esxi" and "vmkwarning" in [tags] {  
file { path =\> "/var/log/vmkwarning-%{+YYYY-MM-dd}" }

else if [type] == "esxi" and "achtung" in [tags] {  
file { path =\> "/var/log/achtung-%{+YYYY-MM-dd}" }

else if [type] == "esxi" and "alert" in [tags] {  
file { path =\> "/var/log/alert-%{+YYYY-MM-dd}" }

else if [type] == "esxi" and "failed\_to" in [tags] {  
file { path =\> "/var/log/failed\_to-%{+YYYY-MM-dd}" }

else if [type] == "esxi" and "iorm" in [tags] {  
file { path =\> "/var/log/iorm-%{+YYYY-MM-dd}" }

else [type] == "esxi" and "\_grokparsefailure" in [tags] {  
file { path =\> "/var/log/failed\_syslog\_events-%{+YYYY-MM-dd}" }

}  
}

We have alarms such as the below text comeing into this server over port 514 from ESXi hosts:-

"\*\*\* CRITICAL \*\*\* Storage: All Paths Down (APD)":

And we want to build a dashboard based on the syslog messages comeing into this server from the VMware ESXi hosts  
Unfortunately at the moment we get no outputs, even from the last line of the output section

else [type] == "esxi" and "\_grokparsefailure" in [tags] {  
file { path =\> "/var/log/failed\_syslog\_events-%{+YYYY-MM-dd}" }

Which is meant to catch all the items not caught be the former output lines.

br,  
Daragh

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 14, 2016, 9:40pm UTC](https://discuss.elastic.co/t/vmware-and-syslog/65479/4 "2016-11-14T21:40:56Z")

</div>

There's a lot happening here so start with the basics.

Are events making it through the pipeline? Are they in the right format.

---

<div class="post-metadata">

**Author:** ![dmcn\_es](https://avatars.discourse-cdn.com/v4/letter/d/f19dbf/32.png) [@dmcn\_es](https://discuss.elastic.co/u/dmcn_es)\
**Post date:** [November 15, 2016, 10:10am UTC](https://discuss.elastic.co/t/vmware-and-syslog/65479/5 "2016-11-15T10:10:26Z")

</div>

Hi Mark,  
we know ESXi logs are comeing through since we changed the IP of syslog field  
in ESXi to aim at this server.

In addition when we used the below simpler script that caught all the failures with  
the below output block, then we could build basic dashboard of these failures.

But we need more detail just catching all the failures as a test, instead we need to  
be able to re-write the block to get and visualize the ESXi syslog events.

So yes events are comeing in for sure constantly.  
And they are in correct format since can visualize some basic ones as a trial as below.

br,  
Daragh

[root@logstash221 conf.d]# cat logstash-syslog.conf  
input {  
udp {  
type =\> syslog  
port =\> 514  
}  
}  
filter {  
if [type] == "syslog" {  
grok {  
match =\> { "message" =\> "\<%{POSINT:syslog\_pri}\>%{SYSLOGTIMESTAMP:syslog\_timestamp}

%{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?:

%{GREEDYDATA:syslog\_message}" }  
}  
}  
}  
output {  
elasticsearch {  
}  
if [type] == "syslog" and "\_grokparsefailure" in [tags] {  
file { path =\> "/var/log/failed\_syslog\_events-%{+YYYY-MM-dd}" }  
}  
}  
[root@logstash221 conf.d]#

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 13, 2016, 10:10am UTC](https://discuss.elastic.co/t/vmware-and-syslog/65479/6 "2016-12-13T10:10:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.