# Logstash setup for syslog udp input **URL:** \ **Category:** Logstash\ **Created:** [May 24, 2016, 8:56am UTC](https://discuss.elastic.co/t/logstash-setup-for-syslog-udp-input/50810 "2016-05-24T08:56:04Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![panopt1k](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/panopt1k/32/9904_2.png) [@panopt1k](https://discuss.elastic.co/u/panopt1k)\ **Post date:** [May 24, 2016, 8:56am UTC](https://discuss.elastic.co/t/logstash-setup-for-syslog-udp-input/50810/1 "2016-05-24T08:56:04Z")
Hi guys! I need your help in advanced setting up for ELK server. I have installed ELK stack into Ubuntu 14.04 Basically I setup logstash server with filebeats and successfully configured logstash filter for parsing logs I can see all logs came from filebeats in kibana. Now I want to switch log collecting from filebeats directly into rsyslog input So I setup one of my application servers to syslog output into my logstash server Here is my configuration for nginx application node: `access_log syslog:server=my-elk-server.domain:1514 mylog;` Logstash configuration I have separated into different configs with prefix input- filter- output- Next my input files looks like this: **/etc/logstash/conf.d/nput.filebeat.conf** ``` input { beats { port => 5044 type => "filebeat_log" } } ``` **/etc/logstash/conf.d/input-rsyslog.conf** ``` input { udp { port => 1514 } tcp { port => 1514 } } ``` Filter configuration is a little complicated, but I believe that for experienced members it will not be a problem **/etc/logstash/conf.d/filter-prod-filebeats.conf** ``` filter { grok { match => [ "message", '%{SYSLOGTIMESTAMP:timestamp} %{NOTSPACE:node_name} %{HTTPDUSER:ident}: (?:%{IP:remote_client_ip}|-) \[%{IP:client_ip}\] - (?:%{USER:protocol}|-) \[%{HTTPDATE:time_local}\] "%{IPORHOST:httphost}" "(?:%{WORD:verb} %{NOTSPACE:request})(?: HTTP/%{NUMBER:httpversion})" %{NUMBER:response_code} %{NUMBER:request_length} (?:%{NUMBER:bytes_sent}|-) (?:%{NUMBER:request_time}|-) %{QS:http_device} %{QS:http_user_agent}' ] add_tag => ["nginx_access_log", "prod"] remove_tag => ["_grokparsefailure"] } # production error.log processing grok { match => [ "message", '%{SYSLOGTIMESTAMP:timestamp} %{NOTSPACE:node_name} %{HTTPDUSER:ident}: %{GREEDYDATA:error}host: "%{IPORHOST:httphost}"' ] add_tag => ["nginx_error_log", "prod", "error"] } grok { match => [ "message", '%{SYSLOGTIMESTAMP:timestamp} %{NOTSPACE:node_name} %{HTTPDUSER:ident}: %{GREEDYDATA:error}' ] add_tag => ["nginx_error_log", "prod", "error"] } date { match => ["timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"] target => "@timestamp" } } ``` Output configured as elasticsearch and file **/etc/logstash/conf.d/output-elasticsearch.conf** ``` output { elasticsearch { hosts => ["localhost:9200"] sniffing => true manage_template => false index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } ``` **/etc/logstash/conf.d/output-file.conf** ``` output { file { path => "/var/log/logstash/output_line.log" codec => "line" } } ``` So when I attached syslog input. I can see new records in output file together with filebeats entries, but udp inputs has a little difference > 2016-05-20T06:40:20.842Z 172.31.10.228 \<190\>May 20 06:40:20 node04 nginx: 217.119.17.14 [172.11.22.12] - https [20/May/2016:06:40:20 +0000] "[customer.example.com](http://customer.example.com)" "PUT /command/ HTTP/1.1" 200 1050 302 0.192 "{017ef056-1aaa-4712-a8e7-7584612aea3b}" "M Windows Query: 2" > 2016-05-20T06:40:12.000Z node01 May 20 06:40:12 node03 nginx: 197.17.12.19 [172.11.22.110] - - [20/May/2016:06:40:12 +0000] "[customer.example.com](http://customer.example.com)" "PUT /command/ HTTP/1.1" 200 440 325 0.252 "" "M Windows Query: 4" There are an extra sequence **\<190\>** after timestamp in log which come from udp And there is a problem that this lines from udp have not enter in kibana and in elasticsearch Only lines came from filebeats (without \<190\>) are present in kibana My questions are: What I had wrong with UDP (syslog) input or filter? Why does UDP input shown in file with extra \<190\> ? And most important: why does UDP input not parsed with grok pattern if there are the same input from different sources ? And in the and if filter does not work, can I save into elasticsearch at least unparsed data (something like fallback action) ? Thank you in advance for any suggestions and advice! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 29, 2016, 9:48am UTC](https://discuss.elastic.co/t/logstash-setup-for-syslog-udp-input/50810/2 "2016-05-29T09:48:49Z")
> What I had wrong with UDP (syslog) input or filter? > Why does UDP input shown in file with extra \<190\> ? "\<190\>" is the facility and severity marker that's part of the syslog protocol. It's not present in on-disk files. You can use the syslog\_pri filter to convert the number into something human readable. > And most important: why does UDP input not parsed with grok pattern if there are the same input from different sources ? Because your grok filter doesn't take the facility/severity token into account? > And in the and if filter does not work, can I save into elasticsearch at least unparsed data (something like fallback action) ? That's what it does by default. About the only occasion where Logstash drops an event (except when you explicitly request it to, obviously) is for mapping errors, i.e. when ES can't receive the event because because its contents isn't compatible with the index's mappings. ---
**Author:** ![panopt1k](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/panopt1k/32/9904_2.png) [@panopt1k](https://discuss.elastic.co/u/panopt1k)\ **Post date:** [July 27, 2016, 2:52pm UTC](https://discuss.elastic.co/t/logstash-setup-for-syslog-udp-input/50810/3 "2016-07-27T14:52:51Z")
Thanks a lot! You're my Hero! It was a confusing with output using @metadata which was absent during UDP incoming requests. Fixed with adding metadata fields for output section: ``` syslog_pri { add_field => { "[@metadata][type]" => "syslog" } add_field => { "[@metadata][beat]" => "syslog" } } ``` So final config looks like next: ``` input { udp { port => 1514 type => "syslog" } } filter { syslog_pri { add_field => { "[@metadata][type]" => "syslog" } add_field => { "[@metadata][beat]" => "syslog" } } if [type] == "syslog" { # production access.log processing grok { match => [ "message", '<190>%{SYSLOGTIMESTAMP:timestamp} %{NOTSPACE:node_name} %{HTTPDUSER:ident}: (?:%{IP:http_x_forwarded_for}|-) \[%{IP:remote_addr}\] - (?:%{USER:remote_user}|-) \[%{HTTPDATE:time_local}\] "%{IPORHOST:httphost}" "(?:%{WORD:verb} %{NOTSPACE:request})(?: HTTP/%{NUMBER:httpversion})" %{NUMBER:response_code} %{NUMBER:request_length} (?:%{NUMBER:bytes_sent}|-) (?:%{NUMBER:request_time}|-) %{QS:http_device} %{QS:http_user_agent}' ] break_on_match => true add_tag => ["nginx_access_log", "prod"] remove_tag => ["_grokparsefailure"] } # production error.log processing grok { match => [ "message", '%{SYSLOGTIMESTAMP:timestamp} %{NOTSPACE:node_name} %{HTTPDUSER:ident}: %{GREEDYDATA:error}host: "%{IPORHOST:httphost}"' ] add_tag => ["nginx_error_log", "prod", "error"] } grok { match => [ "message", '%{SYSLOGTIMESTAMP:timestamp} %{NOTSPACE:node_name} %{HTTPDUSER:ident}: %{GREEDYDATA:error}' ] add_tag => ["nginx_error_log", "prod", "error"] } } } output { elasticsearch { hosts => ["localhost:9200"] sniffing => true manage_template => false index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } ``` Hope it might help somebody else ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:46am UTC](https://discuss.elastic.co/t/logstash-setup-for-syslog-udp-input/50810/4 "2017-07-06T04:46:17Z")