Thanks Paris. I am glad you replied asap. I really appreciate your help. Thanks a lot !! You made my Day !!
Thanks you very much !!
The Beats shipper automatically sets the type field on the event. You cannot override this setting in the Logstash config. If you specify a setting for the type config option in Logstash, it is ignored.
It helped me a lot to resolve the above issue. I haven't changed the filter configurations.
I have used the following configurations and it resolved the above issue.
input {
udp {
port => 514
type => "hyworks"
}
beats {
port => 5044
}
}
filter {
if [type] == "hyworks" {
grok {
match => {"message" => "<%{INT:loglvl}>HyworksController: %{LOGLEVEL:loglevel} %{USERNAME:user} %{WORD:ORGNAME} (?<ORG_ID>(?:[A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12})) %{WORD:logger}(?:\s+\[%{GREEDYDATA:temp}\]([\.:])?)? %{GREEDYDATA:msg}" }
}
if "_grokparsefailure" not in [tags] {
if [temp] {
mutate {
replace => {"message" => "<%{loglvl}>HyworksController: [%{temp}]: %{msg}"}
remove_field => [ "msg" ]
gsub => ["temp"," ","_"]
}
}
else {
mutate {
replace => {"message" => "<%{loglvl}>HyworksController: %{msg}"}
remove_field => [ "msg" ]
}
}
if [temp] {
mutate {
add_tag => ["%{temp}"]
}
}
if [temp] {
mutate {
remove_field => [ "temp" ]
}
}
if " logged in" in [message] {
mutate {
add_tag => ["logged_in","%{user}_logged_in"]
}
}
if " successfully logged out" in [message] {
mutate {
add_tag => ["logged_out","%{user}_logged_out"]
}
}
if " 'hyworksadmin' successfully logged into " in [message] {
mutate {
add_tag => ["admin_logged_in","hyworks"]
}
}
if " and has been Disconnected." in [message] {
mutate {
add_tag => ["idle_disconnected","%{user}_idle_disconnected"]
}
}
if " and has been Logout." in [message] {
mutate {
add_tag => ["idle_logout","%{user}_idle_logout"]
}
}
mutate {
add_tag => ["_user_%{user}"]
}
}
else {
mutate {
add_field => {"loglevel" => "ERROR"}
add_tag => ["error_log","error_traceback"]
}
}
}
}
output {
if [type] == "hyworks" {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "hyworks-%{+YYYY.MM.dd}"
}
}
else{
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
}
Now, the above issue is resolved for UDP and beats, but if I have beats of Filebeat and Metricbeat then how to filter the output based on the type.
Any suggestions ?