Hi there!
I'm trying to send logs through filebeats to logstash,
I have:
Logstash--on local machine
filebeat-on local machine
Elastic search---on host machine(server)
In filebeat.yml I enabled logstash output.
My filebeat.yml
================================
filebeat.prospectors:
- input_type: log
paths:
- /home/sandhyarani/Downloads/logstash-5.4.1/bin/*.log
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["xx.xx.xxx.xx:9200"] # Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"] # Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"#================================
Logstash Conf file:
=========================================================
input {
stdin {
}
}filter {
mutate{ add_field => { "Source" => "SKY"}
}
environment{
add_field => ["my_environment", "Hello World, from %{host}"]
}
if "exception" not in [tags] { # example output:
grok {
match => {
message => "%{DATESTAMP:timestamp} %{LOGLEVEL:level}( +)\[%{DATA:thread}\] \[%{DATA:mdc}\] %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}"
#message => "%{DATESTAMP:timestamp} %{LOGLEVEL:level}( +)\[%{DATA:thread}\] \[%{DATA:mdc}\] %{JAVACLASS:class} - %{GREEDYDATA:logmsg}"
}
add_tag => ["log"]
} if "log" in [tags] { grok {
match => {
mdc => "%{DATA:username}:%{DATA:deviceId:int}:%{DATA:sessionId}:%{DATA:userInteraction:int}:%{GREEDYDATA:workflowName}"
}
}
date {
timezone => GMT
match => [
# "16-12-16 21:58:20,606"
"timestamp", "yy-MM-dd HH:mm:ss,SSS"
]
} } }
if [level] in ["ERROR", "error"] or [level] in ["FATAL", "fatal"]{
mutate {
add_tag => ["alert"]
}
}
if [level] in ["TRACE", "trace"] {
mutate {
replace => {
"level" => "%{level}, 0"
}
}
}
else if [level] in ["DEBUG", "debug"]{
mutate {
replace => {
"level" => "%{level}, 1"
}
}
}
else if [level] in ["INFO", "info"]{
mutate {
replace => {
"level" => "%{level}, 2"
}
}
}
else if [level] in ["WARN", "warn"]{
mutate {
replace => {
"level" => "%{level}, 3"
}
}
}
else if [level] in ["ERROR", "error"]{
mutate {
replace => {
"level" => "%{level}, 4"
}
}
}
else if [level] in ["FATAL", "fatal"]{
mutate {
replace => {
"level" => "%{level}, 5"
}
}
}}
output {
if "_grokparsefailure" in [tags] {
stdout { codec => rubydebug {metadata => true }}
}
if "log" in [tags]{
if "ERROR" in [level]{
elasticsearch { hosts => ["xxx.xxx.xxx.com"] }
}
else if "WARN" in [level]{
elasticsearch { hosts => ["xxx.xxx.xxx.com"] }
}
else if "INFO" in [level]{
elasticsearch { hosts => ["xxx.xxx.xxx.com"] }
}
else if "FATAL" in [level]{
elasticsearch { hosts => ["xxx.xxx.xxx.com"] }
}
}
}
I enabled logstash output in filebeat.yml:
when i run filebeat its showing this
Could any one help with this?
Thanks!