jspeer
(Josh Speer)
September 15, 2017, 4:45pm
1
I'm running filebeats 5.5.1, the 2 fields that I've added into the prospector aren't sending with the logs. Here is my prospector:
- input_type: log
paths:
- /etc/filebeat/log/*.log
fields:
log_type: promolog
type: pricing
fields_under_root: true
Here is the message I'm sending to the log
echo "This is a test" >> test.log
Here is what logstash is seeing
2017-09-15T16:39:48.593Z 1843d9713bf0 This is a test
andrewkroh
(Andrew Kroh)
September 15, 2017, 6:27pm
2
How do you have Logstash configured? Where is that output coming from? Please share your Logstash config.
jspeer
(Josh Speer)
September 15, 2017, 6:35pm
3
Here is the combined logstash config, it's separated into separate files for input, filter, and output
input {
beats {
port => 5044
ssl => false
#ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
#ssl_key => "/etc/pki/tls/private/logstash-beats.key"
}
}
filter {
if [fields][log_type] == "promolog" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:logtimestamp} %{WORD:msg} %
{SYSLOG5424SD:jclass} - %{WORD:rtype} %{SPACE}:%{SPACE}%{GREEDYDATA:payload}" }
}
date {
match => ["logtimestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
target => "@timestamp"
add_field => { "debug" => "timestampMatched"}
}
csv {
source => "payload"
separator => "|"
remove_field => ["payload"]
}
mutate {
add_field => {
"reqtype" => "%{column2}"
"ordernum" => "%{column3}"
"brand" => "%{column4}"
"promo" => "%{column5}"
}
}
}
}
output {
stdout {}
elasticsearch {
hosts => ["localhost"]
manage_template => false
index => "pricing-%{YYYY.MM.dd}"
document_type => "pricing"
user => "elastic"
password => "changeme"
}
}
andrewkroh
(Andrew Kroh)
September 15, 2017, 6:50pm
4
Try using a different codec with the stdout output. This should cause all fields to be logged.
output {
stdout { codec => rubydebug { metadata => true } }
}
jspeer
(Josh Speer)
September 15, 2017, 7:04pm
5
Perfect, thank you! I'm now seeing the fields in the logstash output.
system
(system)
Closed
October 13, 2017, 7:04pm
6
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.