Hi,
I have an issue with my logstash configuration and I can't figure out why.
I have a custom logfile on a server which has filebeat installed. An example of the custom log
02/06/2019 11:51:14 Database Copy: DAG003-MDB159\QQQXM0030
02/06/2019 11:51:14 Server: QQQXM0030
02/06/2019 11:51:14 Activation Preference: 4
02/06/2019 11:51:14 Status: Healthy
02/06/2019 11:51:14 Copy Queue: 0
02/06/2019 11:51:14 Replay Queue: 14124
02/06/2019 11:51:14 Content Index: Healthy
02/06/2019 11:51:14 DAG003-MDB159 is replay lagged on QQQXM0030
02/06/2019 11:51:14 Replay lag is True
02/06/2019 11:51:14 Truncation lag is False
02/06/2019 11:51:14 ---- Processing database DAG003-MDB160
02/06/2019 11:51:14 DAG003-MDB160 has 4 copies
02/06/2019 11:51:14 Database Copy: DAG003-MDB160\QQQXM0023
02/06/2019 11:51:14 Server: QQQXM0023
02/06/2019 11:51:14 Activation Preference: 1
02/06/2019 11:51:14 Status: Mounted
02/06/2019 11:51:14 Copy Queue: 0
02/06/2019 11:51:14 Replay Queue: 0
02/06/2019 11:51:14 Content Index: Healthy
02/06/2019 11:51:14 Replay lag is False
02/06/2019 11:51:14 Truncation lag is False
Running filebeat on the server with multiline and outputting to a file.
filebeat.inputs:
- type: log
enabled: true
paths:
- "/opt/elastic/input/*.log"
fields:
log_type: exchange
fields_under_root: true
tags: ["exchange"]
### Multiline options
multiline.pattern: '------ Checking|Database Copy|---- Checking replication'
multiline.negate: true
multiline.match: after
multiline.flush_pattern: 'Finished checking server|Truncation lag|DBLogReplayKeepingUp'
output.file:
path: "/opt/elastic/"
filename: output
rotate_every_kb: 10000
number_of_files: 7
permissions: 0600
All works fine and the output seems to generate valid JSON. I can copy the output file to my logstash server and put it in a directory for logstash to ingest. All is well, my index is created in Elastic and everything looks good.
Here is a copy of my logstash pipeline with the grok filtering
input {
#beats {
#id => "health_beats_plugin"
#port => 5044
#}
file {
path => "/etc/logstash/data/output"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
json {
source => "message"
target => "doc"
}
if [doc][log_type] == "exchange" {
mutate {
gsub => [ "message", "\\n"," " ]
}
if "grokked" not in [tags] {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => { "message" => "%{DATESTAMP:logdate} ---- Checking replication health for %{HOSTNAME:hostname} %{DATESTAMP} ClusterService %{RESULT:clusterservice} %{DATESTAMP".....................}
overwrite => [ "message" ]
remove_field => ["message"]
add_tag => ["replicationcheck", "grokked", "exchange"]
tag_on_failure => [ ]
}
}
date {
match => [ "logdate", "MM/dd/yyyy HH:mm:ss" ]
target => "@timestamp"
}
if "grokked" not in [tags] {
drop { }
}
}
}
output {
if [doc][log_type] == "exchange" {
elasticsearch {...
The output in elastic index - looks good.
{
"_index": "exchangehealth-2019.02.06",
"_type": "doc",
"_id": "2z0LFmkBzNgyfHMOj0Ey",
"_version": 1,
"_score": null,
"_source": {
"dbcopy": "DAG003-MDB159\\\\QQQXM0030",
"@timestamp": "2019-02-06T11:51:14.000Z",
"dbcopylag": "DAG003-MDB159",
"hostname": "QQQXM0030",
"replaylag": "True",
"host": "qqqsubprd01",
"@version": "1",
"activationpreference": "4",
"hostnamelag": "QQQXM0030",
"logdate": "02/06/2019 11:51:14",
"copyqueue": "0",
"path": "/etc/logstash/data/outputzz",
"contentindex": "Healthy",
"truncationlag": "False",
"doc": {
"offset": 288911,
"log": {
"flags": [
"multiline"
]
},
"@metadata": {
"type": "doc",
"version": "6.5.3",
"beat": "filebeat"
},
"prospector": {
"type": "log"
},
"@timestamp": "2019-02-21T00:58:03.284Z",
"source": "/opt/elastic/input/exchangeserverhealth1.log",
"host": {
"name": "qqqelkrd01"
},
"beat": {
"hostname": "qqqelkrd01",
"version": "6.5.3",
"name": "qqqelkrd01"
},
"input": {
"type": "log"
},
"log_type": "exchange",
"message": "02/06/2019 11:51:14 Database Copy: DAG003-MDB159\\QQQXM0030\n02/06/2019 11:51:14 Server: QQQXM0030\n02/06/2019 11:51:14 Activation Preference: 4\n02/06/2019 11:51:14 Status: Healthy\n02/06/2019 11:51:14 Copy Queue: 0\n02/06/2019 11:51:14 Replay Queue: 14124\n02/06/2019 11:51:14 Content Index: Healthy\n02/06/2019 11:51:14 DAG003-MDB159 is replay lagged on QQQXM0030\n02/06/2019 11:51:14 Replay lag is True\n02/06/2019 11:51:14 Truncation lag is False",
"tags": [
"UK",
"ELK Node 2",
"ELK",
"exchange"
]
},
"status": "Healthy",
"tags": [
"dagcheck",
"grokked",
"exchange"
],
"replayqueue": "14124"
},
"fields": {
"@timestamp": [
"2019-02-06T11:51:14.000Z"
],
"doc.@timestamp": [
"2019-02-21T00:58:03.284Z"
]
},
"sort": [
1549453874000
]
}
If I switch the output from 'filebeat > file' to 'filebeat > logstash' and configure my logstash pipeline to listen for beats I get an Error (actually get lots of errors and the below is an example)
2019-02-22T13:37:42,839][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"02/06/2019 11:51:14 Database Copy: DAG003-MDB159\\QQQXM0030\n02/06/2019 11:51:14 Server: QQQXM0030\n02/06/2019 11:51:14 Activation Preference: 4\n02/06/2019 11:51:14 Status: Healthy\n02/06/2019 11:51:14 Copy Queue: 0\n02/06/2019 11:51:14 Replay Queue: 14124\n02/06/2019 11:51:14 Content Index: Healthy\n02/06/2019 11:51:14 DAG003-MDB159 is replay lagged on QQQXM0030\n02/06/2019 11:51:14 Replay lag is True\n02/06/2019 11:51:14 Truncation lag is False", :exception=>#<LogStash::Json::ParserError: Invalid numeric value: Leading zeroes not allowed
at [Source: (byte[])"02/06/2019 11:51:14 Database Copy: DAG003-MDB159\QQQXM0030
Confused as it's the same filtering in logstash. Just changing the input to beats seems to break it??