Hi @Kaiyan_Sheng
I'm using this version of filebeat because in my compagny, the ES version which is used in 6.4...
I found a trick which is pretty dirty (and not totally satisfying to be honnest, but I can find now my parsed logs in Kibana).
So, as I explained before, the yaml format of the ingest pipeline for ActiveMq module seems to be not recognised :
Error getting pipeline for fileset activemq/log: Error JSON decoding the pipeline file: ingest/pipeline.yml: invalid character '#' looking for beginning of value
So what I did (which as I said is pretty dirty), is to "put" a Json in the file called "pipeline.yml"... And now it works (almost).
So the basic config for pipeline.yml was :
---
description: Pipeline for parsing ActiveMQ logs.
processors:
- grok:
field: message
pattern_definitions:
GREEDYMULTILINE: "(.|\n|\t)"
NOPIPEGREEDYDATA: "((?! \|).)"
THREAD_NAME: "((?! \n).)*"
patterns:
- "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\|%{SPACE}%{LOGLEVEL:log.level}%{SPACE}\|%{SPACE}%{NOPIPEGREEDYDATA:message}%{SPACE}\|%{SPACE}%{NOPIPEGREEDYDATA:activemq.caller}%{SPACE}\|%{SPACE}%{THREAD_NAME:activemq.thread}%{SPACE}?%{GREEDYMULTILINE:activemq.log.stack_trace}"
ignore_missing: true
- date:
if: "ctx.event.timezone == null"
field: timestamp
target_field: "@timestamp"
formats: ["yyyy-MM-dd HH:mm:ss,SSS"]
- date:
if: "ctx.event.timezone != null"
field: "timestamp"
target_field: "@timestamp"
timezone: "{{ event.timezone }}"
formats: ["yyyy-MM-dd HH:mm:ss,SSS"]
- remove:
field:
- timestamp
on_failure:
- set:
field: error.message
value: "{{ _ingest.on_failure_message }}"
And now, this pipeline.yml file is
{
"description": "Pipeline for parsing ActiveMQ logs.",
"processors": [
{
"grok": {
"field": "message",
"pattern_definitions": {
"GREEDYMULTILINE": "(.|\\n|\\t)*",
"NOPIPEGREEDYDATA": "((?! \\|).)*",
"THREAD_NAME": "((?! \n).)*"
},
"patterns": [
"%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\\|%{SPACE}%{LOGLEVEL:log.level}%{SPACE}\\|%{SPACE}%{NOPIPEGREEDYDATA:message}%{SPACE}\\|%{SPACE}%{NOPIPEGREEDYDATA:activemq.caller}%{SPACE}\\|%{SPACE}%{THREAD_NAME:activemq.thread}%{SPACE}?%{GREEDYMULTILINE:activemq.log.stack_trace}"
],
"ignore_missing": true
}
},
{
"remove": {
"field": [
"timestamp"
]
}
}
],
"on_failure": [
{
"set": {
"field": "error.message",
"value": "{{ _ingest.on_failure_message }}"
}
}
]
}
But as you can see, some informations in the json are missing...
Regards.
