Hi there,
Im new to ELK, and trying to extract some custom log no covered by filebeat module into kibana. So I tried to create a ingest node pipeline between filebeat and ES.
So on filebeat.yml: 1) Add the pipeline name on the elasticoutputsection 2) add the logpath for the custom log in the filebeat input
I created the ingest node where I'm using grok processors to parse vsftpd.log and it work with dev tools (where I had to transform the query from grok debugger because it need to be json format).
I want to test with the simulate but the "message" section isn't in json format. I don't understand why it should be in json format as the message is a redirect from vsftpd.log
Completlely lost on how it should work, even after reading documentation. On filebeat side, I see a line that pipeline output is connected to elastic, but no logs are redirect to the default filebeat index.
Here is the simulate output if anyone can help would be appreciate !
POST _ingest/pipeline/_simulate
{
"pipeline": {
"processors": [
{
"grok": {
"field": "message",
"patterns": ["%{SYSLOGTIMESTAMP:timestamp}%{SPACE}+%{YEAR:year}%{SPACE}+.pid%{SPACE}+%{GREEDYDATA:pid}.%{SPACE}+.%{USERNAME:username}.%{SPACE}+%{GREEDYDATA:log_message}%{SPACE}","::ffff:%{IP:sourceIP}","%{SPACE}\"%{PATH:filepath}\",%{SPACE}%{GREEDYDATA:Packet_Size_and_Speed}"]
}
}
]
},
"docs":[
{
"_source": {
"message": "Thu Nov 17 17:31:54 2022 [pid 7512] [rami] OK DOWNLOAD: Client "::ffff:192.168.1.98", "/home/rami/test2", 5 bytes, 8.91Kbyte/sec"
}
}
]
}
Thanks you