Thanks for your response @leandrojmp.
Yes, I am trying to send Wazuh agent logs which gets collected on archives.json file on Wazuh manager. I have installed Filebeat to ship logs from archives.json to logstash.
I am now able to ship the logs from archives.json to logstash but there seems to be a parsing or delimiter issue.
This is the log I am getting on logstash -
`{"log":{"file":{"path":"/var/ossec/logs/archives/archives.json"},"offset":65400666},"ecs":{"version":"1.11.0"},"event":{"original":"{\"timestamp\":\"2023-02-22T11:12:21.971+0800\",\"rule\":{\"level\":8,\"description\":\"New user added to the system.\",\"id\":\"5902\",\"mitre\":{\"id\":[\"T1136\"],\"tactic\":[\"Persistence\"],\"technique\":[\"Create Account\"]},\"firedtimes\":1,\"mail\":false,\"groups\":[\"syslog\",\"adduser\"],\"pci_dss\":[\"10.2.7\",\"10.2.5\",\"8.1.2\"],\"gpg13\":[\"4.13\"],\"gdpr\":[\"IV_35.7.d\",\"IV_32.2\"],\"hipaa\":[\"164.312.b\",\"164.312.a.2.I\",\"164.312.a.2.II\"],\"nist_800_53\":[\"AU.14\",\"AC.7\",\"AC.2\",\"IA.4\"],\"tsc\":[\"CC6.8\",\"CC7.2\",\"CC7.3\"]},\"agent\":{\"id\":\"001\",\"name\":\"bllogstash-virtual-machine\",\"ip\":\"10.x.x.x\"},\"manager\":{\"name\":\"wazuh-virtual-machine\"},\"id\":\"1677035541.34568\",\"full_log\":\"Feb 22 11:12:21 bllogstash-virtual-machine useradd[2292151]: new user: name=cool, UID=1002, GID=1002, home=/home/cool, shell=/bin/bash, from=/dev/pts/2\",\"predecoder\":{\"program_name\":\"useradd\",\"timestamp\":\"Feb 22 11:12:21\",\"hostname\":\"bllogstash-virtual-machine\"},\"decoder\":{\"parent\":\"useradd\",\"name\":\"useradd\"},\"data\":{\"dstuser\":\"cool\",\"uid\":\"1002\",\"gid\":\"1002\",\"home\":\"/home/cool\",\"shell\":\"/bin/bash,\"},\"location\":\"/var/log/auth.log\"}"},"message":"{\"timestamp\":\"2023-02-22T11:12:21.971+0800\",\"rule\":{\"level\":8,\"description\":\"New user added to the system.\",\"id\":\"5902\",\"mitre\":{\"id\":[\"T1136\"],\"tactic\":[\"Persistence\"],\"technique\":[\"Create Account\"]},\"firedtimes\":1,\"mail\":false,\"groups\":[\"syslog\",\"adduser\"],\"pci_dss\":[\"10.2.7\",\"10.2.5\",\"8.1.2\"],\"gpg13\":[\"4.13\"],\"gdpr\":[\"IV_35.7.d\",\"IV_32.2\"],\"hipaa\":[\"164.312.b\",\"164.312.a.2.I\",\"164.312.a.2.II\"],\"nist_800_53\":[\"AU.14\",\"AC.7\",\"AC.2\",\"IA.4\"],\"tsc\":[\"CC6.8\",\"CC7.2\",\"CC7.3\"]},\"agent\":{\"id\":\"001\",\"name\":\"bllogstash-virtual-machine\",\"ip\":\"10.x.x.x\"},\"manager\":{\"name\":\"wazuh-virtual-machine\"},\"id\":\"1677035541.34568\",\"full_log\":\"Feb 22 11:12:21 bllogstash-virtual-machine useradd[2292151]: new user: name=cool, UID=1002, GID=1002, home=/home/cool, shell=/bin/bash, from=/dev/pts/2\",\"predecoder\":{\"program_name\":\"useradd\",\"timestamp\":\"Feb 22 11:12:21\",\"hostname\":\"bllogstash-virtual-machine\"},\"decoder\":{\"parent\":\"useradd\",\"name\":\"useradd\"},\"data\":{\"dstuser\":\"cool\",\"uid\":\"1002\",\"gid\":\"1002\",\"home\":\"/home/cool\",\"shell\":\"/bin/bash,\"},\"location\":\"/var/log/auth.log\"}","agent":{"ephemeral_id":"ca8b925c-0141-42b1-87ba-e5daa218884a","name":"wazuh-virtual-machine","id":"7455124a-f2c5-482a-9ca4-a3e8f26710b8","type":"filebeat","hostname":"wazuh-virtual-machine","version":"7.15.1"},"type":"wazuh_log","host":{"os":{"kernel":"5.19.0-32-generic","platform":"ubuntu","family":"debian","name":"Ubuntu","codename":"jammy","type":"linux","version":"22.04 (Jammy Jellyfish)"},"containerized":false,"name":"wazuh-virtual-machine","mac":["00:0c:29:15:a4:1e"],"ip":["10.x.x.x","fe80::aa26:e157:e495:c628"],"architecture":"x86_64","id":"24ed17383d96472da3dc8e2f251f0333","hostname":"wazuh-virtual-machine"},"@timestamp":"2023-02-22T03:12:22.795Z","tags":["beats_input_codec_plain_applied"],"@version":"1","input":{"type":"log"}}`
Here is my logstash configs-
input {
beats {
port => xxx
type => wazuh_log
}
output {
if [type] == "wazuh_log" {
kafka {
bootstrap_servers => "10.x.x.x:xxx"
topic_id => "wazuh"
codec => "json"
}
file {
path => "/var/log/syslog/wazuh-%{+YYYY-MM-dd}.log"
}
}
}
Can you suggest if I can add any parsing logic or delimiter here ?
The delimiter I added earlier is not helping, hence I removed it -
codec => json_lines {
delimiter => "\0"
}
Thanks.
