How to strip events from message field? Logstash syslog

hello guys, i'm trying to send logs from a syslog-ng server to logstash and logstash to elasticsearch, but all events are coming in the message field. Is it possible to separate these events to create a new field for each thing?

You would need to use a grok filter in Logstash or grok ingest processor on the elasticsearch side to parse the message into fields.

would the kv filter be more viable? because my logs are huge and creating a filter with grok for each field would be almost impossible,I believe that some automatic way of separating would be more viable, do you agree?

Most common logs have a preset where you don't need to parse it all out yourself. What type of logs are these?

For example.

syslog-ng logs..

"<13>Jul 20 15:27:30 time=1626801326|hostname=MGMT-XX|severity=Informational|confidence_level=Medium|product=IPS|action=Detect|ifdir=inbound|ifname=lo|loguid {9x10f3e5yy,0x1c,0x6475cf72,0xeec2e5bc}||originsicname=CN|FW01-Out,O||sequencenum=42|time=1626801326|version=5|attack=SSL Enforcement Violation|attack_info=OpenSSL ChaCha20_Poly1305 Cipher Suites|description_url=CVE_2016_7054_help.html|dst=xxx.xx.35.20|https_inspection_action=Inspect|industry_reference=CVE-2016-7054|lastupdatetime=1626805676|log_id=2|malware_rule_id={13B84A4D-2280-4C37-A24E-6FD1377AE144}|performance_impact=3|policy=FRA-POLICY|policy_time=1625860876|protection_id=asm_dynamic_prop_CVE_2016_7054|protection_name=OpenSSL ChaCha20_Poly1305 Cipher Suites|protection_type=IPS|proto=6|received_bytes=3000|rule_name=Webmail|rule_uid=584d05b2-3722-4729-XXXX-XXXX1f7a7b4|s_port=591XX|sent_bytes=1947|service=XXX|session_id={0x60f6e642e,0x3c,0x6475cf72,0xeec2e5bc}|smartdefense_profile=XXXX-IDS||suppressed_logs=30|layer_name=IPS|layer_name=F-POLICY Threat Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat  Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat Prevention|layer_uuid={364B9452-D032-4D02-8358-XXXXXXX}

I will answer here, rather than in this duplicate thread.

You could try

    dissect { mapping => { "message" => "<%{syslogPri}>%{[@metadata][ts]} %{+[@metadata][ts]} %{+[@metadata][ts]} %{ip} %{[@metadata][restOfLine]}" } }

    date { match => [ "[@metadata][ts]", "MMM dd HH:mm:ss" ] }
    kv { source => "[@metadata][restOfLine]" field_split => "|" }

Note that some of your fields are repeated (layer_name and time) so they will be arrays.

1 Like

Thank you @Badger ! this solution solved my problem