How to strip events from message field? Logstash syslog

Elastic Stack Logstash
1

hello guys, i'm trying to send logs from a syslog-ng server to logstash and logstash to elasticsearch, but all events are coming in the message field. Is it possible to separate these events to create a new field for each thing?

2

You would need to use a grok filter in Logstash or grok ingest processor on the elasticsearch side to parse the message into fields.

3

would the kv filter be more viable? because my logs are huge and creating a filter with grok for each field would be almost impossible,I believe that some automatic way of separating would be more viable, do you agree?

4

Most common logs have a preset where you don't need to parse it all out yourself. What type of logs are these?

For example.

5

syslog-ng logs..

"<13>Jul 20 15:27:30 10.241.20.6 time=1626801326|hostname=MGMT-XX|severity=Informational|confidence_level=Medium|product=IPS|action=Detect|ifdir=inbound|ifname=lo|loguid {9x10f3e5yy,0x1c,0x6475cf72,0xeec2e5bc}|origin=xx.xxx.131.3|originsicname=CN|FW01-Out,O|=MGJ.br.atootc|sequencenum=42|time=1626801326|version=5|attack=SSL Enforcement Violation|attack_info=OpenSSL ChaCha20_Poly1305 Cipher Suites|description_url=CVE_2016_7054_help.html|dst=xxx.xx.35.20|https_inspection_action=Inspect|industry_reference=CVE-2016-7054|lastupdatetime=1626805676|log_id=2|malware_rule_id={13B84A4D-2280-4C37-A24E-6FD1377AE144}|performance_impact=3|policy=FRA-POLICY|policy_time=1625860876|protection_id=asm_dynamic_prop_CVE_2016_7054|protection_name=OpenSSL ChaCha20_Poly1305 Cipher Suites|protection_type=IPS|proto=6|received_bytes=3000|rule_name=Webmail|rule_uid=584d05b2-3722-4729-XXXX-XXXX1f7a7b4|s_port=591XX|sent_bytes=1947|service=XXX|session_id={0x60f6e642e,0x3c,0x6475cf72,0xeec2e5bc}|smartdefense_profile=XXXX-IDS|src=xxx.xxx.249.124|suppressed_logs=30|layer_name=IPS|layer_name=F-POLICY Threat Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat  Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat Prevention|layer_uuid={364B9452-D032-4D02-8358-XXXXXXX}
6

I will answer here, rather than in this duplicate thread.

7

You could try

    dissect { mapping => { "message" => "<%{syslogPri}>%{[@metadata][ts]} %{+[@metadata][ts]} %{+[@metadata][ts]} %{ip} %{[@metadata][restOfLine]}" } }

    date { match => [ "[@metadata][ts]", "MMM dd HH:mm:ss" ] }
    kv { source => "[@metadata][restOfLine]" field_split => "|" }

Note that some of your fields are repeated (layer_name and time) so they will be arrays.

1 Like
8

Thank you @Badger ! this solution solved my problem

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.