Logstash KV filter time field

hello guys, i'm using kv filter to filter syslogs in logstash to be sent to elasticsearch. but the kv is creating a field with time and data instead of being in value, how to fix this?
is reaching the field limit because one is being created every time

image
image

here is my settings:

filter {
      kv {
        field_split => "||"
        value_split => "="
        allow_duplicate_values => false
        source => "message"
      }

If your data is not key-value pairs (and it is obviously not) then use grok or dissect to extract the key-value data from the message. See this example.

hello @Badger , thanks for the help, I saw the link to the example you sent but I couldn't implement it, can you help me with the filter?
anyway, thanks!

You have not given any indication of what your data looks like, so I cannot possibly suggest a pattern to match it.

sorry, i just forgot, here it is:

"<13>Jul 20 15:27:30 10.241.20.6 time=1626801326|hostname=MGMT-XX|severity=Informational|confidence_level=Medium|product=IPS|action=Detect|ifdir=inbound|ifname=lo|loguid={9x10f3e5yy,0x1c,0x6475cf72,0xeec2e5bc}|origin=xx.xxx.131.3|originsicname=CN\\=FW01-Out,O\\=MGJ.br.atootc|sequencenum=42|time=1626801326|version=5|attack=SSL Enforcement Violation|attack_info=OpenSSL ChaCha20_Poly1305 Cipher Suites|description_url=CVE_2016_7054_help.html|dst=xxx.xx.35.20|https_inspection_action=Inspect|industry_reference=CVE-2016-7054|lastupdatetime=1626805676|log_id=2|malware_rule_id={13B84A4D-2280-4C37-A24E-6FD1377AE144}|performance_impact=3|policy=FRA-POLICY|policy_time=1625860876|protection_id=asm_dynamic_prop_CVE_2016_7054|protection_name=OpenSSL ChaCha20_Poly1305 Cipher Suites|protection_type=IPS|proto=6|received_bytes=3000|rule_name=Webmail|rule_uid=584d05b2-3722-4729-XXXX-XXXX1f7a7b4|s_port=591XX|sent_bytes=1947|service=XXX|session_id={0x60f6e642e,0x3c,0x6475cf72,0xeec2e5bc}|smartdefense_profile=XXXX-IDS|src=xxx.xxx.249.124|suppressed_logs=30|layer_name=IPS|layer_name=F-POLICY Threat Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat  Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat Prevention|layer_uuid={364B9452-D032-4D02-8358-XXXXXXX}

and here is my output conf:

output {
  elasticsearch { hosts => ["https://XX.XX.XX.160:9200"]
  codec => cef
  user => "xxxxxx"
  password => "XXXXXXXXX"
  ssl_certificate_verification => false
  index => "syslog" }
  stdout { codec => rubydebug }

Please do not start multiple threads for the same request.

1 Like