Logstash KV filter time field

hello guys, i'm using kv filter to filter syslogs in logstash to be sent to elasticsearch. but the kv is creating a field with time and data instead of being in value, how to fix this?
is reaching the field limit because one is being created every time


here is my settings:

filter {
      kv {
        field_split => "||"
        value_split => "="
        allow_duplicate_values => false
        source => "message"

If your data is not key-value pairs (and it is obviously not) then use grok or dissect to extract the key-value data from the message. See this example.

hello @Badger , thanks for the help, I saw the link to the example you sent but I couldn't implement it, can you help me with the filter?
anyway, thanks!

You have not given any indication of what your data looks like, so I cannot possibly suggest a pattern to match it.

sorry, i just forgot, here it is:

"<13>Jul 20 15:27:30 time=1626801326|hostname=MGMT-XX|severity=Informational|confidence_level=Medium|product=IPS|action=Detect|ifdir=inbound|ifname=lo|loguid={9x10f3e5yy,0x1c,0x6475cf72,0xeec2e5bc}|origin=xx.xxx.131.3|originsicname=CN\\=FW01-Out,O\\=MGJ.br.atootc|sequencenum=42|time=1626801326|version=5|attack=SSL Enforcement Violation|attack_info=OpenSSL ChaCha20_Poly1305 Cipher Suites|description_url=CVE_2016_7054_help.html|dst=xxx.xx.35.20|https_inspection_action=Inspect|industry_reference=CVE-2016-7054|lastupdatetime=1626805676|log_id=2|malware_rule_id={13B84A4D-2280-4C37-A24E-6FD1377AE144}|performance_impact=3|policy=FRA-POLICY|policy_time=1625860876|protection_id=asm_dynamic_prop_CVE_2016_7054|protection_name=OpenSSL ChaCha20_Poly1305 Cipher Suites|protection_type=IPS|proto=6|received_bytes=3000|rule_name=Webmail|rule_uid=584d05b2-3722-4729-XXXX-XXXX1f7a7b4|s_port=591XX|sent_bytes=1947|service=XXX|session_id={0x60f6e642e,0x3c,0x6475cf72,0xeec2e5bc}|smartdefense_profile=XXXX-IDS|src=xxx.xxx.249.124|suppressed_logs=30|layer_name=IPS|layer_name=F-POLICY Threat Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat  Prevention|layer_name=IPS|layer_name=FRA-POLICY Threat Prevention|layer_uuid={364B9452-D032-4D02-8358-XXXXXXX}

and here is my output conf:

output {
  elasticsearch { hosts => ["https://XX.XX.XX.160:9200"]
  codec => cef
  user => "xxxxxx"
  password => "XXXXXXXXX"
  ssl_certificate_verification => false
  index => "syslog" }
  stdout { codec => rubydebug }

Please do not start multiple threads for the same request.

1 Like