How to Remove Prefixes and Ensure Parsing Integrity When Forwarding Logs to QRadar Using Logstash

Elastic Stack Logstash
1

I have two questions that need answers:

  1. When forwarding logs from Logstash to QRadar, the logs come with a Logstash prefix. How can we remove this prefix?

  2. Using Elastic-Agent to send logs to Elasticsearch allows automatic log parsing (parser fields). If Elastic-Agent -> Logstash -> Elasticsearch and uses Logstash as an intermediary, will it affect the automatic log parsing fields?

Hope someone with experience can help answer, thanks

image
image939×302 82.8 KB

input {
  elastic_agent {
    port => 5044
    ssl_enabled => true
    ssl_certificate_authorities => ["/etc/logstash/certs/elasticsearch-ca.pem"]
    ssl_certificate => "/etc/logstash/certs/logstash.crt"
    ssl_key => "/etc/logstash/certs/logstash.pkcs8.key"
    ssl_client_authentication => "required"
  }
}

filter {
  if "syslog" in [tags] {
    grok {
      match => ["message", "%{SYSLOG5424PRI:syslog_index}%{GREEDYDATA:message}"]
      overwrite => ["message"]
    }
  }
}

output {
  stdout {
    codec => line {
      format => "%{message}"
    }
  }
  syslog {
    host => "192.168.3.180"
    port => 514
    protocol => "udp"
    codec => line {
      format => "%{message}"
    }
  }
}
2

I'm not sure you can remove it if you use the syslog output as the output will add it.

But maybe if you switch to the plain udp output, something like this:

  udp {
    host => "192.168.3.180"
    port => 514
    codec => line {
      format => "%{message}"
    }
  }

If you change the original message yes, it can affect and break the parsing in Elasticsearch side depending on the changes.

All Elastic Agent integrations uses Ingest pipelines to parse the messages, those pipelines expect to receive the raw message from the source, if you do any parsing on the message in logstash it may change the raw message to a format that the ingest pipeline cannot parse it.

3

Thanks, this successfully removed the prefix in Logstash. However, as shown in the initial photo, there were two prefixes.

  udp {
    host => "192.168.3.180"
    port => 514
    codec => line {
      format => "%{message}"
    }
  }

I used mutate to remove the Elastic-Agent prefix, but it seems to have no effect.
Do you have a better method?

filter {
  grok {
    match => { "message" => "%{SYSLOG5424PRI}%{GREEDYDATA:message}" }
    overwrite => ["message"]
  }

  mutate {
    gsub => [
      "message", "^\\w{3} \\d{2} \\d{2}:\\d{2}:\\d{2}", "",
      "message", "\\b\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\b", ""
    ]
  }
}

If you change the original message yes, it can affect and break the parsing in Elasticsearch side depending on the changes.

All Elastic Agent integrations uses Ingest pipelines to parse the messages, those pipelines expect to receive the raw message from the source, if you do any parsing on the message in logstash it may change the raw message to a format that the ingest pipeline cannot parse it.

I probably won't change the original messages from the devices, but 1. I might remove the Elastic-agent prefix in Logstash, and 2. I categorize firewall logs into Traffic and UTM (Threat). The Traffic logs will be sent to Elasticsearch, while the UTM (Threat) logs will be sent to SIEM via UDP. My concern is whether these Logstash filters that remove parts of the log could impact the parsing by Elastic Agent integrations, indirectly causing issues in Elasticsearch's ability to parse the logs