# How to Remove Prefixes and Ensure Parsing Integrity When Forwarding Logs to QRadar Using Logstash

**URL:** <https://discuss.elastic.co/t/how-to-remove-prefixes-and-ensure-parsing-integrity-when-forwarding-logs-to-qradar-using-logstash/363385>\
**Category:** Logstash\
**Created:** [July 19, 2024, 3:21am UTC](https://discuss.elastic.co/t/how-to-remove-prefixes-and-ensure-parsing-integrity-when-forwarding-logs-to-qradar-using-logstash/363385 "2024-07-19T03:21:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![wangsubo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangsubo/32/133865_2.png) [@wangsubo](https://discuss.elastic.co/u/wangsubo)\
**Post date:** [July 19, 2024, 3:21am UTC](https://discuss.elastic.co/t/how-to-remove-prefixes-and-ensure-parsing-integrity-when-forwarding-logs-to-qradar-using-logstash/363385/1 "2024-07-19T03:21:34Z")

</div>

I have two questions that need answers:

1. When forwarding logs from Logstash to QRadar, the logs come with a Logstash prefix. How can we remove this prefix?

2. Using Elastic-Agent to send logs to Elasticsearch allows automatic log parsing (parser fields). If Elastic-Agent -\> Logstash -\> Elasticsearch and uses Logstash as an intermediary, will it affect the automatic log parsing fields?

Hope someone with experience can help answer, thanks

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/2/a2dfcdb9d54e74ca46e24b214e08b99427a03682.png)

```auto
input {
  elastic_agent {
    port => 5044
    ssl_enabled => true
    ssl_certificate_authorities => ["/etc/logstash/certs/elasticsearch-ca.pem"]
    ssl_certificate => "/etc/logstash/certs/logstash.crt"
    ssl_key => "/etc/logstash/certs/logstash.pkcs8.key"
    ssl_client_authentication => "required"
  }
}

filter {
  if "syslog" in [tags] {
    grok {
      match => ["message", "%{SYSLOG5424PRI:syslog_index}%{GREEDYDATA:message}"]
      overwrite => ["message"]
    }
  }
}

output {
  stdout {
    codec => line {
      format => "%{message}"
    }
  }
  syslog {
    host => "192.168.3.180"
    port => 514
    protocol => "udp"
    codec => line {
      format => "%{message}"
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 19, 2024, 3:50am UTC](https://discuss.elastic.co/t/how-to-remove-prefixes-and-ensure-parsing-integrity-when-forwarding-logs-to-qradar-using-logstash/363385/2 "2024-07-19T03:50:38Z")

</div>

> [@wangsubo](#):
>
> When forwarding logs from Logstash to QRadar, the logs come with a Logstash prefix. How can we remove this prefix?

I'm not sure you can remove it if you use the `syslog` output as the output will add it.

But maybe if you switch to the plain `udp` output, something like this:

```auto
  udp {
    host => "192.168.3.180"
    port => 514
    codec => line {
      format => "%{message}"
    }
  }

```

> [@wangsubo](#):
>
> If Elastic-Agent -\> Logstash -\> Elasticsearch and uses Logstash as an intermediary, will it affect the automatic log parsing fields?

If you change the original message yes, it can affect and break the parsing in Elasticsearch side depending on the changes.

All Elastic Agent integrations uses Ingest pipelines to parse the messages, those pipelines expect to receive the raw message from the source, if you do any parsing on the message in logstash it may change the raw message to a format that the ingest pipeline cannot parse it.

---

<div class="post-metadata">

**Author:** ![wangsubo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangsubo/32/133865_2.png) [@wangsubo](https://discuss.elastic.co/u/wangsubo)\
**Post date:** [July 19, 2024, 6:34am UTC](https://discuss.elastic.co/t/how-to-remove-prefixes-and-ensure-parsing-integrity-when-forwarding-logs-to-qradar-using-logstash/363385/3 "2024-07-19T06:34:41Z")

</div>

> Thanks, this successfully removed the prefix in Logstash. However, as shown in the initial photo, there were two prefixes.

```auto
  udp {
    host => "192.168.3.180"
    port => 514
    codec => line {
      format => "%{message}"
    }
  }

```

> I used mutate to remove the Elastic-Agent prefix, but it seems to have no effect.  
> Do you have a better method?

```auto
filter {
  grok {
    match => { "message" => "%{SYSLOG5424PRI}%{GREEDYDATA:message}" }
    overwrite => ["message"]
  }

  mutate {
    gsub => [
      "message", "^\\w{3} \\d{2} \\d{2}:\\d{2}:\\d{2}", "",
      "message", "\\b\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\b", ""
    ]
  }
}

```

> If you change the original message yes, it can affect and break the parsing in Elasticsearch side depending on the changes.
> 
> All Elastic Agent integrations uses Ingest pipelines to parse the messages, those pipelines expect to receive the raw message from the source, if you do any parsing on the message in logstash it may change the raw message to a format that the ingest pipeline cannot parse it.

`I probably won't change the original messages from the devices, but 1. I might remove the Elastic-agent prefix in Logstash, and 2. I categorize firewall logs into Traffic and UTM (Threat). The Traffic logs will be sent to Elasticsearch, while the UTM (Threat) logs will be sent to SIEM via UDP. My concern is whether these Logstash filters that remove parts of the log could impact the parsing by Elastic Agent integrations, indirectly causing issues in Elasticsearch's ability to parse the logs`