Rsyslog to remote elk

[Ayah]

Dears,

I am new in ELK, we have ELK system to collect logs from a radsecproxy.log in other server. the logstash is configured to accept radsecproxy log format only and shapping them.

Now we need to forward same logs to another remote server using rsyslog,
So, we have decided to change the logDistination of radsecproxy to go to rsyslog then take a copy to radsecproxy.log then to go to logstash.

the logs coming directly from radsecproxy to logstash was starting like below format
Jun 29 08:49:32 2022: Access-Accept for user ..... (acceptable formate)

when we changed to rsyslog it become like below.
Nov 3 14:26:43 eduroam2 radsecproxy[30987]: Access-Accept for user .... (Not acceptable )

any body have idea to solve this challenge.

[warkolm (Mark Walkom)]

Welcome to our community!

Are you saying that when you changed to use rsyslog the format changed? If that is the case then it's likely rsyslog making those changes and not the Elastic Stack and you may need to ask the rsyslog community.

However you might also find someone here that has experience with this so posting more information, like your rsyslog config, might get you an answer.

[Rios]

Can you show:

• few lines as samples
• your Logstash .conf file

No need for ES IP/hostname, username or pass, just to see how things are configured.
Might be your @timestemp is not used from source rsyslog but from default LS time.

[Ayah]

below is the filter config which written to filter radsecproxy logs format.

 if [fields][server] == "radsecproxy" and [fields][logtype] == "radsecproxy" {
    grok {
      # Parse radsecproxy logs (modified to include Operator_Name and CUI)
      # Pattern as below
      # Jul 27 02:11:24 2018: Access-Accept for user testuser@nus.edu.sg stationid 02-00-00-00-00-01 from SG_NRS_1 to 172.23.4.1 (172.23.4.1) (Operator_Name reannz.co.nz) (CUI 6e931447d4a9a42153d19ef3c0c8875d31b98e69)
      # NOTE:
      #  * timestamp - custom expression built on grok basics
      #  * reply_packet_type: Access-Acept OR Access-Reject (OR possibly Accounting-Response)
      #  * logusername: either full username or just from '@' onwards - either is OK
      #    (empty user part accepted when parsing username to get realm)
      #    TODO: support empty local username when parsing email address
      #  * OPTIONAL: stationid: may be MAC, "undisclosed", partly-hashed (preserving vendor part of the MAC),
      #    or fully hashed
      #    - e.g., 02-00-00-fe0a29e37dd070ea6c6192380e323bf8810d2ee4e61ccc28db59ee9494c90d1d for vendor-hashed
      #    - may be completely skipped when the rq has now Calling_Station_Id attribute
      #  * reply_server_name: (conf entry of REPLY server) OR "_self_"
      #  * OPTIONAL: reply_msg: optionally enclosed in "()"
      #    * enclosed as long as asprintf does not fail, so lets say always encloded in "()"
      #    * directly following reply_server_name
      #  * client_short_name: conf name of original RQ server
      #  * client_server_ip: IP address of RQ server (NOTE: this is different from packet_src_ip in FreeRadius linelog)
      #  And our custom extension (make them optional)
      #  * operator_name: logged as-is (so with Namespace_ID prefix) (or blank)
      #  * CUI: logged as-is (or blank)
      #
      # NOTE: replylog() in radsecproxy can also log two other messages - one when username is missing, and one
      # when closing an Access-Request without having received a Response.
      # A request without a username is not relevant from eduroam perspective,
      # so we are not interested in a response to it...
      match => { "message" => "(?<timestamp>[A-Z][a-z]{2}  ?\d{1,2} %{TIME} %{YEAR}): %{USERNAME:reply_packet_type} for user (?<username>[^@ ]*@%{HOSTNAME})(?: stationid (?<calling_station_id>[a-zA-Z0-9:_-]*))? from (?<reply_server_name>[^()]+)(?:\((?<reply_msg>[^() ]*)\))? to %{NOTSPACE:client_short_name} \(%{IP:request_server_ip}\)(?: \(Operator_Name 1?%{USERNAME:operator_name}?\) \(CUI (?:%{USERNAME:chargeable_user_identity})?\))?" }

      # we have no inner message to replace so leave this out
      #overwrite => [ "message" ]
    }
    if [fields][timezone] {
      mutate {
        add_field => { "timestamp_with_timezone" => "%{timestamp} %{[fields][timezone]}" }
      }
      date {
        match => [ "timestamp_with_timezone", "MMM dd HH:mm:sss YYYY ZZZ", "MMM  d HH:mm:sss YYYY ZZZ"]
      }
    } else {
      date {
        # use separate pattern for single-digit day-in-month ("Jul  1")
        match => [ "timestamp", "MMM dd HH:mm:sss YYYY", "MMM  d HH:mm:sss YYYY" ]
      }
    }

    # parse username/realm
    if [username] {
      grok {
        # Parse username: get realm
        match => { "username" => "[^@]*@%{USERNAME:realm}" }
      }
      grok {
        # Parse realm: get country
        match => { "realm" => "[-_\.a-zA-Z0-9]*\.(?<realm_country>[-_a-zA-Z0-9]+)" }
      }
    }
    # END parse username/realm

    # add realm_country_is_local
    if [realm_country] {
      if [realm_country] == "om" {
        mutate { add_field => { "realm_country_is_local" => 1 } }
      } else {
        mutate { add_field => { "realm_country_is_local" => 0 } }
      }
    }

    # parse operator_name/site
    if [operator_name] {
      grok {
        # Parse operator_name/site: get country
        match => { "operator_name" => "[-_\.a-zA-Z0-9]*\.(?<site_country>[-_a-zA-Z0-9]+)" }
      }
    }
    # END parse operator_name/site

    # add site_country_is_local
    if [site_country] {
      if [site_country] == "om" {
        mutate { add_field => { "site_country_is_local" => 1 } }
      } else {
        mutate { add_field => { "site_country_is_local" => 0 } }
      }
    }