ES 7.15 FileBeats Sophos XG module not separating data into variables

Elastic Stack Beats
1

Following This link to configure sophos Module.
www.elastic.co/.../filebeat-module-sophos.html

I followed the documentation at www.elastic.co/.../filebeat-module-sophos.html and was able to get the data into ES, issue is I cannot see the fields (varibale) in the message. All data is inside a field called message.

I am using ES 7.15.1 / Filebeat 7.15.1 / Ubuntu 18.04 / Logstash 7.15.1

In high level I am using Filebeat Sophos-xg module, logstash and ES. I did enable sophos module, uploaded the template with commands "filebeat modules enable sophos" and "filebeat setup -e -E output.logstash.enabled=false -E output.Elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601" and I am getting the data, without data being split into fields/variables. I am not an ELK stack expert. Not sure if the issue is with filebeat module / logstash or somewhere else. Any troubleshooting steps let me know please?

Sample Message - I want what is inside message in variables
"message": "<30>device_name="SFW" timestamp="2021-11-07T18:14:59-0600" device_model="SFVH" device_serial_id="C01001BQC8XXXXX" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" duration=38 fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="10:BF:48:7D:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.10" src_country="R1" dst_ip="1.1.1.1" dst_country="AUS" protocol="ICMP" icmp_type=8 packets_sent=2 packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Stop" con_id="2907267392" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2"",

2

Hi @Samy_Weee Welcome to the community

First I would usually try to send direct from Filebeat -> Elasticsearch and get that working?

Once you get that working .. .then I would move on to this Architecture

Filebeat -> Logstash -> Elasticsearch

And the question is... Do you actually need logstash? (it is not required)

If you do ... once you have Filebeat -> Elasticsearch working go into your filebeat.yml and then direct filebeat output to logstash output.

Then your logstash.conf should look something like this... this is a "Passthrough" configuration what is most likely happening is that you are missing this line

pipeline => "%{[@metadata][pipeline]}" which tells Elasticsearch which pipeline (AKA Parser) to use. When you go direct from Filebeat -> Elasticsearch that info is automatically passed along .. when you put logstash in the middle it is not.

Start Logstash first, and then filebeat...

################################################
# beats->logstash->es default config.
################################################
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      user => "elastic"
      password => "secret"
    }
  }
}
3

@stephenb As explained.. It worked fine without logstash. Then it also worked with your modified logstash passthrough config. I wish I knew I do not need logstash, to get this working. Going forward, I will remove the logstash and only use filebeat as it seems more efficient that way for my purpose.

Just another question out of this: Is it possible to run 2 instances of filebeat with 2 configuration files, so that one instance can directly forward to ES, while second filebeat instance forward to LS?

Thank you very much for the help.

4

Apologies, I did not see anywhere in your initial post where you indicated it all worked fine with a direct filebeat-> elasticsearch integration.

Yes, the docs are not always easy to navigate, The quick start guides are meant to help.

Possible...yes... You would need basically need 2 separate installations if filebeat because it keeps track of files have been read etc..

Not clear why you would want to do that generally when folks want to send them more than one destination that is exactly when they use logstash... filebeat -> logstash to more than one output

5

@stephenb What I meant was I never tried it that way until you asked me to. It all started working after your guideline. I am learning. Thank you.

1 Like
6

I am able to feed data from sophos module using file beat to Elasticsearch. I have a different issue. There are fields which are marked as string in ES, which should have been a numeric . Here is an example. ; Variable: "sophos.xg.bytes_received" is coming in as a string variable. (My Flow : Sophos --> filebeat-sophos-module --> Elasticsearch)

I went into modules directory (filebeat/module/sophos/xg/ingest/sophos) to check the definition for this variable and it seems to be defined correctly as a type: long , where has this gone wrong. Why this field is not coming in as a long variable how to trouble-shoot. Most variables are strings, except few numeric.

- convert:
    field: sophos.xg.recv_bytes
    target_field: destination.bytes
    type: long
    ignore_failure: true
    ignore_missing: true
    if: "ctx.sophos?.xg?.recv_bytes != null"

sophos_es

See the screenshot above. I also do not see any field with name "destination.bytes" in Elasticsearch. But I do see sophos.xg.recv_bytes as a "t" - string field.

Thank you if someone can help please.

7

Did you run filebeat setup before you initially started filebeat? If not that could explain this behavior meaning the template was not loaded / applied before you started to send data and thus the mapping may not be correct.

8

Yes, 'filebeat setup -e' was run before I started filebeat with systemctl start filebeat (I use Ubuntu). Is there a step to check if relevant template is in place? What else I can check?

I ran the 'filebeat setup -e' again and I see following output towards the end of the run . Still my variables are of wrong type. Output seems to suggest it is loading the pipelines from the modules directory related to my module of concertn (sophos-xg)

....
....
2021-11-14T16:37:20.662-0600    INFO    [esclientleg]   eslegclient/connection.go:100   elasticsearch url: http://localhost:9200
2021-11-14T16:37:20.663-0600    INFO    [esclientleg]   eslegclient/connection.go:273   Attempting to connect to Elasticsearch version 7.15.1
2021-11-14T16:37:20.664-0600    INFO    [esclientleg]   eslegclient/connection.go:100   elasticsearch url: http://localhost:9200
2021-11-14T16:37:20.665-0600    INFO    [esclientleg]   eslegclient/connection.go:273   Attempting to connect to Elasticsearch version 7.15.1
2021-11-14T16:37:20.696-0600    INFO    [modules]       fileset/pipelines.go:133        Elasticsearch pipeline loaded.  {"pipeline": "filebeat-7.15.1-sophos-xg-pipeline"}
2021-11-14T16:37:20.709-0600    INFO    [modules]       fileset/pipelines.go:133        Elasticsearch pipeline loaded.  {"pipeline": "filebeat-7.15.1-sophos-xg-systemhealth"}
2021-11-14T16:37:20.727-0600    INFO    [modules]       fileset/pipelines.go:133        Elasticsearch pipeline loaded.  {"pipeline": "filebeat-7.15.1-sophos-xg-firewall"}
2021-11-14T16:37:20.745-0600    INFO    [modules]       fileset/pipelines.go:133        Elasticsearch pipeline loaded.  {"pipeline": "filebeat-7.15.1-sophos-xg-idp"}
2021-11-14T16:37:20.760-0600    INFO    [modules]       fileset/pipelines.go:133        Elasticsearch pipeline loaded.  {"pipeline": "filebeat-7.15.1-sophos-xg-atp"}
2021-11-14T16:37:20.774-0600    INFO    [modules]       fileset/pipelines.go:133        Elasticsearch pipeline loaded.  {"pipeline": "filebeat-7.15.1-sophos-xg-antivirus"}
2021-11-14T16:37:20.787-0600    INFO    [modules]       fileset/pipelines.go:133        Elasticsearch pipeline loaded.  {"pipeline": "filebeat-7.15.1-sophos-xg-sandstorm"}
2021-11-14T16:37:20.800-0600    INFO    [modules]       fileset/pipelines.go:133        Elasticsearch pipeline loaded.  {"pipeline": "filebeat-7.15.1-sophos-xg-cfilter"}
2021-11-14T16:37:20.814-0600    INFO    [modules]       fileset/pipelines.go:133        Elasticsearch pipeline loaded.  {"pipeline": "filebeat-7.15.1-sophos-xg-event"}
2021-11-14T16:37:20.827-0600    INFO    [modules]       fileset/pipelines.go:133        Elasticsearch pipeline loaded.  {"pipeline": "filebeat-7.15.1-sophos-xg-waf"}
2021-11-14T16:37:20.839-0600    INFO    [modules]       fileset/pipelines.go:133        Elasticsearch pipeline loaded.  {"pipeline": "filebeat-7.15.1-sophos-xg-antispam"}
2021-11-14T16:37:20.851-0600    INFO    [modules]       fileset/pipelines.go:133        Elasticsearch pipeline loaded.  {"pipeline": "filebeat-7.15.1-sophos-xg-wifi"}
2021-11-14T16:37:20.851-0600    INFO    cfgfile/reload.go:262   Loading of config files completed.
2021-11-14T16:37:20.851-0600    INFO    [load]  cfgfile/list.go:129     Stopping 1 runners ...
Loaded Ingest pipelines
9

That field is not set in every message are you sure you are looking at one of the messages that that is parsed for? Also have there been and changes to the Sophos config? you may meed to dig in and take a look at which pipeline is being called...

antispam.yml:    target_field: destination.bytes
antivirus.yml:    target_field: destination.bytes
event.yml:    target_field: destination.bytes
firewall.yml:    target_field: destination.bytes
waf.yml:    target_field: destination.bytes
10

Here is the input msg to the filebeat, this should get processed by firewall.yml module file ; I do not understand how it gets directed to firewall.yml file to process, but I do not see destination.bytes field in the received message in Kibana. I see a sophos.xg.recv_bytes field. I think the conversion step do not happen in the filebeat module or filebeat module don't know which .yml file to apply for the input, Where should I look? Thank you.

device_name="SFW" timestamp="2021-11-14T17:03:29-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFEB" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="8A:0D:85:58:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.136" src_country="R1" dst_ip="35.82.255.30" dst_country="USA" protocol="TCP" src_port=53440 dst_port=443 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Start" con_id="2425650304" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2"

No changes were made to sophos-xg firewall logging. I am using default logging feature available.

This may be where it is going wrong. Any post or guideline if you can point me to I can do some trouble shooting.

Appreciate your help.

11

The logic is in the

filebeat-7.15.1-sophos-xg-pipeline

Pipeline you can look at it in detail in Kibana -> Stack Management -> Ingest Pipelines

So I took the sample message you provided and put in a %{SYSLOG5424PRI} the <44> and it parsed it fine...

Your example does not have the bytes sent and received bytes in the source so I can not see what is / is not happening... you will need to get one of the right type and try it. and try this simulate...

POST /_ingest/pipeline/filebeat-7.15.2-sophos-xg-pipeline/_simulate
{
  "docs": [
    {
      "_index": "m-index",
      "_id": "kMpUTHoBr7SFhhL5-98P",
      "_source": {
        "@timestamp" : "2021-11-09T03:17:20.0516241Z",
        "message": """<44>device_name="SFW" timestamp="2021-11-14T17:03:29-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFEB" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="8A:0D:85:58:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.136" src_country="R1" dst_ip="35.82.255.30" dst_country="USA" protocol="TCP" src_port=53440 dst_port=443 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Start" con_id="2425650304" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2"""      }
    }
  ]
}

run this and you will see the results...

{
  "docs" : [
    {
      "doc" : {
        "_index" : "m-index",
        "_type" : "_doc",
        "_id" : "kMpUTHoBr7SFhhL5-98P",
        "_source" : {
          "server" : {
            "port" : 443,
            "mac" : "00:50:56:9F:FF:FF",
            "ip" : "35.82.255.30"
          },
          "log" : {
            "level" : "informational"
          },
          "destination" : {
            "geo" : {
              "continent_name" : "North America",
              "region_iso_code" : "US-OR",
              "city_name" : "Boardman",
              "country_iso_code" : "US",
              "country_name" : "United States",
              "region_name" : "Oregon",
              "location" : {
                "lon" : -119.7257,
                "lat" : 45.8234
              }
            },
            "as" : {
              "number" : 16509,
              "organization" : {
                "name" : "AMAZON-02"
              }
            },
            "port" : 443,
            "mac" : "00:50:56:9F:FF:FF",
            "ip" : "35.82.255.30"
          },
          "rule" : {
            "id" : "12"
          },
          "source" : {
            "port" : 53440,
            "mac" : "8A:0D:85:58:FF:FF",
            "ip" : "192.168.2.136"
          },
          "network" : {
            "transport" : "tcp"
          },
          "observer" : {
            "ingress" : {
              "interface" : {
                "name" : "Port1"
              }
            },
            "product" : "XG",
            "type" : "firewall",
            "vendor" : "Sophos",
            "egress" : {
              "interface" : {
                "name" : "Port2"
              }
            }
          },
          "@timestamp" : "2021-11-14T23:03:29.000Z",
          "related" : {
            "ip" : [
              "192.168.2.136",
              "35.82.255.30"
            ]
          },
          "sophos" : {
            "xg" : {
              "device_model" : "SFVH",
              "in_display_interface" : "Port1",
              "web_policy_id" : "12",
              "out_display_interface" : "Port2",
              "con_id" : "2425650304",
              "fw_rule_type" : "USER",
              "ips_policy_id" : "8",
              "src_trans_ip" : "192.168.1.2",
              "app_is_cloud" : "FALSE",
              "device_name" : "SFW",
              "log_type" : "Firewall",
              "ether_type" : "Unknown (0x0000)",
              "nat_rule_id" : "12",
              "device_serial_id" : "C01001BQC8TFFEB",
              "app_filter_policy_id" : "6",
              "timestamp" : "2021-11-14T17:03:29-0600",
              "severity" : "Information",
              "dst_country" : "USA",
              "log_component" : "Firewall Rule",
              "log_subtype" : "Allowed",
              "dst_zone_type" : "WAN",
              "hb_status" : "No Heartbeat",
              "message_id" : "00001",
              "dst_zone" : "WAN",
              "src_zone_type" : "LAN",
              "con_event" : "Start",
              "src_country" : "R1",
              "src_zone" : "LAN",
              "app_resolved_by" : "Signature",
              "qualifier" : "New",
              "log_version" : "1"
            }
          },
          "client" : {
            "port" : 53440,
            "mac" : "8A:0D:85:58:FF:FF",
            "ip" : "192.168.2.136"
          },
          "event" : {
            "severity" : "6",
            "ingested" : "2021-11-15T02:33:16.528600600Z",
            "original" : """device_name="SFW" timestamp="2021-11-14T17:03:29-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFEB" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="8A:0D:85:58:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.136" src_country="R1" dst_ip="35.82.255.30" dst_country="USA" protocol="TCP" src_port=53440 dst_port=443 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Start" con_id="2425650304" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2""",
            "code" : "010101600001",
            "kind" : "event",
            "module" : "sophos",
            "action" : "allowed",
            "category" : [
              "network"
            ],
            "dataset" : "sophos.xg",
            "outcome" : "success"
          }
        },
        "_ingest" : {
          "timestamp" : "2021-11-15T02:33:16.5286006Z"
        }
      }
    }
  ]
}
12

My apologies ; I think I have picked a wrong message ; Here is the right one ; Don't know how I missed it, they all looked same :frowning: ; This message has bytes_received and bytes_sent in it.

device_name="SFW" timestamp="2021-11-14T21:29:53-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFFF" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" duration=33 fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="10:BF:48:7D:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.121" src_country="R1" dst_ip="1.1.1.1" dst_country="AUS" protocol="ICMP" icmp_type=8 packets_sent=2  packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Stop" con_id="2171700160" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2"

Still no destination.bytes variable found in Kibana

- convert:
    field: sophos.xg.recv_bytes
    target_field: destination.bytes
    type: long

Here is the corresponding full record that I see in Kibana, in JSON Format ;

{
  "_index": "filebeat-7.15.1-2021.11.14-000001",
  "_type": "_doc",
  "_id": "5oekIX0ByPM67hqn4G-2",
  "_version": 1,
  "_score": 1,
  "_source": {
    "server": {
      "ip": "1.1.1.1",
      "mac": "00:50:56:9F:FF:FF"
    },
    "agent": {
      "hostname": "ubu16",
      "name": "ubu16",
      "id": "9dfb5d58-4d2b-4eb2-a1e0-2589d8f95c80",
      "type": "filebeat",
      "ephemeral_id": "f8b595b3-d15f-488d-9022-5b86b38396a1",
      "version": "7.15.1"
    },
    "log": {
      "level": "informational",
      "source": {
        "address": "192.168.2.1:56779"
      }
    },
    "destination": {
      "geo": {
        "continent_name": "Oceania",
        "country_iso_code": "AU",
        "country_name": "Australia",
        "location": {
          "lon": 143.2104,
          "lat": -33.494
        }
      },
      "as": {
        "number": 13335,
        "organization": {
          "name": "CLOUDFLARENET"
        }
      },
      "ip": "1.1.1.1",
      "mac": "00:50:56:9F:FF:FF"
    },
    "rule": {
      "id": "12"
    },
    "source": {
      "ip": "192.168.2.121",
      "mac": "10:BF:48:7D:FF:FF"
    },
    "fileset": {
      "name": "xg"
    },
    "tags": [
      "sophos-xg",
      "forwarded"
    ],
    "network": {
      "transport": "icmp"
    },
    "input": {
      "type": "udp"
    },
    "observer": {
      "ingress": {
        "interface": {
          "name": "Port1"
        }
      },
      "product": "XG",
      "vendor": "Sophos",
      "type": "firewall",
      "egress": {
        "interface": {
          "name": "Port2"
        }
      }
    },
    "@timestamp": "2021-11-14T21:29:53.000-06:00",
    "ecs": {
      "version": "1.11.0"
    },
    "related": {
      "hosts": [
        "ubu16"
      ],
      "ip": [
        "192.168.2.121",
        "1.1.1.1"
      ]
    },
    "sophos": {
      "xg": {
        "icmp_type": "8",
        "device_model": "SFVH",
        "in_display_interface": "Port1",
        "web_policy_id": "12",
        "out_display_interface": "Port2",
        "con_id": "2171700160",
        "bytes_received": "168",
        "fw_rule_type": "USER",
        "ips_policy_id": "8",
        "src_trans_ip": "192.168.1.2",
        "app_is_cloud": "FALSE",
        "device_name": "SFW",
        "log_type": "Firewall",
        "packets_sent": "2 ",
        "ether_type": "Unknown (0x0000)",
        "packets_received": "2",
        "nat_rule_id": "12",
        "device_serial_id": "C01001BQC8TFFFF",
        "app_filter_policy_id": "6",
        "timestamp": "2021-11-14T21:29:53-0600",
        "severity": "Information",
        "dst_country": "AUS",
        "log_component": "Firewall Rule",
        "log_subtype": "Allowed",
        "dst_zone_type": "WAN",
        "hb_status": "No Heartbeat",
        "message_id": "00001",
        "bytes_sent": "168",
        "dst_zone": "WAN",
        "src_zone_type": "LAN",
        "con_event": "Stop",
        "src_country": "R1",
        "src_zone": "LAN",
        "app_resolved_by": "Signature",
        "qualifier": "New",
        "log_version": "1"
      }
    },
    "service": {
      "type": "sophos"
    },
    "host": {
      "name": "ubu16"
    },
    "client": {
      "ip": "192.168.2.121",
      "mac": "10:BF:48:7D:FF:FF"
    },
    "event": {
      "severity": "6",
      "original": "device_name=\"SFW\" timestamp=\"2021-11-14T21:29:53-0600\" device_model=\"SFVH\" device_serial_id=\"C01001BQC8TFFEB\" log_id=\"010101600001\" log_type=\"Firewall\" log_component=\"Firewall Rule\" log_subtype=\"Allowed\" log_version=1 severity=\"Information\" duration=33 fw_rule_id=\"12\" nat_rule_id=\"12\" fw_rule_type=\"USER\" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type=\"Unknown (0x0000)\" in_interface=\"Port1\" out_interface=\"Port2\" src_mac=\"10:BF:48:7D:ED:22\" dst_mac=\"00:50:56:9F:39:33\" src_ip=\"192.168.2.121\" src_country=\"R1\" dst_ip=\"1.1.1.1\" dst_country=\"AUS\" protocol=\"ICMP\" icmp_type=8 packets_sent=2  packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip=\"192.168.1.2\" src_zone_type=\"LAN\" src_zone=\"LAN\" dst_zone_type=\"WAN\" dst_zone=\"WAN\" con_event=\"Stop\" con_id=\"2171700160\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"Port1\" out_display_interface=\"Port2\"",
      "code": "010101600001",
      "timezone": "-06:00",
      "kind": "event",
      "module": "sophos",
      "start": "2021-11-14T21:29:53.000-06:00",
      "duration": 33000000000,
      "ingested": "2021-11-15T03:29:53.331782895Z",
      "action": "allowed",
      "end": "2021-11-14T21:30:26.000-06:00",
      "category": [
        "network"
      ],
      "dataset": "sophos.xg",
      "outcome": "success"
    }
  },
  "fields": {
    "rule.id": [
      "12"
    ],
    "event.category": [
      "network"
    ],
    "sophos.xg.bytes_sent": [
      "168"
    ],
    "sophos.xg.app_resolved_by": [
      "Signature"
    ],
    "client.mac": [
      "10:BF:48:7D:FF:FF"
    ],
    "server.ip": [
      "1.1.1.1"
    ],
    "observer.egress.interface.name": [
      "Port2"
    ],
    "sophos.xg.src_zone_type": [
      "LAN"
    ],
    "observer.ingress.interface.name": [
      "Port1"
    ],
    "sophos.xg.con_id": [
      2171700160
    ],
    "destination.mac": [
      "00:50:56:9F:FF:FF"
    ],
    "service.type": [
      "sophos"
    ],
    "observer.vendor": [
      "Sophos"
    ],
    "sophos.xg.log_version": [
      "1"
    ],
    "sophos.xg.log_type": [
      "Firewall"
    ],
    "sophos.xg.src_trans_ip": [
      "192.168.1.2"
    ],
    "source.ip": [
      "192.168.2.121"
    ],
    "log.level": [
      "informational"
    ],
    "agent.name": [
      "ubu16"
    ],
    "host.name": [
      "ubu16"
    ],
    "event.kind": [
      "event"
    ],
    "sophos.xg.dst_zone_type": [
      "WAN"
    ],
    "sophos.xg.bytes_received": [
      "168"
    ],
    "event.outcome": [
      "success"
    ],
    "event.severity": [
      6
    ],
    "sophos.xg.src_zone": [
      "LAN"
    ],
    "event.original": [
      "device_name=\"SFW\" timestamp=\"2021-11-14T21:29:53-0600\" device_model=\"SFVH\" device_serial_id=\"C01001BQC8TFFFF\" log_id=\"010101600001\" log_type=\"Firewall\" log_component=\"Firewall Rule\" log_subtype=\"Allowed\" log_version=1 severity=\"Information\" duration=33 fw_rule_id=\"12\" nat_rule_id=\"12\" fw_rule_type=\"USER\" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type=\"Unknown (0x0000)\" in_interface=\"Port1\" out_interface=\"Port2\" src_mac=\"10:BF:48:7D:FF:FF\" dst_mac=\"00:50:56:9F:FF:FF\" src_ip=\"192.168.2.121\" src_country=\"R1\" dst_ip=\"1.1.1.1\" dst_country=\"AUS\" protocol=\"ICMP\" icmp_type=8 packets_sent=2  packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip=\"192.168.1.2\" src_zone_type=\"LAN\" src_zone=\"LAN\" dst_zone_type=\"WAN\" dst_zone=\"WAN\" con_event=\"Stop\" con_id=\"2171700160\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"Port1\" out_display_interface=\"Port2\""
    ],
    "sophos.xg.dst_zone": [
      "WAN"
    ],
    "sophos.xg.qualifier": [
      "New"
    ],
    "sophos.xg.device_name": [
      "SFW"
    ],
    "destination.geo.continent_name": [
      "Oceania"
    ],
    "fileset.name": [
      "xg"
    ],
    "sophos.xg.ether_type": [
      "Unknown (0x0000)"
    ],
    "input.type": [
      "udp"
    ],
    "client.ip": [
      "192.168.2.121"
    ],
    "agent.hostname": [
      "ubu16"
    ],
    "tags": [
      "sophos-xg",
      "forwarded"
    ],
    "event.code": [
      "010101600001"
    ],
    "agent.id": [
      "9dfb5d58-4d2b-4eb2-a1e0-2589d8f95c80"
    ],
    "sophos.xg.message_id": [
      "00001"
    ],
    "ecs.version": [
      "1.11.0"
    ],
    "observer.type": [
      "firewall"
    ],
    "log.source.address": [
      "192.168.2.1:56779"
    ],
    "sophos.xg.packets_received": [
      "2"
    ],
    "agent.version": [
      "7.15.1"
    ],
    "related.hosts": [
      "ubu16"
    ],
    "event.start": [
      "2021-11-15T03:29:53.000Z"
    ],
    "destination.geo.country_name": [
      "Australia"
    ],
    "server.mac": [
      "00:50:56:9F:39:33"
    ],
    "sophos.xg.timestamp": [
      "2021-11-15T03:29:53.000Z"
    ],
    "sophos.xg.device_serial_id": [
      "C01001BQC8TFFEB"
    ],
    "sophos.xg.src_country": [
      "R1"
    ],
    "sophos.xg.con_event": [
      "Stop"
    ],
    "sophos.xg.nat_rule_id": [
      "12"
    ],
    "event.end": [
      "2021-11-15T03:30:26.000Z"
    ],
    "sophos.xg.out_display_interface": [
      "Port2"
    ],
    "destination.geo.location": [
      {
        "coordinates": [
          143.2104,
          -33.494
        ],
        "type": "Point"
      }
    ],
    "sophos.xg.app_filter_policy_id": [
      "6"
    ],
    "agent.type": [
      "filebeat"
    ],
    "source.mac": [
      "10:BF:48:7D:ED:22"
    ],
    "event.module": [
      "sophos"
    ],
    "related.ip": [
      "192.168.2.121",
      "1.1.1.1"
    ],
    "sophos.xg.icmp_type": [
      "8"
    ],
    "sophos.xg.severity": [
      "Information"
    ],
    "observer.product": [
      "XG"
    ],
    "event.timezone": [
      "-06:00"
    ],
    "sophos.xg.packets_sent": [
      "2 "
    ],
    "sophos.xg.fw_rule_type": [
      "USER"
    ],
    "sophos.xg.web_policy_id": [
      "12"
    ],
    "sophos.xg.app_is_cloud": [
      "FALSE"
    ],
    "destination.as.number": [
      13335
    ],
    "sophos.xg.hb_status": [
      "No Heartbeat"
    ],
    "destination.as.organization.name.text": [
      "CLOUDFLARENET"
    ],
    "destination.ip": [
      "1.1.1.1"
    ],
    "network.transport": [
      "icmp"
    ],
    "event.duration": [
      33000000000
    ],
    "sophos.xg.dst_country": [
      "AUS"
    ],
    "event.action": [
      "allowed"
    ],
    "sophos.xg.ips_policy_id": [
      8
    ],
    "event.ingested": [
      "2021-11-15T03:29:53.331Z"
    ],
    "@timestamp": [
      "2021-11-15T03:29:53.000Z"
    ],
    "destination.geo.country_iso_code": [
      "AU"
    ],
    "sophos.xg.in_display_interface": [
      "Port1"
    ],
    "sophos.xg.log_subtype": [
      "Allowed"
    ],
    "agent.ephemeral_id": [
      "f8b595b3-d15f-488d-9022-5b86b38396a1"
    ],
    "sophos.xg.log_component": [
      "Firewall Rule"
    ],
    "event.dataset": [
      "sophos.xg"
    ],
    "sophos.xg.device_model": [
      "SFVH"
    ],
    "destination.as.organization.name": [
      "CLOUDFLARENET"
    ]
  }
}

I do not know how to do a POST, I will go through forum posts on how to complete a POST and and figure out how to complete one and let you know my result. Should that POST go to filebeat or to elastic-search?

13

You can do the POST in

Kibana -> Dev Tools

It goes to Elasticsearch

14

I found it!

So in the

filebeat-7.15.2-sophos-xg-firewall

The pipeline is looking for

sent_bytes

but your message has

bytes_sent

same for received etc.

I do not know which one is correct...

I think the easiest fix for now is fix those ingest YMLs module/sophos/xg/ingest/pipeline.yml files to line up with your messages then run this to reload the pipelines.

./filebeat setup --pipelines

I tried this for one field and it worked

The good thing is those values are copied into the Common ECS fields and then removed so the viz if there are any should be removed.

According to these docs your log format looks correct .. I think

https://docs.sophos.com/nsg/sophos-firewall/17.5/Help/en-us/webhelp/onlinehelp/nsg/sfos/concepts/LogFields.html

Looks like your message is correct.

If you could please write a bug against

By the way it looks like it changed because I saw an older version of the sophos log spec and it was the way the pipeline is so perhaps they changed it with the XG or something

15

Great!!

I changed firewall.yml file for one of the variables as follows and did
filebeat setup --pipelines ; restarted filebeat

- convert:
    field: sophos.xg.bytes_received
    target_field: destination.bytes
    type: long
    ignore_failure: true
    ignore_missing: true
    if: "ctx.sophos?.xg?.bytes_received != null"
- set:
    field: server.bytes
    value: '{{destination.bytes}}'
    if: "ctx.destination?.bytes != null"

Still no luck seeing the destination.bytes field in number format. Do not know what I am doing wrong. I will check more tomorrow. There multiple fields such as packets, bytes etc to be modified as well which I can do. I will file a bug report once I am convinced this should work.

For this input ;

POST /_ingest/pipeline/filebeat-7.15.1-sophos-xg-pipeline/_simulate
{
  "docs": [
    {
      "_index": "m-index",
      "_id": "kMpUTHoBr7SFhhL5-98P",
      "_source": {
        "@timestamp" : "2021-11-09T03:17:20.0516241Z",
        "message": """<44>device_name="SFW" timestamp="2021-11-14T21:29:53-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFFF" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" duration=33 fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="10:BF:48:7D:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.121" src_country="R1" dst_ip="1.1.1.1" dst_country="AUS" protocol="ICMP" icmp_type=8 packets_sent=2  packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Stop" con_id="2171700160" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2"""}
    }
  ]
}

I got this output

{
  "docs" : [
    {
      "doc" : {
        "_index" : "m-index",
        "_type" : "_doc",
        "_id" : "kMpUTHoBr7SFhhL5-98P",
        "_source" : {
          "server" : {
            "mac" : "00:50:56:9F:FF:FF",
            "ip" : "1.1.1.1"
          },
          "log" : {
            "level" : "informational"
          },
          "destination" : {
            "geo" : {
              "continent_name" : "Oceania",
              "country_name" : "Australia",
              "location" : {
                "lon" : 143.2104,
                "lat" : -33.494
              },
              "country_iso_code" : "AU"
            },
            "as" : {
              "number" : 13335,
              "organization" : {
                "name" : "CLOUDFLARENET"
              }
            },
            "mac" : "00:50:56:9F:FF:FF",
            "ip" : "1.1.1.1"
          },
          "rule" : {
            "id" : "12"
          },
          "source" : {
            "mac" : "10:BF:48:7D:FF:FF",
            "ip" : "192.168.2.121"
          },
          "network" : {
            "transport" : "icmp"
          },
          "observer" : {
            "ingress" : {
              "interface" : {
                "name" : "Port1"
              }
            },
            "product" : "XG",
            "type" : "firewall",
            "vendor" : "Sophos",
            "egress" : {
              "interface" : {
                "name" : "Port2"
              }
            }
          },
          "@timestamp" : "2021-11-15T03:29:53.000Z",
          "related" : {
            "ip" : [
              "192.168.2.121",
              "1.1.1.1"
            ]
          },
          "sophos" : {
            "xg" : {
              "icmp_type" : "8",
              "device_model" : "SFVH",
              "in_display_interface" : "Port1",
              "web_policy_id" : "12",
              "out_display_interface" : "Port2",
              "con_id" : "2171700160",
              "bytes_received" : "168",
              "fw_rule_type" : "USER",
              "ips_policy_id" : "8",
              "src_trans_ip" : "192.168.1.2",
              "app_is_cloud" : "FALSE",
              "device_name" : "SFW",
              "log_type" : "Firewall",
              "packets_sent" : "2 ",
              "ether_type" : "Unknown (0x0000)",
              "packets_received" : "2",
              "nat_rule_id" : "12",
              "device_serial_id" : "C01001BQC8TFFFF",
              "app_filter_policy_id" : "6",
              "timestamp" : "2021-11-14T21:29:53-0600",
              "severity" : "Information",
              "dst_country" : "AUS",
              "log_component" : "Firewall Rule",
              "log_subtype" : "Allowed",
              "dst_zone_type" : "WAN",
              "hb_status" : "No Heartbeat",
              "message_id" : "00001",
              "bytes_sent" : "168",
              "dst_zone" : "WAN",
              "src_zone_type" : "LAN",
              "con_event" : "Stop",
              "src_country" : "R1",
              "src_zone" : "LAN",
              "app_resolved_by" : "Signature",
              "qualifier" : "New",
              "log_version" : "1"
            }
          },
          "client" : {
            "mac" : "10:BF:48:7D:FF:FF",
            "ip" : "192.168.2.121"
          },
          "event" : {
            "severity" : "6",
            "original" : """device_name="SFW" timestamp="2021-11-14T21:29:53-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFFF" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" duration=33 fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="10:BF:48:7D:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.121" src_country="R1" dst_ip="1.1.1.1" dst_country="AUS" protocol="ICMP" icmp_type=8 packets_sent=2  packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Stop" con_id="2171700160" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2""",
            "code" : "010101600001",
            "kind" : "event",
            "module" : "sophos",
            "start" : "2021-11-15T03:29:53.000Z",
            "duration" : 33000000000,
            "ingested" : "2021-11-15T06:05:44.098946522Z",
            "action" : "allowed",
            "end" : "2021-11-15T03:30:26.000Z",
            "category" : [
              "network"
            ],
            "dataset" : "sophos.xg",
            "outcome" : "success"
          }
        },
        "_ingest" : {
          "timestamp" : "2021-11-15T06:05:44.098946522Z"
        }
      }
    }
  ]
}
16

I just did what you specified and it worked for me...

- convert:
    field: sophos.xg.bytes_sent
    target_field: source.bytes
    type: long
    ignore_failure: true
    ignore_missing: true
    if: "ctx.sophos?.xg?.bytes_sent != null"

- convert:
    field: sophos.xg.bytes_received
    target_field: destination.bytes
    type: long
    ignore_failure: true
    ignore_missing: true
    if: "ctx.sophos?.xg?.bytes_received != null"

Ran this...

POST /_ingest/pipeline/filebeat-7.15.2-sophos-xg-pipeline/_simulate
{
  "docs": [
    {
      "_index": "m-index",
      "_id": "kMpUTHoBr7SFhhL5-98P",
      "_source": {
        "@timestamp" : "2021-11-09T03:17:20.0516241Z",
        "message": """<44>device_name="SFW" timestamp="2021-11-14T21:29:53-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFFF" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" duration=33 fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="10:BF:48:7D:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.121" src_country="R1" dst_ip="1.1.1.1" dst_country="AUS" protocol="ICMP" icmp_type=8 packets_sent=2  packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Stop" con_id="2171700160" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2"""}
    }
  ]
}

Got This... Note the Destination bytes etc... did you save you .ymls, you can delete the existing pipeline to make sure...

{
  "docs" : [
    {
      "doc" : {
        "_index" : "m-index",
        "_type" : "_doc",
        "_id" : "kMpUTHoBr7SFhhL5-98P",
        "_source" : {
          "server" : {
            "bytes" : 168,
            "mac" : "00:50:56:9F:FF:FF",
            "ip" : "1.1.1.1"
          },
          "log" : {
            "level" : "informational"
          },
          "destination" : {
            "geo" : {
              "continent_name" : "Oceania",
              "country_name" : "Australia",
              "location" : {
                "lon" : 143.2104,
                "lat" : -33.494
              },
              "country_iso_code" : "AU"
            },
            "as" : {
              "number" : 13335,
              "organization" : {
                "name" : "CLOUDFLARENET"
              }
            },
            "bytes" : 168,
            "mac" : "00:50:56:9F:FF:FF",
            "ip" : "1.1.1.1"
          },
          "rule" : {
            "id" : "12"
          },
          "source" : {
            "bytes" : 168,
            "mac" : "10:BF:48:7D:FF:FF",
            "ip" : "192.168.2.121"
          },
          "network" : {
            "bytes" : 336,
            "transport" : "icmp"
          },
          "observer" : {
            "ingress" : {
              "interface" : {
                "name" : "Port1"
              }
            },
            "product" : "XG",
            "type" : "firewall",
            "vendor" : "Sophos",
            "egress" : {
              "interface" : {
                "name" : "Port2"
              }
            }
          },
          "@timestamp" : "2021-11-15T03:29:53.000Z",
          "related" : {
            "ip" : [
              "192.168.2.121",
              "1.1.1.1"
            ]
          },
          "sophos" : {
            "xg" : {
              "icmp_type" : "8",
              "device_model" : "SFVH",
              "in_display_interface" : "Port1",
              "web_policy_id" : "12",
              "out_display_interface" : "Port2",
              "con_id" : "2171700160",
              "fw_rule_type" : "USER",
              "ips_policy_id" : "8",
              "src_trans_ip" : "192.168.1.2",
              "app_is_cloud" : "FALSE",
              "device_name" : "SFW",
              "log_type" : "Firewall",
              "packets_sent" : "2 ",
              "ether_type" : "Unknown (0x0000)",
              "packets_received" : "2",
              "nat_rule_id" : "12",
              "device_serial_id" : "C01001BQC8TFFFF",
              "app_filter_policy_id" : "6",
              "timestamp" : "2021-11-14T21:29:53-0600",
              "severity" : "Information",
              "dst_country" : "AUS",
              "log_component" : "Firewall Rule",
              "log_subtype" : "Allowed",
              "dst_zone_type" : "WAN",
              "hb_status" : "No Heartbeat",
              "message_id" : "00001",
              "dst_zone" : "WAN",
              "src_zone_type" : "LAN",
              "con_event" : "Stop",
              "src_country" : "R1",
              "src_zone" : "LAN",
              "app_resolved_by" : "Signature",
              "qualifier" : "New",
              "log_version" : "1"
            }
          },
          "client" : {
            "bytes" : 168,
            "mac" : "10:BF:48:7D:FF:FF",
            "ip" : "192.168.2.121"
          },
          "event" : {
            "severity" : "6",
            "original" : """device_name="SFW" timestamp="2021-11-14T21:29:53-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFFF" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" duration=33 fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="10:BF:48:7D:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.121" src_country="R1" dst_ip="1.1.1.1" dst_country="AUS" protocol="ICMP" icmp_type=8 packets_sent=2  packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Stop" con_id="2171700160" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2""",
            "code" : "010101600001",
            "kind" : "event",
            "module" : "sophos",
            "start" : "2021-11-15T03:29:53.000Z",
            "duration" : 33000000000,
            "ingested" : "2021-11-15T06:47:08.476196100Z",
            "action" : "allowed",
            "end" : "2021-11-15T03:30:26.000Z",
            "category" : [
              "network"
            ],
            "dataset" : "sophos.xg",
            "outcome" : "success"
          }
        },
        "_ingest" : {
          "timestamp" : "2021-11-15T06:47:08.4761961Z"
        }
      }
    }
  ]
}
17

Thanks for flagging this. I've created an issue to address the bug in our pipeline - Sophos XG - sent_bytes bug · Issue #2158 · elastic/integrations · GitHub

1 Like
18

Thank you for creating a bug report. However this is not only limited to sent_bytes. All following need addressed and they are all numeric fields as well. ;

sent_bytes --change-to --> bytes_sent
bytes_received --change-to --> bytes_received
sent_pkts --change-to --> packets_sent
recv_pkts --change-to --> packets_received

I also updated your bug report with this information.

1 Like
19

I had to do

./filebeat setup --pipelines sophos

and that worked.

1 Like
20

Here is another issue.

Following input should be processed by cfilter.yml pipeline - file as This has log_type="Content Filtering" ; However there is no line in cfilter.yml to convert bytes_sent, bytes_received to numberic values. I think they were missing from the start. If @stephenb and @jamie.hynds agree you should add this to the bug report as well. These fields are important as they are required for visualizations.

Thank you for all the help.

device_name="SFW" timestamp="2021-11-15T17:33:25-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFFF" log_id=050901616001 log_type="Content Filtering" log_component="HTTP" log_subtype="Allowed" log_version=1 severity="Information" fw_rule_id="12" web_policy_id=12 http_category="Video hosting" http_category_type="Unproductive" url="https://www.youtube.com" src_ip="192.168.2.162" dst_ip="142.250.115.136" protocol="TCP" src_port=55644 dst_port=443 bytes_sent=8347 bytes_received=20257 domain="www.youtube.com" http_status="0" con_id=2617086912 app_is_cloud="FALSE" used_quota="0" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" src_country="R1" dst_country="USA"
1 Like