# ES 7.15 FileBeats Sophos XG module not separating data into variables **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 8, 2021, 12:30am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577 "2021-11-08T00:30:58Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Samy\_Weee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samy_weee/32/81251_2.png) [@Samy\_Weee](https://discuss.elastic.co/u/Samy_Weee)\ **Post date:** [November 8, 2021, 12:30am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/1 "2021-11-08T00:30:58Z")
Following This link to configure sophos Module. [www.elastic.co/.../filebeat-module-sophos.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-sophos.html) I followed the documentation at [www.elastic.co/.../filebeat-module-sophos.html and](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-sophos.html%C2%A0and) was able to get the data into ES, issue is I cannot see the fields (varibale) in the message. All data is inside a field called message. I am using ES 7.15.1 / Filebeat 7.15.1 / Ubuntu 18.04 / Logstash 7.15.1 In high level I am using Filebeat Sophos-xg module, logstash and ES. I did enable sophos module, uploaded the template with commands "filebeat modules enable sophos" and "filebeat setup -e -E output.logstash.enabled=false -E output.Elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601" and I am getting the data, without data being split into fields/variables. I am not an ELK stack expert. Not sure if the issue is with filebeat module / logstash or somewhere else. Any troubleshooting steps let me know please? **Sample Message - I want what is inside message in variables** "message": "\<30\>device\_name="SFW" timestamp="2021-11-07T18:14:59-0600" device\_model="SFVH" device\_serial\_id="C01001BQC8XXXXX" log\_id="010101600001" log\_type="Firewall" log\_component="Firewall Rule" log\_subtype="Allowed" log\_version=1 severity="Information" duration=38 fw\_rule\_id="12" nat\_rule\_id="12" fw\_rule\_type="USER" web\_policy\_id=12 ips\_policy\_id=8 app\_filter\_policy\_id=6 ether\_type="Unknown (0x0000)" in\_interface="Port1" out\_interface="Port2" src\_mac="10:BF:48:7D:FF:FF" dst\_mac="00:50:56:9F:FF:FF" src\_ip="192.168.2.10" src\_country="R1" dst\_ip="1.1.1.1" dst\_country="AUS" protocol="ICMP" icmp\_type=8 packets\_sent=2 packets\_received=2 bytes\_sent=168 bytes\_received=168 src\_trans\_ip="192.168.1.2" src\_zone\_type="LAN" src\_zone="LAN" dst\_zone\_type="WAN" dst\_zone="WAN" con\_event="Stop" con\_id="2907267392" hb\_status="No Heartbeat" app\_resolved\_by="Signature" app\_is\_cloud="FALSE" qualifier="New" in\_display\_interface="Port1" out\_display\_interface="Port2"", ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 8, 2021, 1:32am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/2 "2021-11-08T01:32:08Z")
Hi @Samy_Weee Welcome to the community First I would usually try to send direct from Filebeat -\> Elasticsearch and get that working? Once you get that working .. .then I would move on to this Architecture Filebeat -\> Logstash -\> Elasticsearch And the question is... Do you actually need logstash? (it is not required) If you do ... once you have Filebeat -\> Elasticsearch working go into your filebeat.yml and then direct filebeat output to logstash output. Then your logstash.conf should look something like this... this is a "Passthrough" configuration what is most likely happening is that you are missing this line `pipeline => "%{[@metadata][pipeline]}" ` which tells Elasticsearch which pipeline (AKA Parser) to use. When you go direct from Filebeat -\> Elasticsearch that info is automatically passed along .. when you put logstash in the middle it is not. Start Logstash first, and then filebeat... ```auto ################################################ # beats->logstash->es default config. ################################################ input { beats { port => 5044 } } output { if [@metadata][pipeline] { elasticsearch { hosts => "http://localhost:9200" manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}" pipeline => "%{[@metadata][pipeline]}" user => "elastic" password => "secret" } } else { elasticsearch { hosts => "http://localhost:9200" manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}" user => "elastic" password => "secret" } } } ``` ---
**Author:** ![Samy\_Weee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samy_weee/32/81251_2.png) [@Samy\_Weee](https://discuss.elastic.co/u/Samy_Weee)\ **Post date:** [November 8, 2021, 3:37am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/3 "2021-11-08T03:37:34Z")
@stephenb As explained.. It worked fine without logstash. Then it also worked with your modified logstash passthrough config. I wish I knew I do not need logstash, to get this working. Going forward, I will remove the logstash and only use filebeat as it seems more efficient that way for my purpose. Just another question out of this: Is it possible to run 2 instances of filebeat with 2 configuration files, so that one instance can directly forward to ES, while second filebeat instance forward to LS? Thank you very much for the help. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 8, 2021, 4:44am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/4 "2021-11-08T04:44:07Z")
> [@Samy\_Weee](#): > > As explained.. It worked fine without logstash. Apologies, I did not see anywhere in your initial post where you indicated it all worked fine with a direct filebeat-\> elasticsearch integration. > [@Samy\_Weee](#): > > I wish I knew I do not need logstash, to get this working. Yes, the docs are not always easy to navigate, The [quick start guides](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html) are meant to help. > [@Samy\_Weee](#): > > Is it possible to run 2 instances of filebeat with 2 configuration files, so that one instance can directly forward to ES, while second filebeat instance forward to LS? Possible...yes... You would need basically need 2 separate installations if filebeat because it keeps track of files have been read etc.. Not clear why you would want to do that generally when folks want to send them more than one destination that is exactly when they use logstash... filebeat -\> logstash to more than one output ---
**Author:** ![Samy\_Weee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samy_weee/32/81251_2.png) [@Samy\_Weee](https://discuss.elastic.co/u/Samy_Weee)\ **Post date:** [November 8, 2021, 1:37pm UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/5 "2021-11-08T13:37:39Z")
> [@stephenb](#): > > > [@Samy\_Weee](#): > > > > As explained.. It worked fine without logstash. > > Apologies, I did not see anywhere in your initial post where you indicated it all worked fine with a direct filebeat-\> elasticsearch integration. @stephenb What I meant was I never tried it that way until you asked me to. It all started working after your guideline. I am learning. Thank you. ---
**Author:** ![Samy\_Weee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samy_weee/32/81251_2.png) [@Samy\_Weee](https://discuss.elastic.co/u/Samy_Weee)\ **Post date:** [November 14, 2021, 7:33am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/6 "2021-11-14T07:33:16Z")
I am able to feed data from sophos module using file beat to Elasticsearch. I have a different issue. There are fields which are marked as string in ES, which should have been a numeric . Here is an example. ; Variable: "sophos.xg.bytes\_received" is coming in as a string variable. (My Flow : Sophos --\> filebeat-sophos-module --\> Elasticsearch) I went into modules directory (filebeat/module/sophos/xg/ingest/sophos) to check the definition for this variable and it seems to be defined correctly as a type: long , where has this gone wrong. Why this field is not coming in as a long variable how to trouble-shoot. Most variables are strings, except few numeric. ```auto - convert: field: sophos.xg.recv_bytes target_field: destination.bytes type: long ignore_failure: true ignore_missing: true if: "ctx.sophos?.xg?.recv_bytes != null" ``` ![sophos_es](https://us1.discourse-cdn.com/elastic/original/3X/0/d/0d68873132468c697943a4f1bb1ae96437afad68.png) See the screenshot above. I also do not see any field with name "destination.bytes" in Elasticsearch. But I do see sophos.xg.recv\_bytes as a "t" - string field. Thank you if someone can help please. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 14, 2021, 4:19pm UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/7 "2021-11-14T16:19:16Z")
Did you run `filebeat setup` _ **before** _ you initially started filebeat? If not that could explain this behavior meaning the template was not loaded / applied before you started to send data and thus the mapping may not be correct. ---
**Author:** ![Samy\_Weee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samy_weee/32/81251_2.png) [@Samy\_Weee](https://discuss.elastic.co/u/Samy_Weee)\ **Post date:** [November 14, 2021, 10:41pm UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/8 "2021-11-14T22:41:36Z")
> [@stephenb](#): > > Did you run `filebeat setup` _ **before** _ you initially started filebeat? If not that could explain this behavior meaning the template was not loaded / applied before you started to send data and thus the mapping may not be correct. Yes, 'filebeat setup -e' was run before I started filebeat with systemctl start filebeat (I use Ubuntu). Is there a step to check if relevant template is in place? What else I can check? I ran the 'filebeat setup -e' again and I see following output towards the end of the run . Still my variables are of wrong type. Output seems to suggest it is loading the pipelines from the modules directory related to my module of concertn (sophos-xg) ```auto .... .... 2021-11-14T16:37:20.662-0600 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200 2021-11-14T16:37:20.663-0600 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.15.1 2021-11-14T16:37:20.664-0600 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: http://localhost:9200 2021-11-14T16:37:20.665-0600 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.15.1 2021-11-14T16:37:20.696-0600 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-sophos-xg-pipeline"} 2021-11-14T16:37:20.709-0600 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-sophos-xg-systemhealth"} 2021-11-14T16:37:20.727-0600 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-sophos-xg-firewall"} 2021-11-14T16:37:20.745-0600 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-sophos-xg-idp"} 2021-11-14T16:37:20.760-0600 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-sophos-xg-atp"} 2021-11-14T16:37:20.774-0600 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-sophos-xg-antivirus"} 2021-11-14T16:37:20.787-0600 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-sophos-xg-sandstorm"} 2021-11-14T16:37:20.800-0600 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-sophos-xg-cfilter"} 2021-11-14T16:37:20.814-0600 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-sophos-xg-event"} 2021-11-14T16:37:20.827-0600 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-sophos-xg-waf"} 2021-11-14T16:37:20.839-0600 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-sophos-xg-antispam"} 2021-11-14T16:37:20.851-0600 INFO [modules] fileset/pipelines.go:133 Elasticsearch pipeline loaded. {"pipeline": "filebeat-7.15.1-sophos-xg-wifi"} 2021-11-14T16:37:20.851-0600 INFO cfgfile/reload.go:262 Loading of config files completed. 2021-11-14T16:37:20.851-0600 INFO [load] cfgfile/list.go:129 Stopping 1 runners ... Loaded Ingest pipelines ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 15, 2021, 12:18am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/9 "2021-11-15T00:18:11Z")
> [@Samy\_Weee](#): > > `destination.bytes` That field is not set in every message are you sure you are looking at one of the messages that that is parsed for? Also have there been and changes to the Sophos config? you may meed to dig in and take a look at which pipeline is being called... ```auto antispam.yml: target_field: destination.bytes antivirus.yml: target_field: destination.bytes event.yml: target_field: destination.bytes firewall.yml: target_field: destination.bytes waf.yml: target_field: destination.bytes ``` ---
**Author:** ![Samy\_Weee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samy_weee/32/81251_2.png) [@Samy\_Weee](https://discuss.elastic.co/u/Samy_Weee)\ **Post date:** [November 15, 2021, 12:41am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/10 "2021-11-15T00:41:01Z")
> [@stephenb](#): > > That field is not set in every message are you sure you are looking at one of the messages that that is parsed for? Here is the input msg to the filebeat, this should get processed by firewall.yml module file ; I do not understand how it gets directed to firewall.yml file to process, but I do not see destination.bytes field in the received message in Kibana. I see a sophos.xg.recv\_bytes field. I think the conversion step do not happen in the filebeat module or filebeat module don't know which .yml file to apply for the input, Where should I look? Thank you. ```auto device_name="SFW" timestamp="2021-11-14T17:03:29-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFEB" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="8A:0D:85:58:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.136" src_country="R1" dst_ip="35.82.255.30" dst_country="USA" protocol="TCP" src_port=53440 dst_port=443 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Start" con_id="2425650304" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2" ``` > [@stephenb](#): > > Also have there been and changes to the Sophos config? No changes were made to sophos-xg firewall logging. I am using default logging feature available. > [@stephenb](#): > > you may meed to dig in and take a look at which pipeline is being called... This may be where it is going wrong. Any post or guideline if you can point me to I can do some trouble shooting. Appreciate your help. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 15, 2021, 2:38am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/11 "2021-11-15T02:38:22Z")
> [@Samy\_Weee](#): > > `filebeat-7.15.1-sophos-xg-pipeline` The logic is in the `filebeat-7.15.1-sophos-xg-pipeline` Pipeline you can look at it in detail in Kibana -\> Stack Management -\> Ingest Pipelines So I took the sample message you provided and put in a `%{SYSLOG5424PRI}` the `<44>` and it parsed it fine... Your example _ **does not have the bytes sent and received bytes** _ in the source so I can not see what is / is not happening... you will need to get one of the right type and try it. and try this simulate... ```auto POST /_ingest/pipeline/filebeat-7.15.2-sophos-xg-pipeline/_simulate { "docs": [ { "_index": "m-index", "_id": "kMpUTHoBr7SFhhL5-98P", "_source": { "@timestamp" : "2021-11-09T03:17:20.0516241Z", "message": """<44>device_name="SFW" timestamp="2021-11-14T17:03:29-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFEB" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="8A:0D:85:58:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.136" src_country="R1" dst_ip="35.82.255.30" dst_country="USA" protocol="TCP" src_port=53440 dst_port=443 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Start" con_id="2425650304" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2""" } } ] } ``` run this and you will see the results... ```auto { "docs" : [ { "doc" : { "_index" : "m-index", "_type" : "_doc", "_id" : "kMpUTHoBr7SFhhL5-98P", "_source" : { "server" : { "port" : 443, "mac" : "00:50:56:9F:FF:FF", "ip" : "35.82.255.30" }, "log" : { "level" : "informational" }, "destination" : { "geo" : { "continent_name" : "North America", "region_iso_code" : "US-OR", "city_name" : "Boardman", "country_iso_code" : "US", "country_name" : "United States", "region_name" : "Oregon", "location" : { "lon" : -119.7257, "lat" : 45.8234 } }, "as" : { "number" : 16509, "organization" : { "name" : "AMAZON-02" } }, "port" : 443, "mac" : "00:50:56:9F:FF:FF", "ip" : "35.82.255.30" }, "rule" : { "id" : "12" }, "source" : { "port" : 53440, "mac" : "8A:0D:85:58:FF:FF", "ip" : "192.168.2.136" }, "network" : { "transport" : "tcp" }, "observer" : { "ingress" : { "interface" : { "name" : "Port1" } }, "product" : "XG", "type" : "firewall", "vendor" : "Sophos", "egress" : { "interface" : { "name" : "Port2" } } }, "@timestamp" : "2021-11-14T23:03:29.000Z", "related" : { "ip" : [ "192.168.2.136", "35.82.255.30" ] }, "sophos" : { "xg" : { "device_model" : "SFVH", "in_display_interface" : "Port1", "web_policy_id" : "12", "out_display_interface" : "Port2", "con_id" : "2425650304", "fw_rule_type" : "USER", "ips_policy_id" : "8", "src_trans_ip" : "192.168.1.2", "app_is_cloud" : "FALSE", "device_name" : "SFW", "log_type" : "Firewall", "ether_type" : "Unknown (0x0000)", "nat_rule_id" : "12", "device_serial_id" : "C01001BQC8TFFEB", "app_filter_policy_id" : "6", "timestamp" : "2021-11-14T17:03:29-0600", "severity" : "Information", "dst_country" : "USA", "log_component" : "Firewall Rule", "log_subtype" : "Allowed", "dst_zone_type" : "WAN", "hb_status" : "No Heartbeat", "message_id" : "00001", "dst_zone" : "WAN", "src_zone_type" : "LAN", "con_event" : "Start", "src_country" : "R1", "src_zone" : "LAN", "app_resolved_by" : "Signature", "qualifier" : "New", "log_version" : "1" } }, "client" : { "port" : 53440, "mac" : "8A:0D:85:58:FF:FF", "ip" : "192.168.2.136" }, "event" : { "severity" : "6", "ingested" : "2021-11-15T02:33:16.528600600Z", "original" : """device_name="SFW" timestamp="2021-11-14T17:03:29-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFEB" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="8A:0D:85:58:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.136" src_country="R1" dst_ip="35.82.255.30" dst_country="USA" protocol="TCP" src_port=53440 dst_port=443 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Start" con_id="2425650304" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2""", "code" : "010101600001", "kind" : "event", "module" : "sophos", "action" : "allowed", "category" : [ "network" ], "dataset" : "sophos.xg", "outcome" : "success" } }, "_ingest" : { "timestamp" : "2021-11-15T02:33:16.5286006Z" } } } ] } ``` ---
**Author:** ![Samy\_Weee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samy_weee/32/81251_2.png) [@Samy\_Weee](https://discuss.elastic.co/u/Samy_Weee)\ **Post date:** [November 15, 2021, 3:52am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/12 "2021-11-15T03:52:22Z")
> [@stephenb](#): > > Your example _ **does not have the bytes sent and received bytes** _ in the source My apologies ; I think I have picked a wrong message ; Here is the right one ; Don't know how I missed it, they all looked same ☹ ; This message has bytes\_received and bytes\_sent in it. ```auto device_name="SFW" timestamp="2021-11-14T21:29:53-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFFF" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" duration=33 fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="10:BF:48:7D:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.121" src_country="R1" dst_ip="1.1.1.1" dst_country="AUS" protocol="ICMP" icmp_type=8 packets_sent=2 packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Stop" con_id="2171700160" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2" ``` Still no destination.bytes variable found in Kibana ```auto - convert: field: sophos.xg.recv_bytes target_field: destination.bytes type: long ``` Here is the corresponding full record that I see in Kibana, in JSON Format ; ```auto { "_index": "filebeat-7.15.1-2021.11.14-000001", "_type": "_doc", "_id": "5oekIX0ByPM67hqn4G-2", "_version": 1, "_score": 1, "_source": { "server": { "ip": "1.1.1.1", "mac": "00:50:56:9F:FF:FF" }, "agent": { "hostname": "ubu16", "name": "ubu16", "id": "9dfb5d58-4d2b-4eb2-a1e0-2589d8f95c80", "type": "filebeat", "ephemeral_id": "f8b595b3-d15f-488d-9022-5b86b38396a1", "version": "7.15.1" }, "log": { "level": "informational", "source": { "address": "192.168.2.1:56779" } }, "destination": { "geo": { "continent_name": "Oceania", "country_iso_code": "AU", "country_name": "Australia", "location": { "lon": 143.2104, "lat": -33.494 } }, "as": { "number": 13335, "organization": { "name": "CLOUDFLARENET" } }, "ip": "1.1.1.1", "mac": "00:50:56:9F:FF:FF" }, "rule": { "id": "12" }, "source": { "ip": "192.168.2.121", "mac": "10:BF:48:7D:FF:FF" }, "fileset": { "name": "xg" }, "tags": [ "sophos-xg", "forwarded" ], "network": { "transport": "icmp" }, "input": { "type": "udp" }, "observer": { "ingress": { "interface": { "name": "Port1" } }, "product": "XG", "vendor": "Sophos", "type": "firewall", "egress": { "interface": { "name": "Port2" } } }, "@timestamp": "2021-11-14T21:29:53.000-06:00", "ecs": { "version": "1.11.0" }, "related": { "hosts": [ "ubu16" ], "ip": [ "192.168.2.121", "1.1.1.1" ] }, "sophos": { "xg": { "icmp_type": "8", "device_model": "SFVH", "in_display_interface": "Port1", "web_policy_id": "12", "out_display_interface": "Port2", "con_id": "2171700160", "bytes_received": "168", "fw_rule_type": "USER", "ips_policy_id": "8", "src_trans_ip": "192.168.1.2", "app_is_cloud": "FALSE", "device_name": "SFW", "log_type": "Firewall", "packets_sent": "2 ", "ether_type": "Unknown (0x0000)", "packets_received": "2", "nat_rule_id": "12", "device_serial_id": "C01001BQC8TFFFF", "app_filter_policy_id": "6", "timestamp": "2021-11-14T21:29:53-0600", "severity": "Information", "dst_country": "AUS", "log_component": "Firewall Rule", "log_subtype": "Allowed", "dst_zone_type": "WAN", "hb_status": "No Heartbeat", "message_id": "00001", "bytes_sent": "168", "dst_zone": "WAN", "src_zone_type": "LAN", "con_event": "Stop", "src_country": "R1", "src_zone": "LAN", "app_resolved_by": "Signature", "qualifier": "New", "log_version": "1" } }, "service": { "type": "sophos" }, "host": { "name": "ubu16" }, "client": { "ip": "192.168.2.121", "mac": "10:BF:48:7D:FF:FF" }, "event": { "severity": "6", "original": "device_name=\"SFW\" timestamp=\"2021-11-14T21:29:53-0600\" device_model=\"SFVH\" device_serial_id=\"C01001BQC8TFFEB\" log_id=\"010101600001\" log_type=\"Firewall\" log_component=\"Firewall Rule\" log_subtype=\"Allowed\" log_version=1 severity=\"Information\" duration=33 fw_rule_id=\"12\" nat_rule_id=\"12\" fw_rule_type=\"USER\" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type=\"Unknown (0x0000)\" in_interface=\"Port1\" out_interface=\"Port2\" src_mac=\"10:BF:48:7D:ED:22\" dst_mac=\"00:50:56:9F:39:33\" src_ip=\"192.168.2.121\" src_country=\"R1\" dst_ip=\"1.1.1.1\" dst_country=\"AUS\" protocol=\"ICMP\" icmp_type=8 packets_sent=2 packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip=\"192.168.1.2\" src_zone_type=\"LAN\" src_zone=\"LAN\" dst_zone_type=\"WAN\" dst_zone=\"WAN\" con_event=\"Stop\" con_id=\"2171700160\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"Port1\" out_display_interface=\"Port2\"", "code": "010101600001", "timezone": "-06:00", "kind": "event", "module": "sophos", "start": "2021-11-14T21:29:53.000-06:00", "duration": 33000000000, "ingested": "2021-11-15T03:29:53.331782895Z", "action": "allowed", "end": "2021-11-14T21:30:26.000-06:00", "category": [ "network" ], "dataset": "sophos.xg", "outcome": "success" } }, "fields": { "rule.id": [ "12" ], "event.category": [ "network" ], "sophos.xg.bytes_sent": [ "168" ], "sophos.xg.app_resolved_by": [ "Signature" ], "client.mac": [ "10:BF:48:7D:FF:FF" ], "server.ip": [ "1.1.1.1" ], "observer.egress.interface.name": [ "Port2" ], "sophos.xg.src_zone_type": [ "LAN" ], "observer.ingress.interface.name": [ "Port1" ], "sophos.xg.con_id": [ 2171700160 ], "destination.mac": [ "00:50:56:9F:FF:FF" ], "service.type": [ "sophos" ], "observer.vendor": [ "Sophos" ], "sophos.xg.log_version": [ "1" ], "sophos.xg.log_type": [ "Firewall" ], "sophos.xg.src_trans_ip": [ "192.168.1.2" ], "source.ip": [ "192.168.2.121" ], "log.level": [ "informational" ], "agent.name": [ "ubu16" ], "host.name": [ "ubu16" ], "event.kind": [ "event" ], "sophos.xg.dst_zone_type": [ "WAN" ], "sophos.xg.bytes_received": [ "168" ], "event.outcome": [ "success" ], "event.severity": [ 6 ], "sophos.xg.src_zone": [ "LAN" ], "event.original": [ "device_name=\"SFW\" timestamp=\"2021-11-14T21:29:53-0600\" device_model=\"SFVH\" device_serial_id=\"C01001BQC8TFFFF\" log_id=\"010101600001\" log_type=\"Firewall\" log_component=\"Firewall Rule\" log_subtype=\"Allowed\" log_version=1 severity=\"Information\" duration=33 fw_rule_id=\"12\" nat_rule_id=\"12\" fw_rule_type=\"USER\" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type=\"Unknown (0x0000)\" in_interface=\"Port1\" out_interface=\"Port2\" src_mac=\"10:BF:48:7D:FF:FF\" dst_mac=\"00:50:56:9F:FF:FF\" src_ip=\"192.168.2.121\" src_country=\"R1\" dst_ip=\"1.1.1.1\" dst_country=\"AUS\" protocol=\"ICMP\" icmp_type=8 packets_sent=2 packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip=\"192.168.1.2\" src_zone_type=\"LAN\" src_zone=\"LAN\" dst_zone_type=\"WAN\" dst_zone=\"WAN\" con_event=\"Stop\" con_id=\"2171700160\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"Port1\" out_display_interface=\"Port2\"" ], "sophos.xg.dst_zone": [ "WAN" ], "sophos.xg.qualifier": [ "New" ], "sophos.xg.device_name": [ "SFW" ], "destination.geo.continent_name": [ "Oceania" ], "fileset.name": [ "xg" ], "sophos.xg.ether_type": [ "Unknown (0x0000)" ], "input.type": [ "udp" ], "client.ip": [ "192.168.2.121" ], "agent.hostname": [ "ubu16" ], "tags": [ "sophos-xg", "forwarded" ], "event.code": [ "010101600001" ], "agent.id": [ "9dfb5d58-4d2b-4eb2-a1e0-2589d8f95c80" ], "sophos.xg.message_id": [ "00001" ], "ecs.version": [ "1.11.0" ], "observer.type": [ "firewall" ], "log.source.address": [ "192.168.2.1:56779" ], "sophos.xg.packets_received": [ "2" ], "agent.version": [ "7.15.1" ], "related.hosts": [ "ubu16" ], "event.start": [ "2021-11-15T03:29:53.000Z" ], "destination.geo.country_name": [ "Australia" ], "server.mac": [ "00:50:56:9F:39:33" ], "sophos.xg.timestamp": [ "2021-11-15T03:29:53.000Z" ], "sophos.xg.device_serial_id": [ "C01001BQC8TFFEB" ], "sophos.xg.src_country": [ "R1" ], "sophos.xg.con_event": [ "Stop" ], "sophos.xg.nat_rule_id": [ "12" ], "event.end": [ "2021-11-15T03:30:26.000Z" ], "sophos.xg.out_display_interface": [ "Port2" ], "destination.geo.location": [ { "coordinates": [ 143.2104, -33.494 ], "type": "Point" } ], "sophos.xg.app_filter_policy_id": [ "6" ], "agent.type": [ "filebeat" ], "source.mac": [ "10:BF:48:7D:ED:22" ], "event.module": [ "sophos" ], "related.ip": [ "192.168.2.121", "1.1.1.1" ], "sophos.xg.icmp_type": [ "8" ], "sophos.xg.severity": [ "Information" ], "observer.product": [ "XG" ], "event.timezone": [ "-06:00" ], "sophos.xg.packets_sent": [ "2 " ], "sophos.xg.fw_rule_type": [ "USER" ], "sophos.xg.web_policy_id": [ "12" ], "sophos.xg.app_is_cloud": [ "FALSE" ], "destination.as.number": [ 13335 ], "sophos.xg.hb_status": [ "No Heartbeat" ], "destination.as.organization.name.text": [ "CLOUDFLARENET" ], "destination.ip": [ "1.1.1.1" ], "network.transport": [ "icmp" ], "event.duration": [ 33000000000 ], "sophos.xg.dst_country": [ "AUS" ], "event.action": [ "allowed" ], "sophos.xg.ips_policy_id": [ 8 ], "event.ingested": [ "2021-11-15T03:29:53.331Z" ], "@timestamp": [ "2021-11-15T03:29:53.000Z" ], "destination.geo.country_iso_code": [ "AU" ], "sophos.xg.in_display_interface": [ "Port1" ], "sophos.xg.log_subtype": [ "Allowed" ], "agent.ephemeral_id": [ "f8b595b3-d15f-488d-9022-5b86b38396a1" ], "sophos.xg.log_component": [ "Firewall Rule" ], "event.dataset": [ "sophos.xg" ], "sophos.xg.device_model": [ "SFVH" ], "destination.as.organization.name": [ "CLOUDFLARENET" ] } } ``` > [@stephenb](#): > > POST /\_ingest/pipeline/filebeat-7.15.2-sophos-xg-pipeline/\_simulate I do not know how to do a POST, I will go through forum posts on how to complete a POST and and figure out how to complete one and let you know my result. Should that POST go to filebeat or to elastic-search? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 15, 2021, 4:17am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/13 "2021-11-15T04:17:59Z")
You can do the POST in Kibana -\> Dev Tools It goes to Elasticsearch ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 15, 2021, 4:54am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/14 "2021-11-15T04:54:50Z")
I found it! So in the `filebeat-7.15.2-sophos-xg-firewall` The pipeline is looking for `sent_bytes` but your message has `bytes_sent` same for received etc. I do not know which one is correct... I think the easiest fix for now is fix those ingest YMLs `module/sophos/xg/ingest/pipeline.yml` files to line up with your messages then run this to reload the pipelines. `./filebeat setup --pipelines` I tried this for one field and it worked The good thing is those values are copied into the Common ECS fields and then removed so the viz if there are any should be removed. According to these docs your log format looks correct .. I think [https://docs.sophos.com/nsg/sophos-firewall/17.5/Help/en-us/webhelp/onlinehelp/nsg/sfos/concepts/LogFields.html](https://docs.sophos.com/nsg/sophos-firewall/17.5/Help/en-us/webhelp/onlinehelp/nsg/sfos/concepts/LogFields.html) Looks like your message is correct. If you could please write a bug against > **[Issues · elastic/beats](https://github.com/elastic/beats/issues)** > > :tropical\_fish: Beats - Lightweight shippers for Elasticsearch & Logstash - Issues · elastic/beats By the way it looks like it changed because I saw an older version of the sophos log spec and it was the way the pipeline is so perhaps they changed it with the XG or something ---
**Author:** ![Samy\_Weee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samy_weee/32/81251_2.png) [@Samy\_Weee](https://discuss.elastic.co/u/Samy_Weee)\ **Post date:** [November 15, 2021, 6:07am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/15 "2021-11-15T06:07:00Z")
> [@stephenb](#): > > I found it! Great!! I changed firewall.yml file for one of the variables as follows and did filebeat setup --pipelines ; restarted filebeat ```auto - convert: field: sophos.xg.bytes_received target_field: destination.bytes type: long ignore_failure: true ignore_missing: true if: "ctx.sophos?.xg?.bytes_received != null" - set: field: server.bytes value: '{{destination.bytes}}' if: "ctx.destination?.bytes != null" ``` Still no luck seeing the destination.bytes field in number format. Do not know what I am doing wrong. I will check more tomorrow. There multiple fields such as packets, bytes etc to be modified as well which I can do. I will file a bug report once I am convinced this should work. For this input ; ```auto POST /_ingest/pipeline/filebeat-7.15.1-sophos-xg-pipeline/_simulate { "docs": [ { "_index": "m-index", "_id": "kMpUTHoBr7SFhhL5-98P", "_source": { "@timestamp" : "2021-11-09T03:17:20.0516241Z", "message": """<44>device_name="SFW" timestamp="2021-11-14T21:29:53-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFFF" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" duration=33 fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="10:BF:48:7D:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.121" src_country="R1" dst_ip="1.1.1.1" dst_country="AUS" protocol="ICMP" icmp_type=8 packets_sent=2 packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Stop" con_id="2171700160" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2"""} } ] } ``` I got this output ```auto { "docs" : [ { "doc" : { "_index" : "m-index", "_type" : "_doc", "_id" : "kMpUTHoBr7SFhhL5-98P", "_source" : { "server" : { "mac" : "00:50:56:9F:FF:FF", "ip" : "1.1.1.1" }, "log" : { "level" : "informational" }, "destination" : { "geo" : { "continent_name" : "Oceania", "country_name" : "Australia", "location" : { "lon" : 143.2104, "lat" : -33.494 }, "country_iso_code" : "AU" }, "as" : { "number" : 13335, "organization" : { "name" : "CLOUDFLARENET" } }, "mac" : "00:50:56:9F:FF:FF", "ip" : "1.1.1.1" }, "rule" : { "id" : "12" }, "source" : { "mac" : "10:BF:48:7D:FF:FF", "ip" : "192.168.2.121" }, "network" : { "transport" : "icmp" }, "observer" : { "ingress" : { "interface" : { "name" : "Port1" } }, "product" : "XG", "type" : "firewall", "vendor" : "Sophos", "egress" : { "interface" : { "name" : "Port2" } } }, "@timestamp" : "2021-11-15T03:29:53.000Z", "related" : { "ip" : [ "192.168.2.121", "1.1.1.1" ] }, "sophos" : { "xg" : { "icmp_type" : "8", "device_model" : "SFVH", "in_display_interface" : "Port1", "web_policy_id" : "12", "out_display_interface" : "Port2", "con_id" : "2171700160", "bytes_received" : "168", "fw_rule_type" : "USER", "ips_policy_id" : "8", "src_trans_ip" : "192.168.1.2", "app_is_cloud" : "FALSE", "device_name" : "SFW", "log_type" : "Firewall", "packets_sent" : "2 ", "ether_type" : "Unknown (0x0000)", "packets_received" : "2", "nat_rule_id" : "12", "device_serial_id" : "C01001BQC8TFFFF", "app_filter_policy_id" : "6", "timestamp" : "2021-11-14T21:29:53-0600", "severity" : "Information", "dst_country" : "AUS", "log_component" : "Firewall Rule", "log_subtype" : "Allowed", "dst_zone_type" : "WAN", "hb_status" : "No Heartbeat", "message_id" : "00001", "bytes_sent" : "168", "dst_zone" : "WAN", "src_zone_type" : "LAN", "con_event" : "Stop", "src_country" : "R1", "src_zone" : "LAN", "app_resolved_by" : "Signature", "qualifier" : "New", "log_version" : "1" } }, "client" : { "mac" : "10:BF:48:7D:FF:FF", "ip" : "192.168.2.121" }, "event" : { "severity" : "6", "original" : """device_name="SFW" timestamp="2021-11-14T21:29:53-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFFF" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" duration=33 fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="10:BF:48:7D:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.121" src_country="R1" dst_ip="1.1.1.1" dst_country="AUS" protocol="ICMP" icmp_type=8 packets_sent=2 packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Stop" con_id="2171700160" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2""", "code" : "010101600001", "kind" : "event", "module" : "sophos", "start" : "2021-11-15T03:29:53.000Z", "duration" : 33000000000, "ingested" : "2021-11-15T06:05:44.098946522Z", "action" : "allowed", "end" : "2021-11-15T03:30:26.000Z", "category" : [ "network" ], "dataset" : "sophos.xg", "outcome" : "success" } }, "_ingest" : { "timestamp" : "2021-11-15T06:05:44.098946522Z" } } } ] } ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 15, 2021, 6:50am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/16 "2021-11-15T06:50:37Z")
I just did what you specified and it worked for me... ```auto - convert: field: sophos.xg.bytes_sent target_field: source.bytes type: long ignore_failure: true ignore_missing: true if: "ctx.sophos?.xg?.bytes_sent != null" ``` ```auto - convert: field: sophos.xg.bytes_received target_field: destination.bytes type: long ignore_failure: true ignore_missing: true if: "ctx.sophos?.xg?.bytes_received != null" ``` Ran this... ```auto POST /_ingest/pipeline/filebeat-7.15.2-sophos-xg-pipeline/_simulate { "docs": [ { "_index": "m-index", "_id": "kMpUTHoBr7SFhhL5-98P", "_source": { "@timestamp" : "2021-11-09T03:17:20.0516241Z", "message": """<44>device_name="SFW" timestamp="2021-11-14T21:29:53-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFFF" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" duration=33 fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="10:BF:48:7D:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.121" src_country="R1" dst_ip="1.1.1.1" dst_country="AUS" protocol="ICMP" icmp_type=8 packets_sent=2 packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Stop" con_id="2171700160" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2"""} } ] } ``` Got This... Note the Destination bytes etc... did you save you .ymls, you can delete the existing pipeline to make sure... ```auto { "docs" : [ { "doc" : { "_index" : "m-index", "_type" : "_doc", "_id" : "kMpUTHoBr7SFhhL5-98P", "_source" : { "server" : { "bytes" : 168, "mac" : "00:50:56:9F:FF:FF", "ip" : "1.1.1.1" }, "log" : { "level" : "informational" }, "destination" : { "geo" : { "continent_name" : "Oceania", "country_name" : "Australia", "location" : { "lon" : 143.2104, "lat" : -33.494 }, "country_iso_code" : "AU" }, "as" : { "number" : 13335, "organization" : { "name" : "CLOUDFLARENET" } }, "bytes" : 168, "mac" : "00:50:56:9F:FF:FF", "ip" : "1.1.1.1" }, "rule" : { "id" : "12" }, "source" : { "bytes" : 168, "mac" : "10:BF:48:7D:FF:FF", "ip" : "192.168.2.121" }, "network" : { "bytes" : 336, "transport" : "icmp" }, "observer" : { "ingress" : { "interface" : { "name" : "Port1" } }, "product" : "XG", "type" : "firewall", "vendor" : "Sophos", "egress" : { "interface" : { "name" : "Port2" } } }, "@timestamp" : "2021-11-15T03:29:53.000Z", "related" : { "ip" : [ "192.168.2.121", "1.1.1.1" ] }, "sophos" : { "xg" : { "icmp_type" : "8", "device_model" : "SFVH", "in_display_interface" : "Port1", "web_policy_id" : "12", "out_display_interface" : "Port2", "con_id" : "2171700160", "fw_rule_type" : "USER", "ips_policy_id" : "8", "src_trans_ip" : "192.168.1.2", "app_is_cloud" : "FALSE", "device_name" : "SFW", "log_type" : "Firewall", "packets_sent" : "2 ", "ether_type" : "Unknown (0x0000)", "packets_received" : "2", "nat_rule_id" : "12", "device_serial_id" : "C01001BQC8TFFFF", "app_filter_policy_id" : "6", "timestamp" : "2021-11-14T21:29:53-0600", "severity" : "Information", "dst_country" : "AUS", "log_component" : "Firewall Rule", "log_subtype" : "Allowed", "dst_zone_type" : "WAN", "hb_status" : "No Heartbeat", "message_id" : "00001", "dst_zone" : "WAN", "src_zone_type" : "LAN", "con_event" : "Stop", "src_country" : "R1", "src_zone" : "LAN", "app_resolved_by" : "Signature", "qualifier" : "New", "log_version" : "1" } }, "client" : { "bytes" : 168, "mac" : "10:BF:48:7D:FF:FF", "ip" : "192.168.2.121" }, "event" : { "severity" : "6", "original" : """device_name="SFW" timestamp="2021-11-14T21:29:53-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFFF" log_id="010101600001" log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" log_version=1 severity="Information" duration=33 fw_rule_id="12" nat_rule_id="12" fw_rule_type="USER" web_policy_id=12 ips_policy_id=8 app_filter_policy_id=6 ether_type="Unknown (0x0000)" in_interface="Port1" out_interface="Port2" src_mac="10:BF:48:7D:FF:FF" dst_mac="00:50:56:9F:FF:FF" src_ip="192.168.2.121" src_country="R1" dst_ip="1.1.1.1" dst_country="AUS" protocol="ICMP" icmp_type=8 packets_sent=2 packets_received=2 bytes_sent=168 bytes_received=168 src_trans_ip="192.168.1.2" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" con_event="Stop" con_id="2171700160" hb_status="No Heartbeat" app_resolved_by="Signature" app_is_cloud="FALSE" qualifier="New" in_display_interface="Port1" out_display_interface="Port2""", "code" : "010101600001", "kind" : "event", "module" : "sophos", "start" : "2021-11-15T03:29:53.000Z", "duration" : 33000000000, "ingested" : "2021-11-15T06:47:08.476196100Z", "action" : "allowed", "end" : "2021-11-15T03:30:26.000Z", "category" : [ "network" ], "dataset" : "sophos.xg", "outcome" : "success" } }, "_ingest" : { "timestamp" : "2021-11-15T06:47:08.4761961Z" } } } ] } ``` ---
**Author:** ![jamie.hynds](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamie.hynds/32/84205_2.png) [@jamie.hynds](https://discuss.elastic.co/u/jamie.hynds)\ **Post date:** [November 15, 2021, 11:42am UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/17 "2021-11-15T11:42:49Z")
Thanks for flagging this. I've created an issue to address the bug in our pipeline - [Sophos XG - sent\_bytes bug · Issue #2158 · elastic/integrations · GitHub](https://github.com/elastic/integrations/issues/2158) ---
**Author:** ![Samy\_Weee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samy_weee/32/81251_2.png) [@Samy\_Weee](https://discuss.elastic.co/u/Samy_Weee)\ **Post date:** [November 15, 2021, 11:25pm UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/18 "2021-11-15T23:25:21Z")
> [@jamie.hynds](#): > > Thanks for flagging this. I've created an issue to address the bug in our pipeline - [[sophos] XG - sent\_bytes bug · Issue #2158 · elastic/integrations · GitHub](https://github.com/elastic/integrations/issues/2158) Thank you for creating a bug report. However this is not only limited to sent\_bytes. All following need addressed and they are all numeric fields as well. ; ```auto sent_bytes --change-to --> bytes_sent bytes_received --change-to --> bytes_received sent_pkts --change-to --> packets_sent recv_pkts --change-to --> packets_received ``` I also updated your bug report with this information. ---
**Author:** ![Samy\_Weee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samy_weee/32/81251_2.png) [@Samy\_Weee](https://discuss.elastic.co/u/Samy_Weee)\ **Post date:** [November 15, 2021, 11:38pm UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/19 "2021-11-15T23:38:02Z")
> [@stephenb](#): > > `./filebeat setup --pipelines` > > I tried this for one field and it worked I had to do ./filebeat setup --pipelines sophos and that worked. ---
**Author:** ![Samy\_Weee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samy_weee/32/81251_2.png) [@Samy\_Weee](https://discuss.elastic.co/u/Samy_Weee)\ **Post date:** [November 15, 2021, 11:57pm UTC](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577/20 "2021-11-15T23:57:54Z")
Here is another issue. Following input should be processed by cfilter.yml pipeline - file as This has log\_type="Content Filtering" ; However there is no line in cfilter.yml to convert bytes\_sent, bytes\_received to numberic values. I think they were missing from the start. If @stephenb and @jamie.hynds agree you should add this to the bug report as well. These fields are important as they are required for visualizations. Thank you for all the help. ```auto device_name="SFW" timestamp="2021-11-15T17:33:25-0600" device_model="SFVH" device_serial_id="C01001BQC8TFFFF" log_id=050901616001 log_type="Content Filtering" log_component="HTTP" log_subtype="Allowed" log_version=1 severity="Information" fw_rule_id="12" web_policy_id=12 http_category="Video hosting" http_category_type="Unproductive" url="https://www.youtube.com" src_ip="192.168.2.162" dst_ip="142.250.115.136" protocol="TCP" src_port=55644 dst_port=443 bytes_sent=8347 bytes_received=20257 domain="www.youtube.com" http_status="0" con_id=2617086912 app_is_cloud="FALSE" used_quota="0" src_zone_type="LAN" src_zone="LAN" dst_zone_type="WAN" dst_zone="WAN" src_country="R1" dst_country="USA" ``` [Next page](https://discuss.elastic.co/t/es-7-15-filebeats-sophos-xg-module-not-separating-data-into-variables/288577.md?page=2)