Filebeat Cisco AMP Module and host.name Error

Elastic Stack Beats
1

We are currently using Python to poll the Cisco AMP API, then Logstash picks up the results, but I noticed there is a new Cisco AMP module for Filebeat, so I figured I would give it a try. There are a few issues I have noticed with the new module, but I think the most important to address is the fact all messages are coming through with host details for the device Filebeat is running on, instead of the device the event relates to.

It appears pipeline.yml attempts to use the event's source details:

- rename:
    field: cisco.amp.computer.hostname
    target_field: host.name
    ignore_missing: true
- set:
    field: host.hostname
    value: "{{ host.name }}"
    if: ctx?.host?.name != null
- rename:
    field: cisco.amp.computer.user
    target_field: host.user.name
    ignore_missing: true

but each event is indexed with an error error.message : field [host.name] already exists.

A few of the other issues include vulnerability details being a non-searchable array of hashes (shouldn't these also utilize the ECS Vulnerability fields?); I feel computer.user should be mapped to user.name instead of host.user.name for correlation-sake; file-related events do not map to ECS File fields...

I do not see any note that this module is considered Beta. Are there plans to work more on this one? We are reverting back to our Python/Logstash method for now, but would love to come back and test this again once some of the issues are addressed.

2

Hello @MakoWish . Happy to see some early adoption on the Cisco AMP module, it's a brand new one that I made, and any feedback is always appreciated.

Trying to tackle your issues one at a time:

  1. Filebeat modules usually comes pre-configured with a choice to not add filebeat host details to the event, this is done through tags, so for example the default configuration for Cisco AMP module with this configuration:

Config template:
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

Default values:

 - name: tags
   default: [cisco-amp, forwarded]

You might be setting your own tags on top, which could be overwriting this setting, so if you add the tag forwarded, this should be resolved for you as well.

  1. For the ECS vulnerability fields, it is something we are currently working on, the issue right now is that its supposed to be a field for a single vulnerability, compared to arrays, so since these are nested objects it's a bit trickier, there is something done to bypass that for now, by adding all CVE's to a searchable field called cisco.amp.related.cve, are there other specific fields you might would need to search?
    That is not meant to be a very long-term solution, but it could be a current workaround.
3

Thank you for the reply, @Marius_Iversen !

I just checked the 'filebeat-7.12.0-cisco-amp-pipeline' pipeline and filebeat.yml, but I do not see either of those settings you mention.

I would have to fire the module back up again to see if there are any additional tags added, but I do not believe I had anything. I will try and get it fired back up to check today.

4

Ah yeah the settings are hidden behind default values in our beat modules, the first one is added to all filebeat modules that does not require host.* fields to be populated, and requires the "forwarded" tag, which is also a default value.
However I could see if you add your own tag, that the custom tag would be removed at that point.

If you would have the possibility or want to help out with some testing, I am more than happy to try to give you some direction, and if any bugs is found I can apply them to the next possible release, we don't want any of these causing issues for users so :slight_smile:

5

Just fired the module back up again, and here are the tags in the Cisco AMP events:

tags : cisco-amp, forwarded, beats_input_codec_plain_applied

6

... and I am absolutely willing to help out however I can. :slight_smile:

The nice thing is that with my Python method, I am indexing to cisco_amp-* instead of the default filebeat-* indices, so I can test the module any time without messing with our "production" indices for the events.

7

Sounds good to me! You should be able to send data to another index as well with filebeat if you want, but I can see how it makes it easier to test.

So even with the tags, do you still get the wrong host.* fields populated?
Does all or any of the indexed documents have error.message populated?

8

Yes, the host.* fields are populated with details about the host Filebeat is running on. Every event has the error.

filebeat-cisco-amp-module-errors
filebeat-cisco-amp-module-errors1452×374 31.2 KB

9

Just had to confirm something @MakoWish , its actually the Logstash that appends the hostname based on @metadata fields from filebeat, the way to resolve this is 2 ways.

First, I am guessing you are on 7.12 Logstash as well? 7.12 Added a new ecs_compatibility mode, could you set this option to v1 ? That will resolve your host issues.

If you are on anything before 7.12 Logstash, you would need to remove the host.* fields with a Logstash pipeline before sending it to ES unfortunately.

10

I actually have not yet upgraded our Logstash servers to 7.12.0. They are currently on 7.11.1, but I can work to get them upgraded today. I will get that done and add the ecs_compatibility => "v1" to the Beats input. I will post back with what I get after that.

I sincerely appreciate your help with this!

1 Like
11

No problem at all, that's why we are here :slight_smile:

Would be good to know once you get past the pipeline errors what you think about the rest. A lot of the parsing will currently just not happen, so the end result should be easier to look at once the Logstash part has been resolved.

Since Cisco AMP is a brand new integration there might be some small issues or ECS mappings that could be done better, so we could always discuss that after this issue has been resolved so we can see how the end result looks like.

If you want to look at some of the current parsed output, we do have public testdata, both raw and in its transformed state here (*-expected.json is parsed data): beats/x-pack/filebeat/module/cisco/amp/test at master · elastic/beats · GitHub

12

Unfortunately, the issue remains. :frowning:

13

This is the Beats input running on all our 7.12.0 Logstash servers after the suggested change:

input {
  beats {
    port => 5044
    ecs_compatibility => "v1"
    ssl => true
    ssl_certificate_authorities => [
      "./certs/intermediate.crt",
      "./certs/ca.crt"
    ]
    ssl_certificate => "./certs/servercert.crt"
    ssl_key => "./certs/servercert.key"
    ssl_verify_mode => "force_peer"
    add_field => {"[logstash][port]" => "5044"}
    add_field => {"[logstash][protocol]" => "beats"}
    add_field => {"[logstash][secured]" => "true"}
  }
}
14

Hmm, just thinking about the best way to debug this, its a bit tedious, but I would say the easiest way to get a 100% assurance where the host fields are added, you could make a file output instead of elasticsearch first on filebeat, and then on Logstash (if the host.* fields are missing from filebeat only).

The general idea seems to just be that for some reason it has the host fields, and we would need to find out where its added before we can debug it further.

In filebeat.yml, could you comment out the elasticsearch output and just add something like:

output.file:
  path: "/tmp/filebeat"
  filename: filebeat

And see if the host fields are there? And just to confirm, you did enable the filebeat module compared to using a raw input in filebeat.yml right?

On the logstash side, you could also confirm that the logstash plugin is actually the correct version, if you run:
bin/logstash-plugin list

And check that the beats input plugin is version 6.1.0.

15

I will add the file output to see what we get.

That is correct. I enabled the Filebeat Cisco module. Within ./modules.d/cisco.yml, I disabled all but the AMP input.

Running ./bin/logstash-plugin list --verbose shows that logstash-input-beats is version 6.1.0.

16

Here is one sanitized event from the file output:

{"@timestamp":"2021-03-29T16:16:18.558Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.12.0","pipeline":"filebeat-7.12.0-cisco-amp-pipeline","_id":"bc92555d02c032dad9f8fcb565275eb2c6e9fda56590bed750e84693688b720b"},"message":"{\"data\":{\"computer\":{\"active\":true,\"connector_guid\":\"REDACTED\",\"external_ip\":\"REDACTED\",\"hostname\":\"REDACTED.contoso.com\",\"links\":{\"computer\":\"https://api.amp.cisco.com/v1/computers/REDACTED\",\"group\":\"https://api.amp.cisco.com/v1/groups/REDACTED\",\"trajectory\":\"https://api.amp.cisco.com/v1/computers/REDACTED/trajectory\"},\"network_addresses\":[{\"ip\":\"REDACTED\",\"mac\":\"REDACTED\"}]},\"connector_guid\":\"REDACTED\",\"date\":\"2021-03-28T14:01:04+00:00\",\"event_type\":\"Scan Started\",\"event_type_id\":554696714,\"group_guids\":[\"REDACTED\"],\"id\":6944704694472147000,\"scan\":{\"description\":\"Flash Scan\"},\"timestamp\":1616940064,\"timestamp_nanoseconds\":486000000},\"metadata\":{\"links\":{\"next\":\"https://api.amp.cisco.com/v1/events?limit=100\\u0026start_date=2021-03-25T16%3A14%3A53%2B00%3A00\\u0026offset=9000\",\"prev\":\"https://api.amp.cisco.com/v1/events?limit=100\\u0026start_date=2021-03-25T16%3A14%3A53%2B00%3A00\\u0026offset=8800\",\"self\":\"https://api.amp.cisco.com/v1/events?limit=100\\u0026start_date=2021-03-25T16%3A14%3A53%2B00%3A00\\u0026offset=8900\"},\"results\":{\"current_item_count\":100,\"index\":8900,\"items_per_page\":100,\"total\":31347}},\"version\":\"v1.2.0\"}","ecs":{"version":"1.7.0"},"event":{"dataset":"cisco.amp","timezone":"PDT","created":"2021-03-29T16:16:18.558Z","module":"cisco"},"fileset":{"name":"amp"},"service":{"type":"cisco"},"input":{"type":"httpjson"},"json":{"metadata":{"links":{"self":"https://api.amp.cisco.com/v1/events?limit=100&start_date=2021-03-25T16%3A14%3A53%2B00%3A00&offset=8900","next":"https://api.amp.cisco.com/v1/events?limit=100&start_date=2021-03-25T16%3A14%3A53%2B00%3A00&offset=9000","prev":"https://api.amp.cisco.com/v1/events?limit=100&start_date=2021-03-25T16%3A14%3A53%2B00%3A00&offset=8800"},"results":{"current_item_count":100,"index":8900,"items_per_page":100,"total":31347}},"version":"v1.2.0","data":{"connector_guid":"REDACTED","date":"2021-03-28T14:01:04+00:00","id":6944704694472147000,"timestamp":1616940064,"timestamp_nanoseconds":486000000,"computer":{"active":true,"connector_guid":"REDACTED","external_ip":"REDACTED","hostname":"REDACTED.contoso.com","links":{"group":"https://api.amp.cisco.com/v1/groups/REDACTED","trajectory":"https://api.amp.cisco.com/v1/computers/REDACTED/trajectory","computer":"https://api.amp.cisco.com/v1/computers/REDACTED"},"network_addresses":[{"ip":"REDACTED","mac":"REDACTED"}]},"event_type":"Scan Started","event_type_id":554696714,"group_guids":["REDACTED"],"scan":{"description":"Flash Scan"}}},"agent":{"name":"REDACTED","type":"filebeat","version":"7.12.0","hostname":"REDACTED","ephemeral_id":"REDACTED","id":"REDACTED"},"host":{"hostname":"REDACTED","architecture":"x86_64","os":{"family":"debian","name":"Ubuntu","kernel":"5.4.0-70-generic","codename":"focal","type":"linux","platform":"ubuntu","version":"20.04.2 LTS (Focal Fossa)"},"id":"b78fd072ff7d4ab48cd30bbf3951b041","containerized":false,"ip":["REDACTED","REDACTED"],"mac":["REDACTED"]},"tags":["cisco-amp","forwarded"]}

Let me know if you would like me to grab more.

17

Okay so in your case it seems that host details are still being added, which usually points to a misconfiguration in either filebeat.yml or ./modules.d/cisco.yml

Just out of curiosity, is the *.yml files above quite old, and have been following upgrades for a longer time, or is this from a fresh installation?

I will also try to reproduce it from our side as well to try and hunt down the culprit :slight_smile:

18

It is a very old filebeat.yml file, but it is also very basic. For kicks, I looked at a new filebeat.yml file, and I noticed there is now when.not.contains.tags: forwarded conditional.

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded

:man_facepalming:

19

Ah that makes sense, yeah it happens from time to time when there is old and custom yml files, not all our new additions would be copied at that point.

Let me know later on how it goes further down the line then!

20

Few things I am noticing ...

Most other Beats agents/modules will list host.name as ie. Device1 where host.hostname is typically a FQDN resembling Device1.Contoso.com, but the pipeline is just copying the full value and not stripping the domain name from host.name. This would cause issues for some of our dashboards, because host.name is a URL in all our index patterns to link to a details page for the device. Being different from all other event sources would link to a page that would show no events from other sources (hope that makes sense), or you followed a link from another source index, it would not include events from Cisco AMP.

Also, for "Install Started" events, the computer name is listed under cisco.amp.hostname where all other events have the computer name in cisco.amp.computer.hostname. Because of this, any "Install Started" events are not getting the host name copied over.

Any thoughts on copying the user name to user.name instead of host.user.name? Same situation there where we have a details page for users, and no events would show for Cisco AMP since it is not using the same field as all other events.