# Filebeat Cisco AMP Module and host.name Error **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [March 26, 2021, 2:49pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510 "2021-03-26T14:49:46Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![MakoWish](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Post date:** [March 26, 2021, 2:49pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/1 "2021-03-26T14:49:46Z")
We are currently using Python to poll the Cisco AMP API, then Logstash picks up the results, but I noticed there is a new Cisco AMP module for Filebeat, so I figured I would give it a try. There are a few issues I have noticed with the new module, but I think the most important to address is the fact all messages are coming through with host details for the device Filebeat is running on, instead of the device the event relates to. It appears pipeline.yml attempts to use the event's source details: ```auto - rename: field: cisco.amp.computer.hostname target_field: host.name ignore_missing: true - set: field: host.hostname value: "{{ host.name }}" if: ctx?.host?.name != null - rename: field: cisco.amp.computer.user target_field: host.user.name ignore_missing: true ``` but each event is indexed with an error `error.message : field [host.name] already exists`. A few of the other issues include vulnerability details being a non-searchable array of hashes (shouldn't these also utilize the ECS [Vulnerability](https://www.elastic.co/guide/en/ecs/current/ecs-vulnerability.html) fields?); I feel `computer.user` should be mapped to `user.name` instead of `host.user.name` for correlation-sake; file-related events do not map to ECS [File](https://www.elastic.co/guide/en/ecs/current/ecs-file.html) fields... I do not see any note that this module is considered Beta. Are there plans to work more on this one? We are reverting back to our Python/Logstash method for now, but would love to come back and test this again once some of the issues are addressed. ---
**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\ **Post date:** [March 29, 2021, 9:46am UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/2 "2021-03-29T09:46:34Z")
Hello @MakoWish . Happy to see some early adoption on the Cisco AMP module, it's a brand new one that I made, and any feedback is always appreciated. Trying to tackle your issues one at a time: 1. Filebeat modules usually comes pre-configured with a choice to not add filebeat host details to the event, this is done through tags, so for example the default configuration for Cisco AMP module with this configuration: Config template: `publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}` Default values: ``` - name: tags default: [cisco-amp, forwarded] ``` You might be setting your own tags on top, which could be overwriting this setting, so if you add the tag `forwarded`, this should be resolved for you as well. 1. For the ECS vulnerability fields, it is something we are currently working on, the issue right now is that its supposed to be a field for a single vulnerability, compared to arrays, so since these are nested objects it's a bit trickier, there is something done to bypass that for now, by adding all CVE's to a searchable field called `cisco.amp.related.cve`, are there other specific fields you might would need to search? That is not meant to be a very long-term solution, but it could be a current workaround. ---
**Author:** ![MakoWish](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Post date:** [March 29, 2021, 2:29pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/3 "2021-03-29T14:29:16Z")
Thank you for the reply, @Marius_Iversen ! I just checked the 'filebeat-7.12.0-cisco-amp-pipeline' pipeline and filebeat.yml, but I do not see either of those settings you mention. I would have to fire the module back up again to see if there are any additional tags added, but I do not believe I had anything. I will try and get it fired back up to check today. ---
**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\ **Post date:** [March 29, 2021, 2:38pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/4 "2021-03-29T14:38:18Z")
Ah yeah the settings are hidden behind default values in our beat modules, the first one is added to all filebeat modules that does not require host.\* fields to be populated, and requires the "forwarded" tag, which is also a default value. However I could see if you add your own tag, that the custom tag would be removed at that point. If you would have the possibility or want to help out with some testing, I am more than happy to try to give you some direction, and if any bugs is found I can apply them to the next possible release, we don't want any of these causing issues for users so ๐Ÿ™‚ ---
**Author:** ![MakoWish](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Post date:** [March 29, 2021, 2:41pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/5 "2021-03-29T14:41:00Z")
Just fired the module back up again, and here are the tags in the Cisco AMP events: tags : cisco-amp, forwarded, beats\_input\_codec\_plain\_applied ---
**Author:** ![MakoWish](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Post date:** [March 29, 2021, 2:45pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/6 "2021-03-29T14:45:31Z")
... and I am absolutely willing to help out however I can. ๐Ÿ™‚ The nice thing is that with my Python method, I am indexing to `cisco_amp-*` instead of the default `filebeat-*` indices, so I can test the module any time without messing with our "production" indices for the events. ---
**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\ **Post date:** [March 29, 2021, 2:47pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/7 "2021-03-29T14:47:09Z")
Sounds good to me! You should be able to send data to another index as well with filebeat if you want, but I can see how it makes it easier to test. So even with the tags, do you still get the wrong` host.*` fields populated? Does all or any of the indexed documents have `error.message` populated? ---
**Author:** ![MakoWish](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Post date:** [March 29, 2021, 2:53pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/8 "2021-03-29T14:53:09Z")
Yes, the `host.*` fields are populated with details about the host Filebeat is running on. Every event has the error. ![filebeat-cisco-amp-module-errors](https://us1.discourse-cdn.com/elastic/original/3X/6/8/687842eb137f58795fc69a229227228353cc88e8.png) ---
**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\ **Post date:** [March 29, 2021, 3:16pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/9 "2021-03-29T15:16:12Z")
Just had to confirm something @MakoWish , its actually the Logstash that appends the hostname based on @metadata fields from filebeat, the way to resolve this is 2 ways. First, I am guessing you are on 7.12 Logstash as well? 7.12 Added a new ecs\_compatibility mode, could you set this option to `v1` ? That will resolve your host issues. > **[Beats input plugin | Logstash Reference \[7.12\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-ecs_compatibility)** If you are on anything before 7.12 Logstash, you would need to remove the host.\* fields with a Logstash pipeline before sending it to ES unfortunately. ---
**Author:** ![MakoWish](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Post date:** [March 29, 2021, 3:22pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/10 "2021-03-29T15:22:03Z")
I actually have not yet upgraded our Logstash servers to 7.12.0. They are currently on 7.11.1, but I can work to get them upgraded today. I will get that done and add the `ecs_compatibility => "v1"` to the Beats input. I will post back with what I get after that. I sincerely appreciate your help with this! ---
**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\ **Post date:** [March 29, 2021, 3:37pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/11 "2021-03-29T15:37:19Z")
No problem at all, that's why we are here ๐Ÿ™‚ Would be good to know once you get past the pipeline errors what you think about the rest. A lot of the parsing will currently just not happen, so the end result should be easier to look at once the Logstash part has been resolved. Since Cisco AMP is a brand new integration there might be some small issues or ECS mappings that could be done better, so we could always discuss that after this issue has been resolved so we can see how the end result looks like. If you want to look at some of the current parsed output, we do have public testdata, both raw and in its transformed state here (\*-expected.json is parsed data): [beats/x-pack/filebeat/module/cisco/amp/test at master ยท elastic/beats ยท GitHub](https://github.com/elastic/beats/tree/master/x-pack/filebeat/module/cisco/amp/test) ---
**Author:** ![MakoWish](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Post date:** [March 29, 2021, 3:49pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/12 "2021-03-29T15:49:51Z")
Unfortunately, the issue remains. โ˜น ---
**Author:** ![MakoWish](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Post date:** [March 29, 2021, 4:00pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/13 "2021-03-29T16:00:30Z")
This is the Beats input running on all our 7.12.0 Logstash servers after the suggested change: ```auto input { beats { port => 5044 ecs_compatibility => "v1" ssl => true ssl_certificate_authorities => [ "./certs/intermediate.crt", "./certs/ca.crt" ] ssl_certificate => "./certs/servercert.crt" ssl_key => "./certs/servercert.key" ssl_verify_mode => "force_peer" add_field => {"[logstash][port]" => "5044"} add_field => {"[logstash][protocol]" => "beats"} add_field => {"[logstash][secured]" => "true"} } } ``` ---
**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\ **Post date:** [March 29, 2021, 4:05pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/14 "2021-03-29T16:05:34Z")
Hmm, just thinking about the best way to debug this, its a bit tedious, but I would say the easiest way to get a 100% assurance where the host fields are added, you could make a file output instead of elasticsearch first on filebeat, and then on Logstash (if the host.\* fields are missing from filebeat only). The general idea seems to just be that for some reason it has the host fields, and we would need to find out where its added before we can debug it further. In filebeat.yml, could you comment out the elasticsearch output and just add something like: ``` output.file: path: "/tmp/filebeat" filename: filebeat ``` And see if the host fields are there? And just to confirm, you did enable the filebeat module compared to using a raw input in `filebeat.yml` right? On the logstash side, you could also confirm that the logstash plugin is actually the correct version, if you run: `bin/logstash-plugin list` And check that the beats input plugin is version `6.1.0`. ---
**Author:** ![MakoWish](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Post date:** [March 29, 2021, 4:12pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/15 "2021-03-29T16:12:05Z")
I will add the file output to see what we get. That is correct. I enabled the Filebeat Cisco module. Within `./modules.d/cisco.yml`, I disabled all but the AMP input. Running `./bin/logstash-plugin list --verbose` shows that `logstash-input-beats` is version 6.1.0. ---
**Author:** ![MakoWish](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Post date:** [March 29, 2021, 4:23pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/16 "2021-03-29T16:23:12Z")
Here is one sanitized event from the file output: ```auto {"@timestamp":"2021-03-29T16:16:18.558Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.12.0","pipeline":"filebeat-7.12.0-cisco-amp-pipeline","_id":"bc92555d02c032dad9f8fcb565275eb2c6e9fda56590bed750e84693688b720b"},"message":"{\"data\":{\"computer\":{\"active\":true,\"connector_guid\":\"REDACTED\",\"external_ip\":\"REDACTED\",\"hostname\":\"REDACTED.contoso.com\",\"links\":{\"computer\":\"https://api.amp.cisco.com/v1/computers/REDACTED\",\"group\":\"https://api.amp.cisco.com/v1/groups/REDACTED\",\"trajectory\":\"https://api.amp.cisco.com/v1/computers/REDACTED/trajectory\"},\"network_addresses\":[{\"ip\":\"REDACTED\",\"mac\":\"REDACTED\"}]},\"connector_guid\":\"REDACTED\",\"date\":\"2021-03-28T14:01:04+00:00\",\"event_type\":\"Scan Started\",\"event_type_id\":554696714,\"group_guids\":[\"REDACTED\"],\"id\":6944704694472147000,\"scan\":{\"description\":\"Flash Scan\"},\"timestamp\":1616940064,\"timestamp_nanoseconds\":486000000},\"metadata\":{\"links\":{\"next\":\"https://api.amp.cisco.com/v1/events?limit=100\\u0026start_date=2021-03-25T16%3A14%3A53%2B00%3A00\\u0026offset=9000\",\"prev\":\"https://api.amp.cisco.com/v1/events?limit=100\\u0026start_date=2021-03-25T16%3A14%3A53%2B00%3A00\\u0026offset=8800\",\"self\":\"https://api.amp.cisco.com/v1/events?limit=100\\u0026start_date=2021-03-25T16%3A14%3A53%2B00%3A00\\u0026offset=8900\"},\"results\":{\"current_item_count\":100,\"index\":8900,\"items_per_page\":100,\"total\":31347}},\"version\":\"v1.2.0\"}","ecs":{"version":"1.7.0"},"event":{"dataset":"cisco.amp","timezone":"PDT","created":"2021-03-29T16:16:18.558Z","module":"cisco"},"fileset":{"name":"amp"},"service":{"type":"cisco"},"input":{"type":"httpjson"},"json":{"metadata":{"links":{"self":"https://api.amp.cisco.com/v1/events?limit=100&start_date=2021-03-25T16%3A14%3A53%2B00%3A00&offset=8900","next":"https://api.amp.cisco.com/v1/events?limit=100&start_date=2021-03-25T16%3A14%3A53%2B00%3A00&offset=9000","prev":"https://api.amp.cisco.com/v1/events?limit=100&start_date=2021-03-25T16%3A14%3A53%2B00%3A00&offset=8800"},"results":{"current_item_count":100,"index":8900,"items_per_page":100,"total":31347}},"version":"v1.2.0","data":{"connector_guid":"REDACTED","date":"2021-03-28T14:01:04+00:00","id":6944704694472147000,"timestamp":1616940064,"timestamp_nanoseconds":486000000,"computer":{"active":true,"connector_guid":"REDACTED","external_ip":"REDACTED","hostname":"REDACTED.contoso.com","links":{"group":"https://api.amp.cisco.com/v1/groups/REDACTED","trajectory":"https://api.amp.cisco.com/v1/computers/REDACTED/trajectory","computer":"https://api.amp.cisco.com/v1/computers/REDACTED"},"network_addresses":[{"ip":"REDACTED","mac":"REDACTED"}]},"event_type":"Scan Started","event_type_id":554696714,"group_guids":["REDACTED"],"scan":{"description":"Flash Scan"}}},"agent":{"name":"REDACTED","type":"filebeat","version":"7.12.0","hostname":"REDACTED","ephemeral_id":"REDACTED","id":"REDACTED"},"host":{"hostname":"REDACTED","architecture":"x86_64","os":{"family":"debian","name":"Ubuntu","kernel":"5.4.0-70-generic","codename":"focal","type":"linux","platform":"ubuntu","version":"20.04.2 LTS (Focal Fossa)"},"id":"b78fd072ff7d4ab48cd30bbf3951b041","containerized":false,"ip":["REDACTED","REDACTED"],"mac":["REDACTED"]},"tags":["cisco-amp","forwarded"]} ``` Let me know if you would like me to grab more. ---
**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\ **Post date:** [March 29, 2021, 6:45pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/17 "2021-03-29T18:45:58Z")
Okay so in your case it seems that host details are still being added, which usually points to a misconfiguration in either `filebeat.yml` or `./modules.d/cisco.yml` Just out of curiosity, is the `*.yml` files above quite old, and have been following upgrades for a longer time, or is this from a fresh installation? I will also try to reproduce it from our side as well to try and hunt down the culprit ๐Ÿ™‚ ---
**Author:** ![MakoWish](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Post date:** [March 29, 2021, 7:37pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/18 "2021-03-29T19:37:08Z")
It is a very old `filebeat.yml` file, but it is also very basic. For kicks, I looked at a new `filebeat.yml` file, and I noticed there is now `when.not.contains.tags: forwarded` conditional. ```auto processors: - add_host_metadata: when.not.contains.tags: forwarded ``` ๐Ÿคฆโ€โ™‚๏ธ ---
**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\ **Post date:** [March 29, 2021, 7:57pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/19 "2021-03-29T19:57:55Z")
Ah that makes sense, yeah it happens from time to time when there is old and custom yml files, not all our new additions would be copied at that point. Let me know later on how it goes further down the line then! ---
**Author:** ![MakoWish](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Post date:** [March 29, 2021, 8:08pm UTC](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510/20 "2021-03-29T20:08:53Z")
Few things I am noticing ... Most other Beats agents/modules will list `host.name` as ie. `Device1` where `host.hostname` is typically a FQDN resembling `Device1.Contoso.com`, but the pipeline is just copying the full value and not stripping the domain name from `host.name`. This would cause issues for some of our dashboards, because `host.name` is a URL in all our index patterns to link to a details page for the device. Being different from all other event sources would link to a page that would show no events from other sources (hope that makes sense), or you followed a link from another source index, it would not include events from Cisco AMP. Also, for "Install Started" events, the computer name is listed under `cisco.amp.hostname` where all other events have the computer name in `cisco.amp.computer.hostname`. Because of this, any "Install Started" events are not getting the host name copied over. Any thoughts on copying the user name to `user.name` instead of `host.user.name`? Same situation there where we have a details page for users, and no events would show for Cisco AMP since it is not using the same field as all other events. [Next page](https://discuss.elastic.co/t/filebeat-cisco-amp-module-and-host-name-error/268510.md?page=2)