Hi All,
I wonder if anyone can offer any assistance. We've recently moved to using the Zeek Filebeat module on some remote sensors, as these integrate nicely with the SIEM feature of Kibana, however, the old function we had was sending CSV separated log data to Logstash and performing a large amount of enrichment on the data (GeoIP, Threat Intel Lookup, MAC OUI Lookup, etc, etc) before writing to an index. With the Zeek Filebeat module, we've lost that, so I'd like to #1 revert to using Logstash for enrichment, but #2 keep the Zeek module enabled.
When we send it to LS via FB, we get this type of warning -
[2019-09-21T17:49:54,439][DEBUG][o.e.a.b.TransportShardBulkAction] [es00] [logstash-2019.09.19-000001][0] failed to execute bulk item (index) index {[logstash][_doc][fU66VG0BWPgQZ2342ET9], source[{"@version":"1","fileset":{"name":"files"},"tags":["zeek.files","beats_input_raw_event"],"input":{"type":"log"},"service":{"type":"zeek"},"ecs":{"version":"1.0.1"},"zeek":{"files":{"missing_bytes":0,"duration":0,"timedout":false,"fuid":"FXQTiFzy14jm29p6l","seen_bytes":868,"tx_hosts":["172.16.10.154"],"md5":"82f856eb911b56cda74a076050ab480c","rx_hosts":["10.10.10.12"],"overflow_bytes":0,"mime_type":"application/x-x509-user-cert","depth":0,"sha1":"f7413b5ec1ba603e956cb1d0ebb6c6cf78477c7b","ts":1.569084593217552E9,"is_orig":false,"session_ids":["CSADZt1R3oG5hdSrUe"],"source":"SSL","local_orig":false,"analyzers":["SHA1","X509","MD5"]}},"agent":{"hostname":"0030180d3ce2","id":"6c4646f2-9580-4103-bc9a-4a4177bb9cef","version":"7.3.2","ephemeral_id":"ffad8e9c-b6f0-4a49-959d-8396b68a814f","type":"filebeat"},"log":{"offset":11902874,"file":{"path":"/var/log/bro/current/files.log"}},"host":{"hostname":"0030180d3ce2","os":{"codename":"kali-rolling","platform":"kali","version":"2019.4","kernel":"5.2.0-kali2-amd64","name":"Kali GNU/Linux","family":""},"containerized":false,"id":"ebbd901b30924f1fbbc0e115cb822857","architecture":"x86_64","name":"0030180d3ce2"},"event":{"dataset":"zeek.files","module":"zeek"},"@timestamp":"2019-09-21T16:49:54.240Z"}]}org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [service] of type [text] in document with id 'fU66VG0BWPgQZ2342ET9'. Preview of field's value: '{type=zeek}'
Any ideas on how to fix that?
Cheers
Andy