Remove IPV6 from host.ip and host.mac field

Elastic Stack Beats
1

Hi,

after filebeat upgrade (installed as daemonset on openshift) from 7.6.2 to 7.9.1 I get very big amount of data per field for host.ip and host.mac. Before upgrade it was just IP version 4 and now I have multiple lines of IP version 6. Example is below:

10.1.56.8, fe80::9063:9cff:fefd:5fea, 10.98.3.1, fe80::78:19ff:fec1:fdbe, fe80::d824:a0ff:fe9f:8e78, fe80::4883:c0ff:fe5d:743a, fe80::80ab:6fff:fe5f:a748, fe80::4cd3:91ff:fe0d:e815, fe80::c841:10ff:fe6d:fb88, fe80::a45e:2cff:fe33:1c5a, fe80::9839:41ff:fee4:bdfc, fe80::549c:9eff:fe6b:b8b8, fe80::f402:4bff:fe25:10c1, fe80::38bf:50ff:fea3:b49c, fe80::8c79:c7ff:fe21:c1a2, fe80::9cc0:16ff:fe3a:cdd0, fe80::f0ea:ffff:fe35:57bd, fe80::101c:88ff:feea:421, fe80::d0a0:9cff:fecf:85e7, fe80::c831:1ff:fe8c:473d, fe80::8039:1dff:fed1:711c, fe80::7451:a7ff:fe17:5805, fe80::ac82:61ff:febc:a5c1, fe80::b8db:35ff:fea5:e2fd, fe80::4c89:fbff:fe80:1908, fe80::f0fc:b3ff:fe53:395d, fe80::1476:afff:fe3f:ac55, fe80::cc81:66ff:feb7:373f, fe80::d429:d6ff:fe58:4497, fe80::f48d:2cff:fefb:a4fc, fe80::c8fb:57ff:fe1a:1ed1, fe80::4ca9:74ff:fe7b:ca2b, fe80::c85f:b8ff:fea3:210f, fe80::fc88:bcff:fe92:fc5a, fe80::6c55:20ff:fe18:b281, fe80::8028:74ff:fef5:b888, fe80::34ae:7cff:fed0:9d19, fe80::10d8:4aff:fea8:8b8c, fe80::c861:14ff:fe92:a05d, fe80::c004:59ff:feef:7285, fe80::2485:89ff:fe2b:cc19, fe80::78b4:47ff:fe64:a024, fe80::bcf2:dcff:febb:1ec5, fe80::8c13:6ff:fec4:e33f, fe80::8092:dbff:fed3:afae, fe80::9c94:7dff:fe9b:71fd, fe80::d412:22ff:fef0:27fd, fe80::d407:36ff:fe1b:deeb, fe80::40ca:3ff:fe34:94bb, fe80::602e:2bff:fe01:6a97, fe80::7884:8cff:fe67:fb7b, fe80::fc79:34ff:fe2b:e1b4, fe80::a884:24ff:fe02:2d18, fe80::b0e5:52ff:fe7a:77dc, fe80::54b0:8ff:fe3f:aa14, fe80::bc13:52ff:fea0:fb70, fe80::7c52:e2ff:fecc:20a, fe80::88fb:dbff:fe41:c9c7, fe80::6cf1:e0ff:fe42:133, fe80::5ca7:c3ff:fe05:7b7a, fe80::3c03:8aff:fe22:16fd, fe80::10f3:41ff:fe86:9e7f, fe80::7413:abff:feff:27af, fe80::902a:f5ff:fec7:593b, fe80::8053:57ff:fe48:e563, fe80::8c09:56ff:febf:9785, fe80::98f5:d3ff:fe57:ce5e, fe80::9819:99ff:fe9e:c38a, fe80::28dd:57ff:fe36:32f0, fe80::982c:e3ff:feef:3c1d, fe80::5885:8fff:fe1d:cad4, fe80::b44b:3ff:fe3e:d0f0, fe80::68b4:a8ff:fea0:5d0b, fe80::807a:b1ff:fe5d:de0d, fe80::d461:dcff:fe8e:e21f, fe80::346d:a9ff:feab:2235, fe80::c04d:4fff:fe54:deea, fe80::c861:15ff:fe2a:ce6e, fe80::b891:8dff:fe61:7e9b, fe80::c80a:89ff:fe16:1de8, fe80::70f9:48ff:fe95:e5f0, fe80::8875:2eff:fe56:59c5, fe80::1cd0:e4ff:fe24:2dc5, fe80::8e6:86ff:fe07:86d6, fe80::a819:a1ff:fee1:1b59, fe80::a803:50ff:fe13:1b9b, fe80::1084:4bff:fef6:5f71, fe80::bcf9:beff:fe68:325f, fe80::8012:9aff:fefd:c720, fe80::6822:d8ff:fec2:371b, fe80::60d0:d8ff:feb3:9adf, fe80::14cf:17ff:fe86:cd9c, fe80::a06d:b2ff:feaf:6a68, fe80::446c:7eff:fe8d:7f2, fe80::3c8b:c3ff:fe2e:860a, fe80::e4da:2ff:fe9f:27e4, fe80::ec0b:84ff:fe1d:c247, fe80::8046:86ff:fe90:ee51, fe80::3c92:30ff:feb5:253a, fe80::cc28:2ff:fe3e:700e, fe80::58c7:1dff:fe0e:19c1, fe80::ac7a:d4ff:feb7:2a92, fe80::b838:4fff:feb0:b657, fe80::4c33:6ff:fe26:caad, fe80::5410:baff:fe15:7b0, fe80::74d9:baff:fee4:53fb, fe80::bc3f:53ff:feaf:5cae, fe80::9411:41ff:fe16:ed13, fe80::d067:53ff:fe80:d57c, fe80::9489:26ff:fe0e:7258, fe80::6890:47ff:fed4:799d, fe80::cca7:adff:feca:4578, fe80::9cdd:efff:fe23:b9cd, fe80::9cd1:29ff:fe00:ebbc, fe80::ac3d:6cff:feb3:25fa, fe80::ac56:acff:fe92:99b2, fe80::3062:e6ff:fea1:e294, fe80::f0e4:b2ff:febc:9bc7, fe80::840a:65ff:feb9:5a42, fe80::a4b5:36ff:fea1:2622, fe80::c815:6dff:fe1f:a81

Is it possible to remove ipv6 address because I don't need it at all?

BR,
Mladen

2

hi!

I think you can't filter those out since they are coming by default from add_host_metadata processor. If you think this is sth team should consider as an enhancement request please feel free to open a Github issue for this so as to let the team evaluate it and schedule its addition.

C.

1 Like
3

Hi Chris,

Thanks a lot for this info. I will open the Github issue for enhancement. Metadata generate a lot data :slight_smile: in my case. With metadata on I have 18GB in two hours and without metadata I have 480MB in two hours :slight_smile:. Probably I will create logstash filter in order to remove host.ip and host.mac fields.

Mladen

4

Hi!

Also script processor might be of help here so as to remove the fields.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.