Office 365 Module - not an IP string literal

Elastic Stack Beats
,
1

Getting the following error with default o365 module setup. Casual glance seems to say this is mostly happening on Exchange related audit data but not enough to know yet.

 May 14 16:51:23 ainfcp1esl00001 filebeat: 2020-05-14T16:51:23.611-0700#011WARN#011[elasticsearch]#011elasticsearch/client.go:384#011Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x0, ext:63724844335, loc:(*time.Location)(nil)}, Meta:{"_id":"17efaaf6-8bb5-4829-27b1-08d7f6153bf3","pipeline":"filebeat-7.7.0-o365-audit-pipeline"}, Fields:{"agent":{"ephemeral_id":"c7b7e853-c8ba-4749-80e0-b52d98db95ed","hostname":"hostname.com","id":"9ff5gad7-a611-4443-bc73-3d377f156e21","type":"filebeat","version":"7.7.0"},"client":{"address":"[1.2.2.3]:7972","ip":"[1.2.2.3]","port":"7972"},"domain":"domain.com","ecs":{"version":"1.5.0"},"error":{"message":"GoError: cannot override existing key with `client.ip`"},"event":{"action":"Create","category":"web","code":"ExchangeItem","dataset":"o365.audit","id":"17efaag6-7bb5-4829-27b1-08d7f6153bf3","kind":"event","module":"o365","outcome":"success","provider":"Exchange","type":"info"},"fields":{},"fileset":{"name":"audit"},"host":{"architecture":"x86_64","containerized":false,"hostname":"hostname.com","id":"a962373b0c22485eae53ee6e404cd357","ip":["10.1.2.200"fe80::e75a:ad3:88ff:1ad0"],"mac":["00:50:56:a7:f8:4d"],"name":"hostname.com","os":{"codename":"Core","family":"redhat","kernel":"3.10.0-1127.el7.x86_64","name":"CentOS Linux","platform":"centos","version":"7 (Core)"}},"input":{"type":"o365audit"},"o365audit":{"ClientIP":"[1.2.2.3]:7972","ClientIPAddress":"[1.2.2.3]:7972","ClientInfoString":"Client=MSExchangeRPC","ClientProcessName":"OUTLOOK.EXE","ClientVersion":"16.0.12730.20144"
...
, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [client.ip] of type [ip] in document with id '17efaaf6-8bb5-4829-27b1-08d7f6153bf3'. Preview of field's value: '[1.2.2.3]'","caused_by":{"type":"illegal_argument_exception","reason":"'[1.2.2.3]' is not an IP string literal."}}
2

Got the same issue, cant index any log inputs from O365 in my setup.

(Update, some data have started to be indexed, might have been a template thing).

3

Also seeing the same issue, with exchange events - i notice earlier on in the warning message the following:

"error":{"message":"GoError: cannot override existing key with client.ip"}

Unfortunately I don't understand enough about what is happening to work out what is going on - anyone?

I should note that some data does get into elastic, just not events where this is happening.

4

Thanks for reporting, I've created the following issue https://github.com/elastic/beats/issues/18587 and will submit a fix asap.

In our tests we observed that this field will be ipv4:port or [ipv6]:port, but never [ipv4]:port. There's no reason to surround an IPv4 in brackets.

2 Likes
5

Here's the candidate fix PR: #18591

For those affected by this problem, can you try replacing your existing module/o365/audit/config/pipeline.js with this one:
https://raw.githubusercontent.com/elastic/beats/665ee9a50e9483e696bd772b941a9bef9089cc35/x-pack/filebeat/module/o365/audit/config/pipeline.js

To see if that solves the problem in all cases.

1 Like
6

So I've added this and it "seems" to be working. I'll monitor for a bit and let you know if I see anything else.

1 Like
7

Many thanks, the fix seems to be working well for me too.

9

The fix will be released in 7.7.1 and 7.8.0. Until then the workaround is to apply the patch above.

1 Like
10

We saw the same error message with client.port in our Exchange audit data... not nearly as prevalent as client.ip though.

error":{"message":"GoError: cannot override existing key with client.port "}

11

@nateut this is after updating to the pipeline above?

If so, can you share a full document with this error?

12

Haven't seen it since we upgraded to 7.7.1... thanks!

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.