App Search: Bug in Filebeat IP parsing

Dear Community,

I currently have the problem that query and click data are not being included in ES. It turns out that this is due to Filebeat not being able to read the log file correctly. An error is thrown:

**Preview of field’s value: ‘[0:0:0:0:0:0:0:1]‘\“,**“caused_by\“:{\“type\“:\“illegal_argument_exception\“,\“reason\“:\“‘[0:0:0:0:0:0:0:1]’ is not an IP string literal.\“}}, dropping event!“

Full:

{
  “log.level”: “warn”,
  “@timestamp”: “2023-07-25T09:43:25.060Z”,
  “log.logger”: “elasticsearch”,
  “log.origin”: {
    “file.name”: “elasticsearch/client.go”,
    “file.line”: 429
  },
  “message”: “Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.July, 25, 9, 43, 20, 320000000, time.Local), Meta:null, Fields:{\“agent\“:{\“ephemeral_id\“:\“92934042-b05d-4045-996d-2503cacdab64\“,\“id\“:\“6da624bf-2e98-4eca-b15b-4c4760c9f0ab\“,\“name\“:\“woehler-sql1\“,\“type\“:\“filebeat\“,\“version\“:\“8.7.0\“},\“data_stream\“:{\“dataset\“:\“app_search.analytics\“,\“namespace\“:\“default\“,\“type\“:\“logs\“},\“ecs\“:{\“version\“:\“1.7.0\“},\“event\“:{\“action\“:\“loco_moco_search\“,\“category\“:\“app-search-analytics\“,\“created\“:\“2023-07-25T09:43:20Z\“,\“dataset\“:\“app-search-analytics\“,\“document_ids\“:null,\“loco_moco_search_request_id\“:\“dd9e4608-8f7b-44cb-9a7c-192b733a3961\“,\“query_string\“:\“\”,\“tags\“:[\“swiftype_documents_api\“]},\“host\“:{\“name\“:\“woehler-sql1\“},\“input\“:{\“type\“:\“log\“},\“labels\“:{\“engine_id\“:\“64bf9890f9bc6b592f0379fc\“,\“index_date\“:\“2023.07.25\“,\“lm_account_id\“:\“64bf96adf9bc6b41d07557e7\“},\“log\“:{\“file\“:{\“path\“:\“/var/log/enterprise-search/filebeat.log\“},\“offset\“:631840},\“related\“:{\“ip\“:\“[0:0:0:0:0:0:0:1]\“}}, Private:file.State{Id:\“native::791127-65024\“, PrevId:\“\”, Finished:false, Fileinfo:(*os.fileStat)(0xc000179ad0), Source:\“/var/log/enterprise-search/filebeat.log\“, Offset:632431, Timestamp:time.Date(2023, time.July, 25, 9, 43, 23, 929801211, time.Local), TTL:-1, Type:\“log\“, Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xc1257, Device:0xfe00}, IdentifierName:\“native\“}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\“type\“:\“document_parsing_exception\“,\“reason\“:\“[1:393] failed to parse field [related.ip] of type [ip] in document with id ‘8WZujIkBkKjv5knQlm4S’. **Preview of field’s value: ‘[0:0:0:0:0:0:0:1]‘\“,\**“caused_by\“:{\“type\“:\“illegal_argument_exception\“,\“reason\“:\“‘[0:0:0:0:0:0:0:1]’ is not an IP string literal.\“}}, dropping event!“,
  “service.name”: “filebeat”,
  “ecs.version”: “1.6.0"
}

When I force access via IPv4, the error does not occur. It is exclusively the case with IPv6, as it does not seem to fit the mapping pattern of Filebeat.

Does anyone have any idea what this could be due to? Is this a bug?

Many greetings
Frederik

The main reason for the bug is the entry in the filebeat.log:

THIS WORKS FINE:

{"data_stream":{"type":"logs","dataset":"app_search.analytics","namespace":"default"},"event":{"dataset":"app-search-analytics","category":"app-search-analytics","action":"loco_moco_search","created":"2023-07-26T12:09:50Z","query_string":"bus","loco_moco_search_request_id":"913dae2f-9078-496b-a4f4-6515aa348a3f","document_ids":["140","58","56","132"],"tags":},"ecs":{"version":"1.7.0"},"@timestamp":"2023-07-26T12:09:50.590+00:00","labels":{"index_date":"2023.07.26","engine_id":"64bf987ff9bc6b68ac0379c3","lm_account_id":"64bf96adf9bc6b41d07557e7"},"related":{"ip":"127.0.0.1"}}

THIS DOES NOT WORK:
{"data_stream":{"type":"logs","dataset":"app_search.analytics","namespace":"default"},"event":{"dataset":"app-search-analytics","category":"app-search-analytics","action":"loco_moco_search","created":"2023-07-26T12:05:36Z","query_string":"fuchs","loco_moco_search_request_id":"49d3bb96-63bb-4b7b-9f88-8a682860c012","document_ids":,"tags":},"ecs":{"version":"1.7.0"},"@timestamp":"2023-07-26T12:05:36.484+00:00","labels":{"index_date":"2023.07.26","engine_id":"64bf987ff9bc6b68ac0379c3","lm_account_id":"64bf96adf9bc6b41d07557e7"},"related":{"ip":"[0:0:0:0:0:0:0:1]"}}

But why is this?

Hi @frederik1 !

That error comes from Elasticsearch itself. Elasticsearch is not able to identify an IP enclosed in square brackets as a valid IP.

I'd say that this is a problem in how App Search writes the IP to the log. That seems to be a legitimate bug on App Search side.

There is a bug already reported for App Search, we'll keep you updated on its progress!

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.