We are trying to fetch MongoDB Atlas logs. In MongoDB Atlas Activity Logs there is an object named “attr” under which there are many dynamic fields that change depending on logs, these fields can't be mentioned in “fields.yml” since there is no proper documentation related to these fields from MongoDB Atlas. So the only solution to fetch these fields is to dump them directly into Elasticsearch with the runtime datatype. Elastic search automatically assigns datatype to these fields based on its value.
The problem with the above approach is that assume there is a field called “priority” under “attr”, assume the value of this field is “6.5” in the first object so its data type will be considered as float but later if there is second object with value “7” Elasticsearch will consider it as long value
{ "t": { "$date": "2023-07-09T15:27:34.420+05:30" }, "s": "I", "c": "CONTROL", "id": 20722, "ctx": "conn386069", "msg": "Node is a member of a replica set", "attr": { "config": { "members": [ { "priority": 7.5 }, { "priority": 7 }, { "priority": 8.5 } ] } } }
and hence this document will be dropped with the following error.
[elastic_agent.filebeat][warn] Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.July, 11, 9, 59, 17, 357098460, time.Local), Meta:{"input_id":"filestream-MongoDB Atlas-6d260c20-1fd1-11ee-ac31-fdbf9d17c3bd","raw_index":"logs-mongodb_atlas.activity-default","stream_id":"filestream-mongodb_atlas.activity-6d260c20-1fd1-11ee-ac31-fdbf9d17c3bd"}, Fields:{"agent":{"ephemeral_id":"c8e524d0-7660-46f4-990b-212c4b36d446","id":"c95182de-21f4-40d2-b985-cf60ad0d0957","name":"docker-fleet-agent","type":"filebeat","version":"8.8.0"},"data_stream":{"dataset":"mongodb_atlas.activity","namespace":"default","type":"logs"},"ecs":{"version":"8.0.0"},"elastic_agent":{"id":"c95182de-21f4-40d2-b985-cf60ad0d0957","snapshot":false,"version":"8.8.0"},"event":{"dataset":"mongodb_atlas.activity"},"input":{"type":"filestream"},"log":{"file":{"path":"/tmp/service_logs/temp.log"},"offset":0},"message":"{ \"t\": { \"$date\": \"2023-07-09T15:27:34.420+05:30\" }, \"s\": \"I\", \"c\": \"CONTROL\", \"id\": 20722, \"ctx\": \"conn386069\", \"msg\": \"Node is a member of a replica set\", \"attr\": { \"config\": { \"members\": [ { \"priority\": 7.5 }, { \"priority\": 7 }, { \"priority\": 8.5 } ] } } }","tags":["preserve_original_event","forwarded","mongodb_atlas-activity"]}, Private:(*input_logfile.updateOp)(0xc0009b65a0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"mapper [mongodb_atlas.activity.attributes.config.members.priority] cannot be changed from type [float] to [long]"}, dropping event!
we are not sure how to solve this issue in elastic-package. can anyone give us any idea about how to deal with dynamic fields?