I have start and end events in a joined index, where the join is from the end event to the start event. Elastic agents relay events from a Spark system to a Logstash pipeline, which then injects the events into the index seamlessly. Subsequently, I match the end events to their corresponding start events through a Watcher, using bulk index updates with the "_doc": idiom. Due to the joined nature of the index, it is necessary to specify the routing in the index action to ensure that a parent and its child are co-located on the same shard.
One approach I tried was to add a "_routing" field alongside the "_id" field to the documents within the _doc array. However, this results in errors such as:
"org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [_routing] of type [_routing] in document with id '8d363b90a99c5a9232f20ab08d5eecb636aeee30'. Preview of field's value: 'be28b73764a6ea562ead49b3153a67568d8e7c08'"
The standard method to route a document in a Watcher index action is unclear, and it is uncertain if this use case has been implemented. Guidance on the correct approach is sought.
Below is the template for the index for reference. Additional details and specific examples can be provided upon request.
{
"index_patterns": ["batch-active-index-*"],
"priority": 200,
"template": {
"settings": {
"index.number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "batch-active",
"index.lifecycle.rollover_alias": "batch-active-index"
},
"mappings": {
"properties": {
"@timestamp": {"type": "date"},
"realm": {"type": "keyword"},
"batch_class": {"type": "keyword"},
"event_kind": {"type": "keyword"},
"event_name": {"type": "text"},
"event_id": {"type": "keyword"},
"matched": {"type": "boolean"},
"log_timestamp": {"type": "date"},
"start_millis": {"type": "unsigned_long"},
"end_millis": {"type": "unsigned_long"},
"duration": {"type": "long"},
"uid": {"type": "keyword"},
"start_uid": {"type": "keyword"},
"start_end": {"type": "join", "relations": {"start": "end"}}
}
}
}
}