Hello,
in our ELK cluster we have various systems sending their logs to ES. I the Logstash configuration we enforced that only events with some mandatory fileds (that allow to identify the source of an event) are passed to the ES index.
One of this fields is called "server".
I want to automatically create aliases named after server names, which will do nothing but filter the logs coming form a particular server
"filter": {
"bool": {
"must": [
{
"term": {
"server": "sample_server_name"
}
} ]
To do it, I wrote a small Python script, which I plan to run on a daily basis:
from elasticsearch import Elasticsearch
es = Elasticsearch([{'host': 'localhost', 'port': 9200}])
res = es.search(index="logstash-*", body={"size":0,"aggs":{"servers":{"terms":{"field":"server.raw"}}}})
for server in res['aggregations']['servers']['buckets']:
if server["key"] and not server["key"].isspace():
print("Creating alias for server: {}".format(server["key"]))
es.indices.update_aliases(body={"actions":[{"add":{"index":"logstash-*","alias":server["key"],"filter":{"bool":{"must":[{"term":{"server":server["key"]}}],"must_not":[]}}}}]})
First question is if there is a simpler solution to achieve this.
Second question is, how to add these aliases as new index patterns in Kibana? Do I have to simulate from my script the sequence of GETs and POSTs which are normally performed when an index pattern is created?
http://localhost:5601/elasticsearch/.kibana/index-pattern/sandbox?op_type=create
http://localhost:5601/elasticsearch/.kibana/_refresh
http://localhost:5601/elasticsearch/.kibana/index-pattern/_search?fields=
http://localhost:5601/elasticsearch/_mget?timeout=0&ignore_unavailable=true&preference=1448361937512
http://localhost:5601/elasticsearch/sandbox/_mapping/field/*?ignore_unavailable=false&allow_no_indices=false&include_defaults=true&_=1448361990053
http://localhost:5601/elasticsearch/.kibana/index-pattern/sandbox