I'm new to the ELK stack, and am hoping I'm not trying to re-invent the wheel. Steep learning curve at the moment and am just after some help and guidance really.
I'm using FileBeat to send logs out from our Kubernetes cluster. I was originally outputting the logs directly into elasticsearch, but found that having everything outputting into one single index wasnt really working how we wanted.
After some research and the following post which I found really usfull I now send the logs through filebeat rather than elasticsearch, this is working nicely.
But.... The mapping for the indexes in elasticsearch are way more heavyweight than what we need.
Ive followed this guide around changing a mapping, but when I repoint the alias using the "atomic" step, although the mapping that Ive created is correct, none of the documents from the original index are present in the new index with simpler map.
When I use the re-index api to move documents in, the mapping in the new index is ignored and all the original indexes are copying over.
Basically all I really need to achieve is that when we use the "Discover link in Kibana, we dont need to see all the kubernetes fieds and other metadata, we are just wanting to see info about the source of the request, and the "message" fields from the mapping.