Hey @benvantende,
Changing the mapping would mean invalidating already indexed documents. Instead, you should create a new index with the correct mappings and reindex your data into that index.
Once you have new index, you may need to recreate index pattern (if it doesn't cover new index name). And if you do so you'll just need to update index pattern id your existing visualizations are based on, see Visualisation don't work even after recreating index templates for details. Try with one vis first, see how well it works and then move on and update the rest.
Let me know if you have any questions.
Best,
Oleg