If you want to run X-Pack on Alpine Linux without machine learning then you need to delete the machine learning native code after installing X-Pack, for example rm -rf $ES_HOME/plugins/x-pack/platform/linux-x86_64. Then also set xpack.ml.enabled: false in elasticsearch.yml.
Obviously this is an unsupported configuration but it should get you back to where you were with 5.3.