I'm doing exactly the same and it works for me, but you must remember to export PORT in the environment of the user that starts Elasticsearch which may be a different user from the one you're logged in as.
One way around this problem is to use a wrapper script to set the environment variable and start Elasticsearch. For instance a start-es.sh with this content:
export PORT=9255 # http.port
export TCP_PORT=9355 # transport.tcp.port
#now start Elasticsearch as a daemon process:
/basedir/es6.5.4/bin/elasticsearch -d -p /basedir/pids/es.pid 2>/basedir/logs/es.stderr
Then you just run ./start.es.sh when you want to start Elasticsearch with the right PORT setting.
Of course, you can then expand the script to add even more environment variables, for instance node and cluster name, zen discovery hosts, data path etc making elasticsearch.yml completely generic across all nodes and even across clusters, making maintenance a breeze.