Native script inconsistent (not found)

I was given a development cluster with 2 nodes, there're about 700 indexes there. I installed a plugin and it works very inconsistently. Sometimes it works normally, sometimes when I modify the code a bit, I can build and install the new code successfully but when I run query, it throws an error that "Native script [abc] not found". Then I modify the code and install again, it works again. Note again that there was no error every time I install the plugin. And of course I did restart elastic every time I install the plugin. Can anyone help pinpoint the possible reason for that behavior? Thanks a lot.

Btw I'm using Elasticsearch 1.7 and the cluster's health status is always yellow!

You haven't given us much to really go on.

What errors did you get, the exact error?
What does the script look like?
What does your ES config look like?

Hi Mark, here's the cluster's health:
{
"cluster_name" : "searchme",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 3711,
"active_shards" : 3711,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 3711,
"number_of_pending_tasks" : 0
}

Currently I am testing the Cosine Similarity Script by Imotov on github:

And here's the error I got:

nested: QueryParsingException[[example_index] script_score the script could not be loaded]; nested: ElasticsearchIllegalArgumentException[Native script [cosine_sim_script_score] not found]

That's a pretty crazy number of shards you have, why so many? And why are so many unassigned?

Anyway, ES says it cannot find the script, so where did you place it?

It's a development cluster so quite a few people used/are using it to test and might forget to delete their data in the end.

The plugin is put in elastic/plugins along with other plugins. Like I said, sometimes it was able to find the script, but sometimes just wasn't.

Btw, if I already have the plugin installed, then I modify the code a little bit, rebuild, then reinstall, the plugin name still stays the same, do I have to restart Elasticsearch? Currently I restart elastic every time I install even though some time the change is very minor.