Using JavaScript in Elasticsearch

[rezaei121 (Ehsan Rezaei)]

hi,
i want to use javaScript in elasticsearch
https://www.elastic.co/guide/en/elasticsearch/plugins/5.1/lang-javascript-usage.html#lang-javascript-file

but I have a problem...
I need to use javascript library. for example:

require('msgpack.min.js');
ctx._source.num = msgpack.encode({foo: "bar"});

not working
how can this be possible?

[nik9000 (Nik Everett)]

require hasn't been supported and lang-javascript is deprecated in 5.x and will be removed in 6.0 so I don't expect it'll ever be supported. Since none of the languages you can embed in the JVM have working of security or ability to limit execute time we built our own. At this point we're advising everyone to use lang-painless because it has those things. It is still new and rough around the edges but it isn't fundamentally broken for our use case.

[rezaei121 (Ehsan Rezaei)]

Thank you

MessagePack is supported by over 50 programming languages and environments.
http://msgpack.org/index.html

Is it possible to use msgpack in Painless scripting?
fore example:
{
"script" : {
"lang": "painless",
"inline": "MessagePack msgpack = new MessagePack(); ctx._source= msgpack .unpack(ctx._source.num)",
"params" : {
"factor" : 4
}
}
}

[nik9000 (Nik Everett)]

Possible but fairly unlikely. For the most part we recommend folks store data in JSON so it works well with Elasticsearch. If you don't want fields to be indexed for search you can set "index": false on them and you can turn off doc values with "doc_values": false. At that point Elasticsearch pretty much just stores the field in the _source json and ignores it. I expect MessagePack would save a bit of space but not much on disk because _source is compressed in blocks with other source documents.

[jprante (Jörg Prante)]

What I try to read through the lines is that your intention is to implement a JSON binary compression mechanism.

Note that this is not possible by scripting. You'd have to dig deeper. Technically, Elasticsearch already uses JSON binary compression under the hood https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/common/xcontent/smile/SmileXContent.java The use of SMILE is internal and triggered automatically.

I understand that many HTTP client do not/can not make use of SMILE because it's a Java API feature. To add blindly another binary compression on top of SMILE just for docs with inline scripts adds significant load on Elasticsearch cluster nodes.

Instead, you would have to examine the XContent API and replace SMILE by your codec, so all server nodes and client nodes could speak the same serialization language. This is even more than an ordinary Elasticsearch plugin could do, you'd have to exchange the transport layer. I hardly can see any justification of such a replacement.

For a comparison of SMILE against MessagePack, see https://www.quora.com/How-does-MessagePack-compare-to-Jacksons-Smile-JSON-encoding/answer/Tatu-Saloranta

Note, in that context, that Elasticsearch did once support Thrift as an optional node protocol beside compressed JSON https://github.com/elastic/elasticsearch-transport-thrift but that project had been stopped because benefit is too low.