Elasticsearch 1.3 Transform Scripts

mattweber · July 25, 2014, 1:22am

Just wanted to bring attention to the new and very useful transform
scripts that were introduced in elasticsearch 1.3 [1]. This feature allows
you to manipulate the source BEFORE it is indexed so you can do things like
add/remove fields, change field values, etc. Groovy scripts will be the
default, but you can write native transform scripts as well [2].

[1]
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-transform.html#mapping-transform
[2] https://github.com/imotov/elasticsearch-native-script-example/pull/7

Thanks,
Matt Weber

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5647d22b-9365-449d-922c-b5ed349c7826%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

nik9000 · July 25, 2014, 2:01am

I wanted to do conditional copy_to and Andrian suggested implementing
scripted transforms instead. Much more flexible. They mesh well with the
shift to groovy too because groovy is much more stable. Stable enough to
run on every insert.

I'm glad you are excited by it. It was fun to build and I hope lots of
people enjoy it.

Nik
On Jul 24, 2014 9:22 PM, "Matt Weber" matt@mattweber.org wrote:

Just wanted to bring attention to the new and very useful transform
scripts that were introduced in elasticsearch 1.3 [1]. This feature allows
you to manipulate the source BEFORE it is indexed so you can do things like
add/remove fields, change field values, etc. Groovy scripts will be the
default, but you can write native transform scripts as well [2].

[1]
Elasticsearch Platform — Find real-time answers at scale | Elastic
[2] Add Native Transform Script Example by mattweber · Pull Request #7 · imotov/elasticsearch-native-script-example · GitHub

Thanks,
Matt Weber

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5647d22b-9365-449d-922c-b5ed349c7826%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5647d22b-9365-449d-922c-b5ed349c7826%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2N6KNnJ%3D7kVtiyS3UE45j36ktGxT%3DiDuPsDxMuwKGAPw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

mattweber · July 25, 2014, 3:52am

Yea, I really like it. I have been thinking about the exact same thing for
a while but never had the time to put it together. I do have some things I
would like to add such as the ability to stop a document from being indexed
when the doc has/does not have a specific value. At any rate, great job!

Thanks,
Matt Weber

On Thu, Jul 24, 2014 at 7:01 PM, Nikolas Everett nik9000@gmail.com wrote:

I wanted to do conditional copy_to and Andrian suggested implementing
scripted transforms instead. Much more flexible. They mesh well with the
shift to groovy too because groovy is much more stable. Stable enough to
run on every insert.

I'm glad you are excited by it. It was fun to build and I hope lots of
people enjoy it.

Nik
On Jul 24, 2014 9:22 PM, "Matt Weber" matt@mattweber.org wrote:

Just wanted to bring attention to the new and very useful transform
scripts that were introduced in elasticsearch 1.3 [1]. This feature allows
you to manipulate the source BEFORE it is indexed so you can do things like
add/remove fields, change field values, etc. Groovy scripts will be the
default, but you can write native transform scripts as well [2].

[1]
Elasticsearch Platform — Find real-time answers at scale | Elastic
[2] Add Native Transform Script Example by mattweber · Pull Request #7 · imotov/elasticsearch-native-script-example · GitHub

Thanks,
Matt Weber

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5647d22b-9365-449d-922c-b5ed349c7826%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5647d22b-9365-449d-922c-b5ed349c7826%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2N6KNnJ%3D7kVtiyS3UE45j36ktGxT%3DiDuPsDxMuwKGAPw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2N6KNnJ%3D7kVtiyS3UE45j36ktGxT%3DiDuPsDxMuwKGAPw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJ3KEoDfkUE%2B1%2BPGx-1KW%2BVQFXy2_JgvU9s27krist3uAV_9cQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

jprante · July 25, 2014, 8:09am

Yeah, I use Groovy for input data transformations for years, so I'm very
enthusiastic about ES embracing Groovy as default.

I use Groovy scripts before the source is passed to the bulk client, in a
key/value stream processing.

It's good to see transformation machinery in the ES mapping, but I do also
off-line transformations, without ES, i.e. converting from file to file ...

Jörg

On Fri, Jul 25, 2014 at 5:52 AM, Matt Weber matt.weber@gmail.com wrote:

Yea, I really like it. I have been thinking about the exact same thing
for a while but never had the time to put it together. I do have some
things I would like to add such as the ability to stop a document from
being indexed when the doc has/does not have a specific value. At any
rate, great job!

Thanks,
Matt Weber

On Thu, Jul 24, 2014 at 7:01 PM, Nikolas Everett nik9000@gmail.com
wrote:

I wanted to do conditional copy_to and Andrian suggested implementing
scripted transforms instead. Much more flexible. They mesh well with the
shift to groovy too because groovy is much more stable. Stable enough to
run on every insert.

I'm glad you are excited by it. It was fun to build and I hope lots of
people enjoy it.

Nik
On Jul 24, 2014 9:22 PM, "Matt Weber" matt@mattweber.org wrote:

Just wanted to bring attention to the new and very useful transform
scripts that were introduced in elasticsearch 1.3 [1]. This feature allows
you to manipulate the source BEFORE it is indexed so you can do things like
add/remove fields, change field values, etc. Groovy scripts will be the
default, but you can write native transform scripts as well [2].

[1]
Elasticsearch Platform — Find real-time answers at scale | Elastic
[2] Add Native Transform Script Example by mattweber · Pull Request #7 · imotov/elasticsearch-native-script-example · GitHub

Thanks,
Matt Weber

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5647d22b-9365-449d-922c-b5ed349c7826%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5647d22b-9365-449d-922c-b5ed349c7826%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2N6KNnJ%3D7kVtiyS3UE45j36ktGxT%3DiDuPsDxMuwKGAPw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2N6KNnJ%3D7kVtiyS3UE45j36ktGxT%3DiDuPsDxMuwKGAPw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJ3KEoDfkUE%2B1%2BPGx-1KW%2BVQFXy2_JgvU9s27krist3uAV_9cQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJ3KEoDfkUE%2B1%2BPGx-1KW%2BVQFXy2_JgvU9s27krist3uAV_9cQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFVdAXP1rcbbpGb5ujP--_xfDOYWFLu28u%3DhhQRZ_bS5w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

amrelhagary · August 4, 2014, 11:52am

I have a problem i need to add dynamic fields according to certain fields value , i use groovy in simple case and it worked but the logic become more complicated and i need to separate it in external groovy script file and pass context as parameters but it didn't work and this must in mapping stage .

vineeth_mohan_2 · August 7, 2014, 8:14am

Hi ,

I was looking for this.
Thanks buddy.

Thanks
Vineeth

On Mon, Aug 4, 2014 at 5:22 PM, amrelhagary amr.elhagary@gmail.com wrote:

I have a problem i need to add dynamic fields according to certain fields
value , i use groovy in simple case and it worked but the logic become more
complicated and i need to separate it in external groovy script file and
pass context as parameters but it didn't work and this must in mapping
stage
.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-1-3-Transform-Scripts-tp4060592p4061206.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1407153137385-4061206.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kuyYyoGcV49ZJ21E9%3DPr3nuqCSutsOH0_O3sHHp61ZqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

amrelhagary · August 7, 2014, 10:36am

Hello ,

after several tries finally i found a solution

gist.github.com

https://gist.github.com/amrelhagary/f664c4bc239698a9b293

curls.sh

curl -XDELETE 'localhost:9200/test'

curl -XPOST 'localhost:9200/test' -d '
{
  "mappings": {
    "test_type": {
      "transform": {
        "lang": "groovy",
        "script": "dynamicTransformScript"
      },

This file has been truncated. show original

dynamicTransformScript.groovy

/**
* adding dynamic fields according to field values
*/
if(ctx._source.field_values != ''){
	def dynamicFields = ctx._source.field_values.split(',');
	for(x in dynamicFields){
		ctx._source[x] = x;
	}
}