Dynamically adding new fields from a root mapper

Elastic Stack Elasticsearch
1

Hi,

I am creating a plugin that analyzes document being indexed and based on
the analysis it adds new fields to it. It is done from a root mapper and it
works similarly as the attachment mapper.

I need to use the root mapper because I need the whole document for
analysis, not individual fields. I also don't have a fixed predefined set
of fields that can be generated.

I'd like to ask what is the correct way of updating the mapping in this
scenario.

If I don't update the mapping then I'm getting a NPE:

Caused by: java.lang.NullPointerException

org.elasticsearch.index.fieldvisitor.FieldsVisitor.postProcess(FieldsVisitor.java:70)

org.elasticsearch.index.get.ShardGetService.innerGetLoadFromStoredFields(ShardGetService.java:333)

org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:212)

org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:106)

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:109)

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:43)

That's because there is no FieldMapper for the newly added field.

I can add the field mapper this
way: context.docMapper().addFieldMappers(ImmutableSet.of(field.mapper)) and
then it seems to work. No NPE, no warnings in the log.

However, to create the field mapper, I need a Mapper.BuilderContext, so
that I can do e.g. field.mapper
= MapperBuilders.stringField(field.name()).build(builderContext));

This seems to be a problem because the BuilderContext is available only
from my root mapper's Builder. I pass the BuilderContext to my root mapper
which then uses it to create the field mappers. But this seems hacky/wrong.

What would be the correct approach? Will the mapping be otherwise updated
somehow by Elasticsearch in this case? Or how exactly should I go about
implementing my root mapper's toXContent() method?

You can see the mapper
here: https://github.com/sindicetech/smartindexing/blob/issue-1/cogito-elasticsearch/src/main/java/net/expertsystem/connector/elasticsearch/index/CogitoMapper.java
(work in progress/playground).

Thanks,

Jakub

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c3722146-2929-4a3d-b82e-07a54ddef6d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hi,

I'm wondering if anyone could provide some hints... :slight_smile:

Pointers where to learn more about Mappers, FieldMappers, RootMappers,
adding new fields, how mapping is updated and its relation to toXContent()
of mappers would be appreciated too. Also paths vs. full paths vs.
multi_field / fields, etc.

Cheers,

Jakub

On Thursday, August 21, 2014 12:47:10 PM UTC+2, Jakub Kotowski wrote:

Hi,

I am creating a plugin that analyzes document being indexed and based on
the analysis it adds new fields to it. It is done from a root mapper and it
works similarly as the attachment mapper.

I need to use the root mapper because I need the whole document for
analysis, not individual fields. I also don't have a fixed predefined set
of fields that can be generated.

I'd like to ask what is the correct way of updating the mapping in this
scenario.

If I don't update the mapping then I'm getting a NPE:

Caused by: java.lang.NullPointerException

org.elasticsearch.index.fieldvisitor.FieldsVisitor.postProcess(FieldsVisitor.java:70)

org.elasticsearch.index.get.ShardGetService.innerGetLoadFromStoredFields(ShardGetService.java:333)

org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:212)

org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:106)

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:109)

org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:43)

That's because there is no FieldMapper for the newly added field.

I can add the field mapper this
way: context.docMapper().addFieldMappers(ImmutableSet.of(field.mapper)) and
then it seems to work. No NPE, no warnings in the log.

However, to create the field mapper, I need a Mapper.BuilderContext, so
that I can do e.g. field.mapper = MapperBuilders.stringField(field.name
()).build(builderContext));

This seems to be a problem because the BuilderContext is available only
from my root mapper's Builder. I pass the BuilderContext to my root mapper
which then uses it to create the field mappers. But this seems hacky/wrong.

What would be the correct approach? Will the mapping be otherwise updated
somehow by Elasticsearch in this case? Or how exactly should I go about
implementing my root mapper's toXContent() method?

You can see the mapper here:
https://github.com/sindicetech/smartindexing/blob/issue-1/cogito-elasticsearch/src/main/java/net/expertsystem/connector/elasticsearch/index/CogitoMapper.java
(work in progress/playground).

Thanks,

Jakub

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/48f1f2a5-95d2-4182-833a-135798e3f44a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.