Problems with java api when upgrading elasticsearch

aldm · September 9, 2013, 3:40pm

Hi,

we upgraded es version from 0.20.2 to 0.90.3 and we have some troubles.
First of all, we had trouble getting terms facets using
class InternalStringTermsFacet.

'term' os returning ByteText object in new version and was returning String
in old version and we had one dependency in our application which created
problem.

I fixed taht but now there is another problem, with bulk indexing using
bulk processor.
For some reason, processor is not indexing all documents.

We had is out logs

BulkProcessor: executing bulk 1 containing 73 actions

But only 28 documents are indexed in this action. What could be problem
here?
Same code works for previous versions.

Is there any changes in bulk processor or bulk indexing in new versions (I
didn't find it on elasticsearch.org).

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

javanna · September 9, 2013, 4:31pm

Hi,
I don't recall any change that might have modified the way BulkProcessor
class works, and if it says "executing 73 actions", that should be the
number of executed actions.
Did you have only index requests in that specific bulk? How do you get back
28 documents? Using the search api? Do you do it in a test or manually? Did
you make sure the index is refreshed if you are using the search api?

On Monday, September 9, 2013 5:40:55 PM UTC+2, aldm wrote:

Hi,

we upgraded es version from 0.20.2 to 0.90.3 and we have some troubles.
First of all, we had trouble getting terms facets using
class InternalStringTermsFacet.

'term' os returning ByteText object in new version and was returning
String in old version and we had one dependency in our application which
created problem.

I fixed taht but now there is another problem, with bulk indexing using
bulk processor.
For some reason, processor is not indexing all documents.

We had is out logs

BulkProcessor: executing bulk 1 containing 73 actions

But only 28 documents are indexed in this action. What could be problem
here?
Same code works for previous versions.

Is there any changes in bulk processor or bulk indexing in new versions (I
didn't find it on elasticsearch.org).

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

brian_yoder · September 9, 2013, 8:29pm

I do not know of any changes related to the class you mentioned. But here
is a diff of the changes that were typical when I migrated from 0.20.4 to
0.90.3. The biggest effort involved combing through the classes to find out
what replaced the AbstractFacetBuilder, but once I discovered it was the
more plainly-named FacetBuilder I was all set; the changes were relatively
small:

@@ -53,7 +53,7 @@
*/
public ComboTerm(TermsFacet.Entry entry, Pattern splitter)
{

 hierarchy = splitter.split(entry.getTerm());

 hierarchy = splitter.split(entry.getTerm().toString());
 count = entry.getCount();
 level = 0;
 if (hierarchy.length > level)

@@ -401,7 +401,7 @@
}

@Override

public AbstractFacetBuilder getFacetRequest()

public FacetBuilder getFacetRequest()
{
return request;
}
@@ -409,7 +409,7 @@
@Override
public void storeFacetResponse(SearchResponse sr)
{

response = (TermsFacet) sr.facets().facetsAsMap().get(facetName);

response = (TermsFacet) sr.getFacets().facetsAsMap().get(facetName);

I didn't see any changes needed to my BulkRequestBuilder usage.

Brian

On Monday, September 9, 2013 11:40:55 AM UTC-4, aldm wrote:

Hi,

we upgraded es version from 0.20.2 to 0.90.3 and we have some troubles.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

aldm · September 10, 2013, 2:22pm

Thanks for replies,

I get 28 documents in index querying elasticsearch directly using curl. It
gave 28 documents as result. And all requests in bulk is indexings. I will
check code again to see what could be wrong, but as I said, same code works
on 0.20.2.

Just for note, one of Java Client: Renamed
IndicesAdminClient.existsAliases() to IndicesAdminClient.aliasesExist
If you know for some other changes in api, please write here so anybody can
see what could be problem when upgrading.

On Monday, September 9, 2013 5:40:55 PM UTC+2, aldm wrote:

Hi,

we upgraded es version from 0.20.2 to 0.90.3 and we have some troubles.
First of all, we had trouble getting terms facets using
class InternalStringTermsFacet.

'term' os returning ByteText object in new version and was returning
String in old version and we had one dependency in our application which
created problem.

I fixed taht but now there is another problem, with bulk indexing using
bulk processor.
For some reason, processor is not indexing all documents.

We had is out logs

BulkProcessor: executing bulk 1 containing 73 actions

But only 28 documents are indexed in this action. What could be problem
here?
Same code works for previous versions.

Is there any changes in bulk processor or bulk indexing in new versions (I
didn't find it on elasticsearch.org).

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · September 10, 2013, 10:16pm

It would be helpful if you could post your bulk indexing code somewhere so
we can understand your code better, step by step. BulkProcessor has not
changed, but behind the indexing call backs, there may be some method APIs
where the naming had changed.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

aldm · September 13, 2013, 2:40pm

Sorry for not replying for few days.
I was working on other things.

It seems problem is with realiasing indices. In this function we first
create index, then alias this index and delete all existing indices with
same alias (because we can have index with same alias) and then activating
index and set it's settings.
After this bulk operation is executed.

So the only thing that could be problem (at least in my mind :)) is that
alias operation is executed too slowly (async operation) so some documents
are indexed and then in the middle of the job realiasing is executed (which
includes deleting indices with same alias and after that indexing another
docs).

I changed this to work with changing order of bulk indexing and realiasing,
so noew I first create index and index all docs with bulk and after this
add alias to documents and eventually delete all other indices (beside
newly created) with same aliases as new index.

Question is:

Is there any change in executing or performance of alias operation in new
elasticsearch ?
I'm interested what caused this problem...

On Monday, September 9, 2013 5:40:55 PM UTC+2, aldm wrote:

Hi,

we upgraded es version from 0.20.2 to 0.90.3 and we have some troubles.
First of all, we had trouble getting terms facets using
class InternalStringTermsFacet.

'term' os returning ByteText object in new version and was returning
String in old version and we had one dependency in our application which
created problem.

I fixed taht but now there is another problem, with bulk indexing using
bulk processor.
For some reason, processor is not indexing all documents.

We had is out logs

BulkProcessor: executing bulk 1 containing 73 actions

But only 28 documents are indexed in this action. What could be problem
here?
Same code works for previous versions.

Is there any changes in bulk processor or bulk indexing in new versions (I
didn't find it on elasticsearch.org).

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

javanna · September 16, 2013, 8:50am

Hi,
I'm not sure I understood what you're actually doing. Maybe if you can
share some code it would help.

Anyway, regarding aliases, they are part of the cluster state, which gets
changed in the master node (e.g. when you add an alias) and gets propagated
to the other nodes. To make sure all the nodes hold the up-to-date cluster
state after you made a change, the best thing to do is to to check if the
response was acknowledged. If it is, it means you can go ahead, all nodes
will know that your new alias maps to some index.

Cheers
Luca

On Fri, Sep 13, 2013 at 4:40 PM, aldm amer.zildzic@gmail.com wrote:

Sorry for not replying for few days.
I was working on other things.

It seems problem is with realiasing indices. In this function we first
create index, then alias this index and delete all existing indices with
same alias (because we can have index with same alias) and then activating
index and set it's settings.
After this bulk operation is executed.

So the only thing that could be problem (at least in my mind :)) is that
alias operation is executed too slowly (async operation) so some documents
are indexed and then in the middle of the job realiasing is executed (which
includes deleting indices with same alias and after that indexing another
docs).

I changed this to work with changing order of bulk indexing and
realiasing, so noew I first create index and index all docs with bulk and
after this add alias to documents and eventually delete all other indices
(beside newly created) with same aliases as new index.

Question is:

Is there any change in executing or performance of alias operation in new
elasticsearch ?
I'm interested what caused this problem...

On Monday, September 9, 2013 5:40:55 PM UTC+2, aldm wrote:

Hi,

we upgraded es version from 0.20.2 to 0.90.3 and we have some troubles.
First of all, we had trouble getting terms facets using class **
InternalStringTermsFacet.

'term' os returning ByteText object in new version and was returning
String in old version and we had one dependency in our application which
created problem.

I fixed taht but now there is another problem, with bulk indexing using
bulk processor.
For some reason, processor is not indexing all documents.

We had is out logs

BulkProcessor: executing bulk 1 containing 73 actions

But only 28 documents are indexed in this action. What could be problem
here?
Same code works for previous versions.

Is there any changes in bulk processor or bulk indexing in new versions
(I didn't find it on elasticsearch.org).

Thanks in advance

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_EyS-YTN45k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

aldm · September 17, 2013, 3:27pm

Hi, thanks for replies.

I solved this problem few days ago with following change: adding alias to
index and deleting and removing all indices with same alias now taking
place after bulk indexing operation. Now I'm passing original index name to
bulk processor and it indexes document with original index name (not
alias), and after bulk is executed all indices with same alias are deleted
and alias is given to new index. This way everything works fine.

On Monday, September 9, 2013 5:40:55 PM UTC+2, aldm wrote:

Hi,

we upgraded es version from 0.20.2 to 0.90.3 and we have some troubles.
First of all, we had trouble getting terms facets using
class InternalStringTermsFacet.

'term' os returning ByteText object in new version and was returning
String in old version and we had one dependency in our application which
created problem.

I fixed taht but now there is another problem, with bulk indexing using
bulk processor.
For some reason, processor is not indexing all documents.

We had is out logs

BulkProcessor: executing bulk 1 containing 73 actions

But only 28 documents are indexed in this action. What could be problem
here?
Same code works for previous versions.

Is there any changes in bulk processor or bulk indexing in new versions (I
didn't find it on elasticsearch.org).

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

aldm · September 17, 2013, 3:29pm

There is another question.

Is there any fix for sorting on non mapped field in elasticsearch? In older
versions it raises mappingnotfound exception and it seems nothing is done
with that ?
Is there any way to ignore non mapped field in sort and execute query like
there is no sort?

On Monday, September 9, 2013 5:40:55 PM UTC+2, aldm wrote:

Hi,

we upgraded es version from 0.20.2 to 0.90.3 and we have some troubles.
First of all, we had trouble getting terms facets using
class InternalStringTermsFacet.

'term' os returning ByteText object in new version and was returning
String in old version and we had one dependency in our application which
created problem.

I fixed taht but now there is another problem, with bulk indexing using
bulk processor.
For some reason, processor is not indexing all documents.

We had is out logs

BulkProcessor: executing bulk 1 containing 73 actions

But only 28 documents are indexed in this action. What could be problem
here?
Same code works for previous versions.

Is there any changes in bulk processor or bulk indexing in new versions (I
didn't find it on elasticsearch.org).

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

javanna · September 17, 2013, 4:46pm

Hi, glad to hear you've solved.

Regarding your new question, have a look at the ignore_unmapped option. You
can read more about it in the refernce page for sorting:
Elasticsearch Platform — Find real-time answers at scale | Elastic .

Cheers
Luca

On Tue, Sep 17, 2013 at 5:29 PM, aldm amer.zildzic@gmail.com wrote:

There is another question.

Is there any fix for sorting on non mapped field in elasticsearch? In
older versions it raises mappingnotfound exception and it seems nothing is
done with that ?
Is there any way to ignore non mapped field in sort and execute query like
there is no sort?

On Monday, September 9, 2013 5:40:55 PM UTC+2, aldm wrote:

Hi,

we upgraded es version from 0.20.2 to 0.90.3 and we have some troubles.
First of all, we had trouble getting terms facets using class **
InternalStringTermsFacet.

'term' os returning ByteText object in new version and was returning
String in old version and we had one dependency in our application which
created problem.

I fixed taht but now there is another problem, with bulk indexing using
bulk processor.
For some reason, processor is not indexing all documents.

We had is out logs

BulkProcessor: executing bulk 1 containing 73 actions

But only 28 documents are indexed in this action. What could be problem
here?
Same code works for previous versions.

Is there any changes in bulk processor or bulk indexing in new versions
(I didn't find it on elasticsearch.org).

Thanks in advance

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_EyS-YTN45k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.