Strange behaviour on cluster of two nodes on aws

Hi,

We have cluster of two machines and they are recognising each other perfectly. Where one is primary node and other is working as a replica. Health check is also Green.

Task scenario is.

  1. Pushing new document on primary.
  2. after pushing, same thread is calculating one more document and try to add one more metadata using update IP in the same document.
  3. Getting Mapping parser exception.

I investigated further, and found while pushing this new metadata in the document, it is not finding same document. Well than i put a sleep of 1 sec between (1 and 2) step. And it starts working.

I doubt, since i added replica and may be because of network latency it is taking time to get publish the main document and script is trying to add metadata for the document which is not existing yet.

i also tried refresh API between setup 1 and 2. But it did not work.

one more strage thing. Some times it starts working sometimes is starts generating mapper parser exception.

I believe i am missing something. Can someone put me in right direction to get this fix.

Thank you.

Hello Terry,

Sorry if I point out the obvious, but I think it would be a good idea to
gather all the document information and insert everything at once.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

On Sat, Nov 24, 2012 at 9:18 AM, Terry tarun@izap.in wrote:

Hi,

We have cluster of two machines and they are recognising each other
perfectly. Where one is primary node and other is working as a replica.
Health check is also Green.

Task scenario is.

  1. Pushing new document on primary.
  2. after pushing, same thread is calculating one more document and try to
    add one more metadata using update IP in the same document.
  3. Getting Mapping parser exception.

I investigated further, and found while pushing this new metadata in the
document, it is not finding same document. Well than i put a sleep of 1 sec
between (1 and 2) step. And it starts working.

I doubt, since i added replica and may be because of network latency it is
taking time to get publish the main document and script is trying to add
metadata for the document which is not existing yet.

i also tried refresh API between setup 1 and 2. But it did not work.

one more strage thing. Some times it starts working sometimes is starts
generating mapper parser exception.

I believe i am missing something. Can someone put me in right direction to
get this fix.

Thank you.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Strange-behaviour-on-cluster-of-two-nodes-on-aws-tp4025951.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--

--

Hi Terry,

It looks strange. Could you gist a curl recreation?
It will help us to reproduce your issue.

IMHO it should work, especialy if you have added the refresh Call.
That said, perhaps there is a latency that put your cluster in Red Health just after the first doc insertion.
Did you try to create your index first? Then wait for Yellow status and then send your first doc and its update?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 24 nov. 2012 à 08:18, Terry tarun@izap.in a écrit :

Hi,

We have cluster of two machines and they are recognising each other
perfectly. Where one is primary node and other is working as a replica.
Health check is also Green.

Task scenario is.

  1. Pushing new document on primary.
  2. after pushing, same thread is calculating one more document and try to
    add one more metadata using update IP in the same document.
  3. Getting Mapping parser exception.

I investigated further, and found while pushing this new metadata in the
document, it is not finding same document. Well than i put a sleep of 1 sec
between (1 and 2) step. And it starts working.

I doubt, since i added replica and may be because of network latency it is
taking time to get publish the main document and script is trying to add
metadata for the document which is not existing yet.

i also tried refresh API between setup 1 and 2. But it did not work.

one more strage thing. Some times it starts working sometimes is starts
generating mapper parser exception.

I believe i am missing something. Can someone put me in right direction to
get this fix.

Thank you.

--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Strange-behaviour-on-cluster-of-two-nodes-on-aws-tp4025951.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--

--

Hi Guys,

Thank you for your reply. Actually it was hurry to deliver a product. So i made something to get this done. but i would like to share what i observe. Because i still need the solution of this problem.

EC2 Layout.

1 primary Es Server (nginx + elasticsearch)
1 Replica Es Server (nginx + elasticsearch)

i want to restrict elasticsearch access by username, password. so that's why i used Nginx which was supposed to be proxy for Es access.

Since php was interfacing elasticsearch via Nginx. And somehow it was creating this problem. for the timebeing, i stopped using nginx and directly using elasticsearch. It is working fine.

Is there any clue i could manage to get this work?

Thank you.

So you have probably misconfigured your nginx instances.

Why do you have two nginx instances? Why do you call one of your es as a primary and the other one as secondary?

Can't you build a cluster of Es nodes and a single Nginx on top of them?
Each ES must be able to talk with each other on 9300-9399.

Not sure it helps, but I'm pretty sure that you should double-check your nginx settings.

HTH

David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 23 déc. 2012 à 19:11, Terry tarun@izap.in a écrit :

Hi Guys,

Thank you for your reply. Actually it was hurry to deliver a product. So i
made something to get this done. but i would like to share what i observe.
Because i still need the solution of this problem.

EC2 Layout.

1 primary Es Server (nginx + elasticsearch)
1 Replica Es Server (nginx + elasticsearch)

i want to restrict elasticsearch access by username, password. so that's why
i used Nginx which was supposed to be proxy for Es access.

Since php was interfacing elasticsearch via Nginx. And somehow it was
creating this problem. for the timebeing, i stopped using nginx and directly
using elasticsearch. It is working fine.

Is there any clue i could manage to get this work?

Thank you.

--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Strange-behaviour-on-cluster-of-two-nodes-on-aws-tp4025951p4027376.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--

--

Thank you David, Yes, you are right. I can do that as long as we have two servers. one primary and one replica.

But we would add more servers in future. And we do need to implement "round ribbon" in future. where our script is suppose to communicate directly with every elasticsearch node. It will help us to distribute the query load on multiple servers in a single thread.

So to have password protected access. we do need to put nigix on every server. Not sure if there is any better solution.

Thank you.

It would be great if you could provide a repro for this issue.

I am not quite sure why you are calling one server "primary" and another
one "replica" since "primary" and "replica" status is assign on per shard
basis. So, unless you have one shard per service, "primary" and "replica"
are not really applicable on the server level. By the way, why does your
script need to communicate directly with every elasticsearch node?

Igor

On Thursday, December 27, 2012 3:06:01 PM UTC-5, Rimpy wrote:

Thank you David, Yes, you are right. I can do that as long as we have two
servers. one primary and one replica.

But we would add more servers in future. And we do need to implement
"round
ribbon" in future. where our script is suppose to communicate directly
with
every elasticsearch node. It will help us to distribute the query load on
multiple servers in a single thread.

So to have password protected access. we do need to put nigix on every
server. Not sure if there is any better solution.

Thank you.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Strange-behaviour-on-cluster-of-two-nodes-on-aws-tp4025951p4027492.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--