ES taking long time to refresh

[Yogesh_BG]

Hi

I am using ES 2.4

I have two nodes one master and one data node. Pumping the traffic heavily. each record is of size 1.6kb and around 6000 such records per sec. After some time when we retrieve the docs it gives old records(3min old). after around 4 min everything gets settled and able to get current second record. This cycle repeats.

In client API initially we had both master and data node ips given as transport address, everything used t o work fine and used to get record real time time. As data used to get pass through master node also OOM used to happen at master node and decided to include only data node as part of transport client address. then started observing above behaviour. Can you pls guide me what factor is affecting this? refresh interval initially was 30sec and i tried changing to 3sec and it didnt work
but every 4min it settles and lag starts happening for next 4min.

[Christian_Dahlqvist]

@Yogesh_BG

I am using ES 2.4

That is pretty old. I would recommend upgrading.

@Yogesh_BG

I have two nodes one master and one data node

There is very little point in having a dedicated master node for a cluster this size. You have more that can go wrong for no benefit in my opinion. I think you would be better off running just a single node with default configuration.

With respect to the merging activity, this depends on what type of storage you are using and how your node and indices are configured. If storage is slow, you can look at this guide for Elasticsearch 2.x to optimise performance.