Transforms weird situation

[fadjar340 (Fadjar Tandabawana)]

Hi experts,

I have several transforms for pivot, this is the situation that make me confused:

1. The transform works as expected
2. The Documents Processes and documents indexed increased as expected with around 500 documents per sec
3. the weird thing is when I checked in discovery of the target index, the count didn't increased as the rate of the documents indexed.

Any information about this situation?
Please brief me what happened in the transforms feature...

ELK stack 7.6.2, BTW

Regards,
Fadjar Tandabawana

[Hendrik_Muhs]

Hi,

without further, more detailed information this is hard to answer. Can you post your transform configuration and some example data. Are you using a continuous transform or batch?

In case of a continuous transform, transform updates(overwrites) existing documents. This might explain that transform stats reports increasing documents indexed, while at the same time the document count is not increasing.

I can give a better answer once I have more details.

[fadjar340 (Fadjar Tandabawana)]

I got the situation, I use continues transform, so, its expected behavior since the transforms overwrites the existing document.

Thank you for the explanation...

[Hendrik_Muhs]

Thanks for the feedback.

Note that in case you use date_histogram as group_by in your transform, its advised to upgrade to 7.7, it reduces the load significantly.

[fadjar340 (Fadjar Tandabawana)]

I know, in other cluster I use 7.7.x and it's significant result, but I have little bit issue using 7.7, because in heartbeat index, the 7.6 use rtt.us as result of ICMP rtt, but in 7.7, it use icmp.rtt.us for the same function. It will break the dashboard, BTW...
And also break the SLA that already settled, if you have any solution for this, I'm happy to know.

Regards,
Fadjar Tandabawana

[Hendrik_Muhs]

Sorry I can't answer that question, I recommend to open a new topic under beats, if you haven't already.