How to split array or string with "Ingest Node Pipeline"

I'm new to ECE and have a filebeat pipeline which sends data to ECE.
This is the format of the message field which I have in ECE (separator would be -> \t):

2021-06-25T13:08:39.559+0200 INFO    12409 com.l7tech.traffic: 2021-06-25T11:08:39.554Z\tno_endpoint\tmonitoring\tno_operation\tno_consumer_timestamp\tCOM-556\t0\ttrue\t0\tno_consumer_certificate\t13\tnon-routing\t\t10.183.135.249\tundefined\tundefined\tundefined\tundefined\tundefined\tundefined\tundefined\t/monitoring\tundefined

The aim is to separate the values by \t and add them to a new field.

I'm able to get the first value (date) with the SPLIT processor

regex *([\s]+)(.*)

-> the regex which should work is ([^\s]+) but I had to add (.*) at the end ... strange. I have not yet understood how the regex processor is working in the SPLIT processor.

But then I'm now struggeling with the other 22 parts.
I tried to first move to a sting value with:
SET processor, value={{field_message}}, which leads to a string:

{0=2021-06-25T13:08:39.559+0200 INFO    12409 com.l7tech.traffic: 2021-06-25T11:08:39.554Z, 1=no_endpoint, 2=monitoring, 3=no_operation, 4=no_consumer_timestamp, 5=COM-556, 6=0, 7=true, 8=0, 9=no_consumer_certificate, 10=13, 11=non-routing, 12=, 13=, 14=undefined, 15=undefined, 16=undefined, 17=undefined, 18=undefined, 19=undefined, 20=undefined, 21=/monitoring, 22=undefined}

Then I tried to separate the new string value with separate SPLIT processors for each value.
The separator regex is working on with the string value above.


But this is not working in the Ingest Node Pipeline.

Is there a better way to store all the individual values from the input to new fields ?

Would be great if someone can put me in the right direction.

the split processor sounds right. Can you share a fully reproducible example using the Simulate Ingest Pipeline API so that others can try to build the same?

Thank you!

Thanks for the response.
In the meantime I solved my issue by using a "Script" processor

int tempStart = ctx.tempFormatted.indexOf("19=",0);
int tempEnd = ctx.tempFormatted.indexOf("20=",0);
ctx.providerMessageId = ctx.tempFormatted.substring(tempStart+3,tempEnd-2);
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.