[Elastic 8.5.2] bug in ingestion pipeline?

[Pierre_LANCASTRE]

Hi,
For the context, I ve got an Elasticagent on my rsyslog which forwards raw udp log to my Elasticstack (running as standalone).

I've defined a K-V action in the pipeline to split the message and create the different key=value pairs. When I take a document in the pipeline menu, i got no error and all awaited variables are there.

But, when coming back to the observability, the logs with the "data" key are not visible. below a log sample :

LOG NOK :
<14>1 2022-12-30T16:16:57+01:00 wab sshproxy 12283 - - [SSH Session] session_id="185639b138cb0a1c00505683f34b" client_ip="192.168.200.1" target_ip="rsyslog.secnumcloud" user="toto" device="RSYSLOG" service="SSH" account="toto" type="KBD_INPUT" data="sudo -i"

LOG OK :
<14>1 2022-12-30T16:16:25+01:00 wab sshproxy 12283 - - [SSH Session] session_id="185639b138cb0a1c00505683f34b" client_ip="192.168.200.2" target_ip="rsyslog.secnumcloud" user="toto" device="RSYSLOG" service="SSH" account="toto" type="SESSION_ESTABLISHED_SUCCESSFULLY"

If I remove my K-V, I can find all the logs.

The K-V options are the following :
Field split : " (?=[a-z_-]+=)"
value split : "="

I'm wondering if I could have any hidden special character or if having "data" as a key could be forbidden by design. I ve tried to rename the key "data" in "charac", but i still get the same issue.

I'm really confused since when running the pipeline into the pipeline config menu it's working.

Could someone help me please ? I've probably missed something. Thanks

Pierre

[stephenb (Stephen Brown)]

Hi @Pierre_LANCASTRE Welcome to the community!

Well I think you ran into a tough one...

Can you show us the whole KV processor definition just so we can see...

So I suspect Here is the issue ... user is an object in ECS and you are trying to just set it as a field.

@Pierre_LANCASTRE

user="toto"

Which explains

@Pierre_LANCASTRE

I'm really confused since when running the pipeline into the pipeline config menu it's working.

Which just simulates / tests the pipeline but does not actually write the document which would then have a mapper parsing exception.

You could test this by actually trying to POST a document with the pipeline

POST myindex/_doc/?pipeline=my-pipeline
{
   "message" : "your message" 
}

You will probably get a mapping exception...

So after the KV you will need to rename the user field to either

user.name
or
user.id

so it is compliant...

[Pierre_LANCASTRE]

Hi Stephen,

Thanks a lot for your feedback. Your answer helped to return on one solution : prefix the generated fields by something. I think the first time the name I used needed to have alread an existing object (I forgot to do that). So there, I configured the K-V to prefix the fields with "event." so I got event.data, event.user, etc. and it works

Thanks a lot for your help

BR

Pierre L