Copy_fields and Syslog parsing out of order

[Hythloday-zero]

When I use a Custom TCP Logs integration with Syslog parsing, I expect that

Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.

..however, I experience the opposite. E.g. I apply

  - copy_fields:
      fields:
        - from: message
          to: event.original
      fail_on_error: false
      ignore_missing: true

to my Processors block when editing my integration and I end up getting all the syslog fields parsed from 'message' field and then 'event.original' is copied from the remaining post-parsed 'message' field. I would expect to see 'event.original' looking like normal syslog and 'message' to look like the message payload contained within the syslog record.

Can we call this a bug and submit a GH issue against it, or am I doing something incorrectly?

[strawgate (William Easton)]

My understanding of how this works is that when you check the syslog parsing field in the integration settings all it's doing is placing the syslog processor first in the list of processors.

You can see this behavior by pressing Actions > View Agent Policy on the agent policy where you've configured the integration, if you look at the resulting yaml you'll see what is actually getting passed down to agent.

When i configure the custom tcp integration with syslog parsing, it renders this
(among other things):

        processors:
          - syslog:
              field: message

And when we add your copy_fields block the agent policy looks like this:

        processors:
          - syslog:
              field: message
          - copy_fields:
              fields:
                - from: message
                  to: event.original
              fail_on_error: false
              ignore_missing: true

So as you've noted your processor is getting added after the syslog parsing, and by that point the original message has been mutated with the timestamp, priority, etc stripped out.

To achieve the behavior you're looking for I think all you'd have to do is disable the syslog parsing check in the custom tcp logs integration and add the syslog processor after your copy_fields processor like this:

  - copy_fields:
      fields:
        - from: message
          to: event.original
      fail_on_error: false
      ignore_missing: true
  - syslog:
      field: message