Understand fleet pipelines and reroute

Elastic Stack Elasticsearch
1

I need to, for a number of reasons, reroute the output of a fleet pipeline. To keep it very simple, lets use the fortigate integration as the example. I want my data to come in via elastic agent(working), be processed by the fortigate ingest pipelines(works) and as the very last step, I want those events to be output to a datastream called logs-main. (no namespace).

I have tried adding a reroute processor for dataset main to my logs-fortinet_fortigate.log@custom pipeline, but it fails processing and no events are ingested. I have also tried adding the reroute to the .fleet-final-pipeline-1 but the same issue occurs here. If I use other processors, they work fine. I know that the reroute processor stops processing of other processors, but it seems that if I have it in final, it should work and I should have my data where I want it. Very confused on the processing order that happens with regard to rewrite. Any help appreciated.

I am running elastic-cloud on 8.18.1.

2

Hello and welcome,

The order is the integration ingest pipeline > the integration custom pipelines > fleet final pipeline (you should not make any change on this one as this is used by all integrations).

But the main issue here is that the reroute processor does not work with Elastic Agent integrations in most of the cases.

Fleet managed agents write into Elasticsearch using an API key, this API key is created with just the permissions required according to the integrations configured in the agent policy.

If you have a policy with just the Fortigate integration, it will have permissions to write only to logs-fortinet_fortigate.log-<namespace>, where <namespace> is the namespace you choose while configuring the integration.

If you try to use a reroute processor to change the dataset or namespace name, you will stop receiving data because the events will be rejected on permissions errors.

There is a change targeting version 9.1 that would allow users to add additional permissions on each integration so the rerouteprocessor could be used.

There is also a workaround where you can add a integration that has permission on logs-*-* to the same integration just so the API key would have the required permissions for the reroute processor, for example you could also a custom log integration to the same policy to add those broarder permissions.

But even if you manage to have the right policy with the right permissions there are some issues in what you want to do.

I want those events to be output to a datastream called logs-main. (no namespace).

This is not possible a namespace is required, the data stream for integrations needs to follow the naming pattern logs-<dataset>-<namespace>.

You also need to take care of the mapping if you change the dataset name or you may have mapping issues, you would need to clone the fortinet fortigate index template and change the matching pattern for the datastream.

3

Thanks for the feedback. A lot of good information here.

Regarding the namespace question, I can use a namespace if I have too. I currently have datastreams in production with namespaces, and some without them and they seem to work fine without the namespace actually being populated. As I have said though, I am fine just using 'default' as a namespace if needed to eliminate that as a complication. Yes, I understand the mapping issue and I would need to do some consolidation on the mapping to handle that.

So, I tried the most basic agent task I could think of, a custom log, and am still not getting the reroute to do anything but die. It must be something basic I would think. I created a custom log agent(the only integration in the policy), deployed it. It has a default pipeline set, as well as an index that should end up being a DS. The pipeline does only two tasks, it sets a value, and it does the reroute, in that order. If I test the ingest pipeline against a document, it works as expected and both the value and the reroute work fine. If I let the agent pick up the log, it fails. If I remove the reroute from the pipeline, it succeeds. So, I know the pipeline is run, and I know it is the reroute that is causing the failure. Any Thoughts?

I am adding the agent policy, pipeline, and index template here as well.
index template:

PUT _index_template/logs-main.varonis-default
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "logs"
        },
        "mode": "logsdb",
        "codec": "best_compression",
        "default_pipeline": "logs-main.varonis-default",
        "mapping": {
          "total_fields": {
            "ignore_dynamic_beyond_limit": "true"
          },
          "ignore_malformed": "true"
        }
      }
    },
    "mappings": {
      "_source": {
        "mode": "synthetic"
      }
    }
  },
  "index_patterns": [
    "logs-main.varonis-default"
  ],
  "data_stream": {
    "hidden": false,
    "allow_custom_routing": false
  },
  "composed_of": [
    "component-ecs-2",
    "component-siem-2"
  ],
  "ignore_missing_component_templates": []
}

pipeline

{
  "processors": [
    {
      "set": {
        "field": "observer.product",
        "value": "varonis2"
      }
    },
    {
      "reroute": {
        "dataset": [
          "main.experiment"
        ],
        "namespace": [
          "default"
        ]
      }
    }
  ],
  "on_failure": [
    {
      "fail": {
        "message": "VARONISFAIL",
        "tag": "varonisfail"
      }
    }
  ]
}

agent

id: fleet-first-agent-policy
revision: 42
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://bbdc4b39ff1.us-east-1.aws.found.io:443'
    preset: balanced
fleet:
  hosts:
    - 'https://46153a88e14ec1df.fleet.us-east-1.aws.found.io:443'
output_permissions:
  default:
    6dd73ce2-61a5-47a4-adce-85f1487f6a6a:
      indices:
        - names:
            - logs-main.varonis-default
          privileges: *ref_0
agent:
  download:
    sourceURI: 'https://artifacts.elastic.co/downloads/'
  monitoring:
    enabled: true
    use_output: default
    logs: true
    metrics: true
    traces: true
    namespace: default
  features: {}
  protection:
    enabled: false
    uninstall_token_hash: 0YRpwBwnnuo0oImA68wdLR4nDlqiU+ppoXsu9r1MvKQ=
    signing_key: >-
      MFkwEwYHKoZIhPb0whFYQVXJNrxQg==
inputs:
  - id: filestream-filestream-6dd73ce2-61a5-47a4-adce-85f1487f6a6a
    name: varonis-1
    revision: 5
    type: filestream
    use_output: default
    meta:
      package:
        name: filestream
        version: 1.1.3
    data_stream:
      namespace: default
    package_policy_id: 6dd73ce2-61a5-47a4-adce-85f1487f6a6a
    streams:
      - id: filestream-filestream.generic-6dd73ce2-61a5-47a4-adce-85f1487f6a6a
        data_stream:
          dataset: main.varonis
        paths:
          - /var/syslog/varonis/1.1.1.5/varonis.log
        pipeline: logs-main.varonis-default
        prospector.scanner.recursive_glob: true
        prospector.scanner.exclude_files:
          - \.gz$
        clean_inactive: -1
        parsers: null
        tags:
          - varonis
        prospector.scanner.fingerprint.enabled: true
        file_identity.fingerprint.enabled: true
        file_identity.fingerprint.offset: 0
        file_identity.fingerprint.length: 1024
signed:
  data: >-
    redacted==
  signature: >-
    redacted==
secret_references: []
namespaces: []