# Fortigate Integration - Separation of logs datastream based on a field value

**URL:** <https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767>\
**Category:** Elastic Agent\
**Tags:** integrations\
**Created:** [August 5, 2025, 4:48pm UTC](https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767 "2025-08-05T16:48:16Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Knight7](https://avatars.discourse-cdn.com/v4/letter/k/7feea3/32.png) [@Knight7](https://discuss.elastic.co/u/Knight7)\
**Post date:** [August 5, 2025, 4:48pm UTC](https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767/1 "2025-08-05T16:48:16Z")

</div>

Hello,

We have integrated Fortinet Fortigate Firewall logs to our SIEM ELK using the Elastic Agent integration. We created also a custom ingest pipeline that separates the logs (for example forward) and send them to a dedicated datastream.

The problem is that ingest pipeline doesnt work and the new datastream/index is not created.

How to solve that or if there any other solutions to separate these logs.

Thanks.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 5, 2025, 6:17pm UTC](https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767/2 "2025-08-05T18:17:25Z")

</div>

Hi @Knight7

If you want help you need to provide the following.

- Elastic version number
- Integration name and version number
- Sample document
- And your ingest pipeline name and contents

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 5, 2025, 6:42pm UTC](https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767/3 "2025-08-05T18:42:25Z")

</div>

> [@Knight7](#):
>
> The problem is that ingest pipeline doesnt work and the new datastream/index is not created.
> 
> How to solve that or if there any other solutions to separate these logs.

What are you changing and how are you redirecting to a different datastreams?

Please describe what are you doing.

Also, what integrations are present on the Agent Policy?

---

<div class="post-metadata">

**Author:** ![Knight7](https://avatars.discourse-cdn.com/v4/letter/k/7feea3/32.png) [@Knight7](https://discuss.elastic.co/u/Knight7)\
**Post date:** [August 6, 2025, 8:30am UTC](https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767/4 "2025-08-06T08:30:50Z")

</div>

Hello guys,

OK, the integration is Fortigate Firewall which collects firewall logs from a log file. This integration is added to an agent policy.

What is important is the following:

- I've added a custom ingest pipeline to the integration policy named : "logs-fortinet\_fortigate.log@custom" and its content is this:

```auto
PUT _ingest/pipeline/logs-fortinet_fortigate.log@custom
{
  "description": "Reroute Firewall logs to sub-datastreams",
  "version": 2,
  "processors": [
    {
      "reroute": {
        "dataset": [
          "fortinet_fortigate.log_forward"
        ],
        "namespace": [
          "default"
        ],
        "if": "ctx.fortinet.firewall.subtype == 'forward'"
      }
    }
  ]
}

```

When testing, I added a document from the fortigate firewall datastream indexes and the output of the test is as follows:

```auto
{
  "docs": [
    {
      "doc": {
        "_index": ".ds-fortinet_fortigate.log_forward-default",
        "_version": "-3",
        "_id": "1gtyfpgBGjSND2_ayxHd",
        "_source": {
          "agent": {
            "name": "<AGENT_NAME>",
            "id": "<AGENT_ID>",
            "type": "filebeat",
            "ephemeral_id": "24a8a5ad-f1ea-4b4a-bde9-ab67c3862d32",
            "version": "8.16.6"
          },
          "log": {
            "file": {
              "path": "/var/logs/firewall.log"
            },
            "offset": 6571314386,
            "level": "notice"
          },
          "elastic_agent": {
            "id": "<AGENT_ID>",
            "version": "8.16.6",
            "snapshot": false
          },
          "destination": {
            "port": 53,
            "bytes": 91,
            "mac": "00-50-56-93-30-50",
            "packets": 1,
            "ip": "192.168.1.5"
          },
          "rule": {
            "ruleset": "policy",
            "name": "<POLICY_NAME>",
            "id": "<POLICY_ID>",
            "category": "unscanned",
            "uuid": "<POLICY_UUID>"
          },
          "source": {
            "port": 40249,
            "bytes": 75,
            "mac": "00-00-00-11-11-11",
            "packets": 1,
            "ip": "192.168.1.6"
          },
          "tags": [
            "fortinet-fortigate",
            "fortinet-firewall",
            "forwarded"
          ],
          "network": {
            "protocol": "dns",
            "transport": "udp",
            "bytes": 166,
            "iana_number": "17",
            "packets": 2
          },
          "input": {
            "type": "log"
          },
          "observer": {
            "ingress": {
              "interface": {
                "name": "<IFACE_NAME>"
              }
            },
            "product": "Fortigate",
            "vendor": "Fortinet",
            "name": "<FIREWALL_NAME>",
            "serial_number": "<FIREWALL_SERIAL_NUMBER>",
            "type": "firewall",
            "egress": {
              "interface": {
                "name": "<IFACE_NAME>"
              }
            }
          },
          "@timestamp": "2025-08-06T10:11:10.000+02:00",
          "ecs": {
            "version": "8.17.0"
          },
          "related": {
            "ip": [
              "<IP1>",
              "<IP2>"
            ]
          },
          "data_stream": {
            "namespace": "default",
            "type": ".ds",
            "dataset": "fortinet_fortigate.log_forward"
          },
          "fortinet": {
            "firewall": {
              "srcintfrole": "lan",
              "logver": "0704072731",
              "dsthwvendor": "<VENDOR>",
              "srcserver": "0",
              "itime": "1754467870",
              "sessionid": "1114085446",
              "itime_converted": "2025-08-06T08:11:10.000Z",
              "type": "traffic",
              "vd": "root",
              "srccountry": "Reserved",
              "dstintfrole": "lan",
              "subtype": "forward",
              "mastersrcmac": "00:50:56:a9:09:31",
              "action": "accept",
              "masterdstmac": "00:50:56:93:30:50",
              "trandisp": "noop",
              "dstcountry": "Reserved",
              "srchwvendor": "<VENDOR>",
              "timestamp": "1754478670",
              "dstserver": "0"
            }
          },
          "event": {
            "code": "0000000013",
            "timezone": "+0200",
            "kind": "event",
            "start": "2025-08-06T10:11:10.115+02:00",
            "type": [
              "connection",
              "end",
              "allowed"
            ],
            "duration": 181000000000,
            "agent_id_status": "verified",
            "ingested": "2025-08-06T08:15:09Z",
            "action": "accept",
            "category": [
              "network"
            ],
            "dataset": "fortinet_fortigate.log_forward",
            "outcome": "success"
          }
        },
        "_ingest": {
          "timestamp": "2025-08-06T08:23:21.418586505Z"
        }
      }
    }
  ]
}

```

Which shows that the ingest pipeline works perfectly and the index ".ds-fortinet\_fortigate.log\_forward-default" will be created.

But, after applying this, the index will not be created and the newly received forward logs will not be indexed.

I'm using Elasticsearch version 8.16.1

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 6, 2025, 2:20pm UTC](https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767/5 "2025-08-06T14:20:38Z")

</div>

Hi @Knight7 You tested in the UI I assume?, yeah that is not always 100% "accurate"

So here is how I would test this actually try to write / reroute a document, I have an idea but I want to see.

First add a `set` processor to your `@custom` pipeline so you can see if it is actually called something like

```auto
  {
    "set": {
      "field": "custome_pipeline_name",
      "value": "logs-fortinet_fortigate.log@custom"
    }
  },

```

Go To Kibana -\> Dev Tools

Get the JSON of the document you want to test.

What goes in the body is what is between the `_source: ` braces

```auto
POST logs-fortinet_fortigate.log-default/_doc

{
          "agent": {
            "name": "<AGENT_NAME>",
            "id": "<AGENT_ID>",
            "type": "filebeat",
            "ephemeral_id": "24a8a5ad-f1ea-4b4a-bde9-ab67c3862d32",
            "version": "8.16.6"
          },
          "log": {
            "file": {
              "path": "/var/logs/firewall.log"
            },
            "offset": 6571314386,
            "level": "notice"
          },
...

}

```

Show the errors ... Keep tract od the backing `_index` and `_id` and you can delete afterwards

---

<div class="post-metadata">

**Author:** ![Knight7](https://avatars.discourse-cdn.com/v4/letter/k/7feea3/32.png) [@Knight7](https://discuss.elastic.co/u/Knight7)\
**Post date:** [August 6, 2025, 4:04pm UTC](https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767/6 "2025-08-06T16:04:55Z")

</div>

Hello @stephenb ,  
I tested your recommandation and the output is OK in testing phase.

Output:

```auto
{
  "docs": [
    {
      "doc": {
        "_index": ".ds-fortinet_fortigate.log_forward-default",
        "_version": "-3",
        "_id": "bx3Uf5gBGjSND2_adBI3",
        "_source": {
          "agent": {
            "name": "<AGENT_NAME>",
            "id": "<AGENT_ID>",
            "type": "filebeat",
            "ephemeral_id": "24a8a5ad-f1ea-4b4a-bde9-ab67c3862d32",
            "version": "8.16.6"
          },
...
          },
          "event": {
            "code": "0000000013",
            "timezone": "+0200",
            "kind": "event",
            "start": "2025-08-06T16:37:28.894+02:00",
            "type": [
              "connection",
              "end",
              "denied"
            ],
            "duration": 0,
            "agent_id_status": "verified",
            "ingested": "2025-08-06T14:41:26Z",
            "action": "deny",
            "category": [
              "network"
            ],
            "dataset": "fortinet_fortigate.log_forward",
            "outcome": "success"
          }
        },
        "_ingest": {
          "timestamp": "2025-08-06T16:04:25.65708688Z"
        }
      }
    }
  ]
}

```

Yet, after applying this, no no datastream is created.

I'm thinking about these problems that you guys talk about : [New index not created by ingestion pipeline - #19 by leandrojmp](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/19)

and it could be a permission problem : Could it be that ?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 6, 2025, 4:22pm UTC](https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767/7 "2025-08-06T16:22:03Z")

</div>

That output does not look like a `POST .../_doc` that looks like a pipeline \_simulate did you actually post a doc?

```auto
{
  "docs": [
    {

```

` "_index": ".ds-fortinet_fortigate.log_forward-default",`

That is incorrect... for sure....

Does this index existing

`GET .ds-fortinet_fortigate.log_forward-default`

BTW it should end up in

`.ds-logs-fortinet_fortigate.log_forward-default`

I Also do not see the field I suggested to set...

Also when you edit the doc and don't show the `data_stream` field it is hard to debug

Please try to include all the information

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 6, 2025, 4:42pm UTC](https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767/8 "2025-08-06T16:42:43Z")

</div>

> [@Knight7](#):
>
> and it could be a permission problem : Could it be that ?

It is probably a permission issue.

The way that Elastic Agent handles permissions is pretty limited, you cannot use the `reroute` processor because the API Key generated for the integration only has permissions to write into the data streams and namespace in the configuration.

You can check this similar [post](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211) where there is a lengthy discussion about it.

In short, per default the `reroute` processor will not work, you cannot change the namespace of a datastream as the API Key generated for the policy does not include it, you can check the agent logs and you will probably have a lot of errors about not being able to index data.

The workaround for this case would be to add an integration in the same policy that has permissions to write into `logs-*-*`, this would make the API Key used by the policy to have those same permissions and it would allow you to use the `reroute` processor, for example you could add a Custom Filestream logs integration that do not collect anything, just to have the permissions.

There was a change planned to 9.1 to allow the user to specify extra permissions on the UI, but I'm not sure if it is already active as I'm on 8.18 still.

Is this change here: [[Fleet] Add UI to add additional datastreams permissions by nchaulet · Pull Request #210935 · elastic/kibana · GitHub](https://github.com/elastic/kibana/pull/210935)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 6, 2025, 4:58pm UTC](https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767/9 "2025-08-06T16:58:36Z")

</div>

What @leandrojmp Said + There is still something wrong with routing...

Leandro I think you mean you CAN change the `data_stream.namespace` but you CAN NOT change the `data_stream.dataset` which because of API KEY permission is what you are referring to

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 6, 2025, 5:14pm UTC](https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767/10 "2025-08-06T17:14:14Z")

</div>

You can change the namespace in the configuration, but after the policy is configured you cannot use the `reroute` processor to change it as the API Key generated was based on the configured namespace.

The `reroute` processor cannot be used if the policy does not have an integration with permissions to write on `logs-*-*`, it is blocked in some way as mentioned in this [github issue](https://github.com/elastic/integrations/issues/12606#issuecomment-2678696935)

If you create a policy with just the Fortigate integration, this would be the permissions generated:

```auto
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices: []
    _elastic_agent_checks:
      cluster:
        - monitor
    2b130d2f-d313-4b73-ac49-0fd0e1606627:
      indices:
        - names:
            - logs-fortinet_fortigate.log-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-fortinet_fortigate.log-default
          privileges:
            - auto_configure
            - create_doc

```

If you try to use a `reroute` processor to change the namespace from `default` to anything else, it would fail.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 6, 2025, 7:10pm UTC](https://discuss.elastic.co/t/fortigate-integration-separation-of-logs-datastream-based-on-a-field-value/380767/11 "2025-08-06T19:10:38Z")

</div>

> [@leandrojmp](#):
>
> You can change the namespace in the configuration, but after the policy is configured you cannot use the `reroute` processor to change it as the API Key generated was based on the configured namespace.

Ahhh good distinction.. thanks