Hello,
We have integrated Fortinet Fortigate Firewall logs to our SIEM ELK using the Elastic Agent integration. We created also a custom ingest pipeline that separates the logs (for example forward) and send them to a dedicated datastream.
The problem is that ingest pipeline doesnt work and the new datastream/index is not created.
How to solve that or if there any other solutions to separate these logs.
Thanks.
Hi @Knight7
If you want help you need to provide the following.
What are you changing and how are you redirecting to a different datastreams?
Please describe what are you doing.
Also, what integrations are present on the Agent Policy?
Hello guys,
OK, the integration is Fortigate Firewall which collects firewall logs from a log file. This integration is added to an agent policy.
What is important is the following:
PUT _ingest/pipeline/logs-fortinet_fortigate.log@custom
{
"description": "Reroute Firewall logs to sub-datastreams",
"version": 2,
"processors": [
{
"reroute": {
"dataset": [
"fortinet_fortigate.log_forward"
],
"namespace": [
"default"
],
"if": "ctx.fortinet.firewall.subtype == 'forward'"
}
}
]
}
When testing, I added a document from the fortigate firewall datastream indexes and the output of the test is as follows:
{
"docs": [
{
"doc": {
"_index": ".ds-fortinet_fortigate.log_forward-default",
"_version": "-3",
"_id": "1gtyfpgBGjSND2_ayxHd",
"_source": {
"agent": {
"name": "<AGENT_NAME>",
"id": "<AGENT_ID>",
"type": "filebeat",
"ephemeral_id": "24a8a5ad-f1ea-4b4a-bde9-ab67c3862d32",
"version": "8.16.6"
},
"log": {
"file": {
"path": "/var/logs/firewall.log"
},
"offset": 6571314386,
"level": "notice"
},
"elastic_agent": {
"id": "<AGENT_ID>",
"version": "8.16.6",
"snapshot": false
},
"destination": {
"port": 53,
"bytes": 91,
"mac": "00-50-56-93-30-50",
"packets": 1,
"ip": "192.168.1.5"
},
"rule": {
"ruleset": "policy",
"name": "<POLICY_NAME>",
"id": "<POLICY_ID>",
"category": "unscanned",
"uuid": "<POLICY_UUID>"
},
"source": {
"port": 40249,
"bytes": 75,
"mac": "00-00-00-11-11-11",
"packets": 1,
"ip": "192.168.1.6"
},
"tags": [
"fortinet-fortigate",
"fortinet-firewall",
"forwarded"
],
"network": {
"protocol": "dns",
"transport": "udp",
"bytes": 166,
"iana_number": "17",
"packets": 2
},
"input": {
"type": "log"
},
"observer": {
"ingress": {
"interface": {
"name": "<IFACE_NAME>"
}
},
"product": "Fortigate",
"vendor": "Fortinet",
"name": "<FIREWALL_NAME>",
"serial_number": "<FIREWALL_SERIAL_NUMBER>",
"type": "firewall",
"egress": {
"interface": {
"name": "<IFACE_NAME>"
}
}
},
"@timestamp": "2025-08-06T10:11:10.000+02:00",
"ecs": {
"version": "8.17.0"
},
"related": {
"ip": [
"<IP1>",
"<IP2>"
]
},
"data_stream": {
"namespace": "default",
"type": ".ds",
"dataset": "fortinet_fortigate.log_forward"
},
"fortinet": {
"firewall": {
"srcintfrole": "lan",
"logver": "0704072731",
"dsthwvendor": "<VENDOR>",
"srcserver": "0",
"itime": "1754467870",
"sessionid": "1114085446",
"itime_converted": "2025-08-06T08:11:10.000Z",
"type": "traffic",
"vd": "root",
"srccountry": "Reserved",
"dstintfrole": "lan",
"subtype": "forward",
"mastersrcmac": "00:50:56:a9:09:31",
"action": "accept",
"masterdstmac": "00:50:56:93:30:50",
"trandisp": "noop",
"dstcountry": "Reserved",
"srchwvendor": "<VENDOR>",
"timestamp": "1754478670",
"dstserver": "0"
}
},
"event": {
"code": "0000000013",
"timezone": "+0200",
"kind": "event",
"start": "2025-08-06T10:11:10.115+02:00",
"type": [
"connection",
"end",
"allowed"
],
"duration": 181000000000,
"agent_id_status": "verified",
"ingested": "2025-08-06T08:15:09Z",
"action": "accept",
"category": [
"network"
],
"dataset": "fortinet_fortigate.log_forward",
"outcome": "success"
}
},
"_ingest": {
"timestamp": "2025-08-06T08:23:21.418586505Z"
}
}
}
]
}
Which shows that the ingest pipeline works perfectly and the index ".ds-fortinet_fortigate.log_forward-default" will be created.
But, after applying this, the index will not be created and the newly received forward logs will not be indexed.
I'm using Elasticsearch version 8.16.1
Hi @Knight7 You tested in the UI I assume?, yeah that is not always 100% "accurate"
So here is how I would test this actually try to write / reroute a document, I have an idea but I want to see.
First add a set processor to your @custom pipeline so you can see if it is actually called something like
{
"set": {
"field": "custome_pipeline_name",
"value": "logs-fortinet_fortigate.log@custom"
}
},
Go To Kibana -> Dev Tools
Get the JSON of the document you want to test.
What goes in the body is what is between the _source: braces
POST logs-fortinet_fortigate.log-default/_doc
{
"agent": {
"name": "<AGENT_NAME>",
"id": "<AGENT_ID>",
"type": "filebeat",
"ephemeral_id": "24a8a5ad-f1ea-4b4a-bde9-ab67c3862d32",
"version": "8.16.6"
},
"log": {
"file": {
"path": "/var/logs/firewall.log"
},
"offset": 6571314386,
"level": "notice"
},
...
}
Show the errors ... Keep tract od the backing _index and _id and you can delete afterwards
Hello @stephenb ,
I tested your recommandation and the output is OK in testing phase.
Output:
{
"docs": [
{
"doc": {
"_index": ".ds-fortinet_fortigate.log_forward-default",
"_version": "-3",
"_id": "bx3Uf5gBGjSND2_adBI3",
"_source": {
"agent": {
"name": "<AGENT_NAME>",
"id": "<AGENT_ID>",
"type": "filebeat",
"ephemeral_id": "24a8a5ad-f1ea-4b4a-bde9-ab67c3862d32",
"version": "8.16.6"
},
...
},
"event": {
"code": "0000000013",
"timezone": "+0200",
"kind": "event",
"start": "2025-08-06T16:37:28.894+02:00",
"type": [
"connection",
"end",
"denied"
],
"duration": 0,
"agent_id_status": "verified",
"ingested": "2025-08-06T14:41:26Z",
"action": "deny",
"category": [
"network"
],
"dataset": "fortinet_fortigate.log_forward",
"outcome": "success"
}
},
"_ingest": {
"timestamp": "2025-08-06T16:04:25.65708688Z"
}
}
}
]
}
Yet, after applying this, no no datastream is created.
I'm thinking about these problems that you guys talk about : New index not created by ingestion pipeline - #19 by leandrojmp
and it could be a permission problem : Could it be that ?
That output does not look like a POST .../_doc that looks like a pipeline _simulate did you actually post a doc?
{
"docs": [
{
"_index": ".ds-fortinet_fortigate.log_forward-default",
That is incorrect... for sure....
Does this index existing
GET .ds-fortinet_fortigate.log_forward-default
BTW it should end up in
.ds-logs-fortinet_fortigate.log_forward-default
I Also do not see the field I suggested to set...
Also when you edit the doc and don't show the data_stream field it is hard to debug
Please try to include all the information
It is probably a permission issue.
The way that Elastic Agent handles permissions is pretty limited, you cannot use the reroute processor because the API Key generated for the integration only has permissions to write into the data streams and namespace in the configuration.
You can check this similar post where there is a lengthy discussion about it.
In short, per default the reroute processor will not work, you cannot change the namespace of a datastream as the API Key generated for the policy does not include it, you can check the agent logs and you will probably have a lot of errors about not being able to index data.
The workaround for this case would be to add an integration in the same policy that has permissions to write into logs-*-*, this would make the API Key used by the policy to have those same permissions and it would allow you to use the reroute processor, for example you could add a Custom Filestream logs integration that do not collect anything, just to have the permissions.
There was a change planned to 9.1 to allow the user to specify extra permissions on the UI, but I'm not sure if it is already active as I'm on 8.18 still.
Is this change here: [Fleet] Add UI to add additional datastreams permissions by nchaulet · Pull Request #210935 · elastic/kibana · GitHub
What @leandrojmp Said + There is still something wrong with routing...
Leandro I think you mean you CAN change the data_stream.namespace but you CAN NOT change the data_stream.dataset which because of API KEY permission is what you are referring to
You can change the namespace in the configuration, but after the policy is configured you cannot use the reroute processor to change it as the API Key generated was based on the configured namespace.
The reroute processor cannot be used if the policy does not have an integration with permissions to write on logs-*-*, it is blocked in some way as mentioned in this github issue
If you create a policy with just the Fortigate integration, this would be the permissions generated:
output_permissions:
default:
_elastic_agent_monitoring:
indices: []
_elastic_agent_checks:
cluster:
- monitor
2b130d2f-d313-4b73-ac49-0fd0e1606627:
indices:
- names:
- logs-fortinet_fortigate.log-default
privileges:
- auto_configure
- create_doc
- names:
- logs-fortinet_fortigate.log-default
privileges:
- auto_configure
- create_doc
If you try to use a reroute processor to change the namespace from default to anything else, it would fail.
Ahhh good distinction.. thanks
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.