# Firewall logs to different Datastream by type **URL:** \ **Category:** SIEM\ **Created:** [April 16, 2025, 4:52pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211 "2025-04-16T16:52:42Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![juancamiloll](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juancamiloll/32/110326_2.png) [@juancamiloll](https://discuss.elastic.co/u/juancamiloll)\ **Post date:** [April 16, 2025, 4:52pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/1 "2025-04-16T16:52:42Z")
I am using the integration “Fortinet FortiGate Firewall Logs”. To date it is working correctly, but I have been asked to ingest the UTM type logs in a different Datastream and I really have no idea how to do this using the integration. I would appreciate any suggestion you can give me. This is the configuration I currently have ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/1/d1e3d2c68c09ee2d9226b5562e696b73aaa1abba.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/5/159ad9304d7bf5d55e8f0a27ead9b0d5818a6482.png) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 16, 2025, 6:04pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/2 "2025-04-16T18:04:16Z")
Hi @juancamiloll What type of the logs are you ingesting today? event, traffic, login? The integration should figure out the different types. If the UTM is coming in on a different list address port etc just create another integration and add it to the put in the correct listen and port Go to the Policy : LogstashPolicy and a new integration and put the correct listener Address and Port I would name the integrations with good names like fortinet\_fortigate\_event fortinet\_fortigate\_utm Hope that helps... ---
**Author:** ![juancamiloll](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juancamiloll/32/110326_2.png) [@juancamiloll](https://discuss.elastic.co/u/juancamiloll)\ **Post date:** [April 16, 2025, 6:26pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/3 "2025-04-16T18:26:13Z")
@stephenb Hello, Thanks for replying, the logs I receive are TRAFFIC, UTM and event. They all come on port 51403 TCP, I have no way to request a certain type of log on a different port. As in this case I am not using a .conf from logstash but I am ingesting with the help of integration so I have no idea. I am hoping that some advanced configuration of the integration will allow me to do this. ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/3/c3dd8057cfeac1a5c63013172458533370d115cc.png) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 16, 2025, 7:03pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/4 "2025-04-16T19:03:22Z")
The Integration can figure that all out... they can all come on the same port When the data comes in... the initial ingest pipeline will figure it all out... and parse all the data correctly ... if you are saying that you "customers" want the data in different data stream I would ask why... as that could affect the capabilities So a key question is WHY they want a different data stream what are they trying to accomplish, this integration is set up this way on purpose. What they are trying to accomplish can affect how you implement (and yes there will be a little work need) Do they want to separate for Access Control, Different ILM Policies etc? ... or just because it "seems" like a good idea because data streams are new to them... And yes with some work you can change the namespace (which is really NOT the intention) but that would put the different types in different but related data stream There should be fields that will provide easy filtering and sorting etc. So find out the Why / What they are trying to accomplish and you will probably need to create a custom ingest pipeline to do some sorting / routing to different namespace based on some fields... not trivial ... but not toooooo hard. In the meantime I would probably read about Elastic Agent and data streams > **[Data streams | Elastic documentation](https://www.elastic.co/docs/reference/fleet/data-streams)** > > Elastic Agent uses data streams to store time series data across multiple indices while giving you a single named resource for requests. Data streams... ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 16, 2025, 7:13pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/5 "2025-04-16T19:13:46Z")
> [@juancamiloll](#): > > To date it is working correctly, but I have been asked to ingest the UTM type logs in a different Datastream and I really have no idea how to do this using the integration. It is not possible at the moment, I have the same requirement and opened an issue about it a couple of months ago: > > > Hello, > > Currently the integration for the Fortinet Fortigate has only one datase…t, \`fortinet\_fortigate.log\`, which means that all logs from fortigate will be stored in the same data stream, but the integration produces different types of logs like \`traffic\`, \`utm\` and \`event\`. > > The goal for this is that this would allow you to have different retention per dataset, for example, you may be required to keep audit logs for a longer time than traffic logs, audit logs have the \`fortinet.firewall.type\` value as \`event\` and traffic logs have the same field as \`traffic\`. > > Since the volume of \`traffic\` logs are higher, to keep only the audit logs you do not have many options, you basically need to create a custom transform to store those events in another index, I'm not sure that you can use a \`reroute\` processor to create a custom dataset/namespace since Fleet integrations permissions are pretty limited. > > Now we have this: > > \- \`fortinet\_fortigate.log\`: all logs from fortigate > > But we should have something like this: > > \- \`fortinet\_fortigate.traffic\`: traffic logs from fortigate > \- \`fortinet\_fortigate.event\`: event logs from fortigate > \- \`fortinet\_fortigate.utm\`: utm logs from fortigate > \- \`fortinet\_fortigate.log\`: new/unmapped logs from fortigate > > We already have separated ingest pipelines for each \`fortinet.firewall.type\` value \[here\](https://github.com/elastic/integrations/blob/ed3158658d5f72af36d63d02aba612fd24f368e2/packages/fortinet\_fortigate/data\_stream/log/elasticsearch/ingest\_pipeline/default.yml#L252-L268). > > \`\`\` > - pipeline: > name: '{{ IngestPipeline "event" }}' > if: ctx.fortinet?.firewall?.type == 'event' > - pipeline: > name: '{{ IngestPipeline "login" }}' > if: \>- > if (ctx.message instanceof String) { > String normalizedMessage = ctx.message.toLowerCase(); > return (normalizedMessage.contains('login') || normalizedMessage.contains('logged in')); > } > return false; > - pipeline: > name: '{{ IngestPipeline "traffic" }}' > if: ctx.fortinet?.firewall?.type == 'traffic' > - pipeline: > name: '{{ IngestPipeline "utm" }}' > if: ctx.fortinet?.firewall?.type == 'utm' || ctx.fortinet?.firewall?.type == 'dns' > \`\`\` > > I think this helps implement separated data streams, I'm not sure how to do that, but with some orientation I could send a PR. > > What we need to do to create a new dataset? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 16, 2025, 7:16pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/6 "2025-04-16T19:16:55Z")
Well now that I know @leandrojmp has requested it... it is certainly legit! 🙂 Did you have some code to share / solve / route? ---
**Author:** ![juancamiloll](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juancamiloll/32/110326_2.png) [@juancamiloll](https://discuss.elastic.co/u/juancamiloll)\ **Post date:** [April 16, 2025, 7:24pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/7 "2025-04-16T19:24:09Z")
Thank you both for your help. So the only option would be using .conf from logstash and according to the documentation use a “processor” as “Reroute” ? > **[Reroute processor | Elastic Documentation](https://www.elastic.co/docs/reference/enrich-processor/reroute-processor)** > > The reroute processor allows to route a document to another target index or data stream. It has two main modes: When setting the destination option, the... ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 16, 2025, 7:41pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/8 "2025-04-16T19:41:27Z")
> [@juancamiloll](#): > > So the only option would be using .conf from logstash and according to the documentation use a “processor” as “Reroute” ? No, it is not that simple, it depends on a couple of things and the `reroute` processor may not work, it is blocked in the majority of the integrations as mentioned in the github issue linked. How are you sending data? Is your Fortigate sending data directly to the Elastic Agent? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 16, 2025, 7:50pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/9 "2025-04-16T19:50:16Z")
@leandrojmp Did you test reroute with just changing the namespace I think that would work... I don't think that's the best solution but I think that could work.. Yes I understand Really not the intention of namespace. And are we sure that reroute will not work with the dataset name? Seems like perhaps we have not tested for this integration I do use reroute like you said on some of the more generic integrations... I don't have any way to test because I don't have a harness for this. You have the right people in that issue looking at it ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 16, 2025, 8:34pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/10 "2025-04-16T20:34:37Z")
As far as I know the `reroute` processor is blocked in the integrations because of how the permissions for the API Key used by the Elastic Agent works, this is mentioned in the linked github issue. If you go into a Fleet Policy and in the _Actions_ Menu select the option to _View the Policy_, you will have something like this for each integration: ```auto output_permissions: 97aad6d8-abc9-497c-8bf2-cb368540a96c: _elastic_agent_checks: cluster: - monitor 67f570f1-e169-4731-a4bd-51a1e1817beb: indices: - names: - logs-system.application-servers privileges: - auto_configure - create_doc - names: - logs-system.security-servers privileges: - auto_configure - create_doc - names: - logs-system.system-servers privileges: - auto_configure - create_doc ``` In this case this is a `system` integration, and the namespace is called `servers`, so if this is the only integration in the policy, these are the permissions that the agent will have, everything is tied up to the namespace, so a `reroute` processor to change the namespace would fail because the API Key being used does not have the permission to write into it. This applies to all integrations with the exception of some with _wildcard_ permissions, like the _AWS Custom Logs_ or _Custom Kafka Logs_ (and some others), those integrations have permissions into `logs-*-*`, so they can use `reroute` to change the dataset and namespace, that's how we can have logs from AWS EKS being parsed by the Kubernetes integration for example. So, in reality, it is not that is not possible for the user to create a custom datastream with a new dataset name for the Fortinet integration, it is just that it depends on what policies are present in the integration, this is like an unofficial workaround. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 16, 2025, 8:51pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/11 "2025-04-16T20:51:30Z")
@leandrojmp Agree... And the right people are looking at it BUT I think you CAN change the namespace (which again is not really the right solution) otherwise you could not do this .... ![Screenshot 2025-04-16 at 1.47.13 PM](https://us1.discourse-cdn.com/elastic/original/3X/4/3/430f0be419b69fc358a920dbd9a266ede9d9c2cb.png) ![Screenshot 2025-04-16 at 1.47.21 PM](https://us1.discourse-cdn.com/elastic/original/3X/3/0/308432f755184ac370ebe6f65ec6d4ad71c176fd.png) I have not kept up with all the is blocked / how etc... the last deep dive I did last year was based on API key sounds like that has / may have changed... Those system integration / indices I think are a bit different.... Well @juancamiloll looks like you are stuck for a while ☹ I would put your comments in the Issue @leandrojmp linked above ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 16, 2025, 9:02pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/12 "2025-04-16T21:02:15Z")
> [@stephenb](#): > > BUT I think you CAN change the namespace (which again is not really the right solution) otherwise you could not do this .... You can change the namespace in the configuration of the integration, but you cannot use a `reroute` processor in a custom ingest pipeline to change it. The permissions would be created after the integration is saved/created. If you create the integration and set the namespace as `prod`, you cannot have a `reroute` processor in the custom pipeline changing the namespace to `production` for example, you would need to edit it in the integration and change from `prod` into `production`, so a new API Key is created with the right permissions. The main issue here is that it is pretty common to have different requirements regarding data retention for firewall logs, you may be required to keep the system logs, which are the logs generated by the firewall device for a longer time than the traffic logs. Currently with an policy with just the Fortinet integration this is not possible because everything is stored in the same data stream and you have no option to split it in different data streams. It can be done, but it requires a couple of work and some workarounds, none of them are documented or are official. I'm planning to work on this next week, I can share my workaround on the Github issue until the integration is updated to have multiple data sets. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 16, 2025, 9:11pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/13 "2025-04-16T21:11:33Z")
> [@leandrojmp](#): > > You can change the namespace in the configuration of the integration, but you cannot use a `reroute` processor in a custom ingest pipeline to change it. I just did ... right this minute 🙂 Elastic 8.17.4 Fortinet FortiGate Firewall Logs 1.31.0 This is my ingest pipeline ```auto GET _ingest/pipeline/logs-fortinet_fortigate.log@custom { "logs-fortinet_fortigate.log@custom": { "processors": [ { "set": { "field": "data_stream.namespace", "value": "newnamespace" } }, { "reroute": {} } ] } } ``` Before / After Before Pipeline ```auto { "_index": ".ds-logs-fortinet_fortigate.log-default-2025.04.16-000001", "_id": "VTBpQJYBRkty1fMz3h7y", "_version": 1, "_source": { "agent": { "name": "stephenb-logginig-test", "id": "438a8475-1a44-4006-81e0-ea28f9b1e8a1", "type": "filebeat", "ephemeral_id": "27a11684-2ce9-4429-8d38-47dd85ed38c5", "version": "8.17.4" }, "log": { "file": { "path": "/home/azureuser/fortigate/fortigage.log" }, .... "related": { "ip": [ "10.1.100.66", "89.160.20.128", "172.16.200.11" ] }, "data_stream": { "namespace": "default", "type": "logs", "dataset": "fortinet_fortigate.log" }, ``` After ```auto { "_index": ".ds-logs-fortinet_fortigate.log-newnamespace-2025.04.16-000001", "_id": "NRxsQJYBq5b1-O15mZd_", "_version": 1, "_source": { "agent": { "name": "stephenb-logginig-test", "id": "438a8475-1a44-4006-81e0-ea28f9b1e8a1", "ephemeral_id": "27a11684-2ce9-4429-8d38-47dd85ed38c5", "type": "filebeat", "version": "8.17.4" }, "log": { "file": { "path": "/home/azureuser/fortigate/fortigage1.log" }, "offset": 24446, ..... ] }, "data_stream": { "namespace": "newnamespace", "type": "logs", "dataset": "fortinet_fortigate.log" }, ``` 🙂 Now the routing would need to be more specific... Like I said I think this is actually API Key based ![Screenshot 2025-04-16 at 2.12.04 PM](https://us1.discourse-cdn.com/elastic/original/3X/f/f/ff9da60f96a1acf0e90f5cf833ada845c647361e.png) So now if there are some fields we can key on we can route to namespace... ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 16, 2025, 9:29pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/14 "2025-04-16T21:29:24Z")
And now I made it write to a new data\_stream.data set 😃 ```auto PUT _ingest/pipeline/logs-fortinet_fortigate.log@custom { "processors": [ { "set": { "field": "data_stream.namespace", "value": "newnamespace" } }, { "set": { "field": "data_stream.dataset", "value": "fortinet_fortigate.log_custom" } }, { "reroute": {} } ] } ``` ![Screenshot 2025-04-16 at 2.27.57 PM](https://us1.discourse-cdn.com/elastic/original/3X/c/6/c6f814cb80a6a3cd09f682e56ac0751c1742e032.png) ![Screenshot 2025-04-16 at 2.28.54 PM](https://us1.discourse-cdn.com/elastic/original/3X/5/9/59bee039dba83758f08177cb5cb57aeda1f380a2.png) I have not done anything really special... ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 16, 2025, 9:43pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/15 "2025-04-16T21:43:14Z")
Weird, I haven't tried because it is a production system that I cannot keep doing those kind of tests, but it was mentioned in the Github issue that the `reroute` process would not work. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 16, 2025, 9:45pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/16 "2025-04-16T21:45:00Z")
This Policy just have the Fortinet integration? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 16, 2025, 9:54pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/17 "2025-04-16T21:54:03Z")
No there is an azure event hub system and fortigate... all really vanilla ... but I AM using one of the Catch All Input Event Hub one of the Azure Custom Logs .. maybe that I why I am getting the leniency. If I get a chance let me run it ... only... I will report back... that may be why.... ![Screenshot 2025-04-16 at 2.48.37 PM](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8bcbe5ec02896df08fc05e9a244b7f674b754e15.png) ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 16, 2025, 10:31pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/18 "2025-04-16T22:31:16Z")
> [@stephenb](#): > > but I AM using one of the Catch All Input Event Hub one of the Azure Custom Logs .. maybe that I why I am getting the leniency. Yeah, that's probably it, this _Custom Azure Logs_ integration seems to have permissions on `logs-*-*`, so the API Key used by this agent would end-up with this permission as well. This is the workaround I mentioned, add a integration with broader permissions to the same policy. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 16, 2025, 10:32pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/19 "2025-04-16T22:32:59Z")
I will do a quick check it will be good to know for future Where do you see the permissions defined? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 16, 2025, 10:38pm UTC](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211/20 "2025-04-16T22:38:01Z")
> [@stephenb](#): > > Where do you see the permissions defined? If you go into the policy page, in the Action button click on View Policy, it will list the policy, the integrations and the permissions for each integration. The documentation for this integration already mentions that you can set any dataset and namespace you want. [Next page](https://discuss.elastic.co/t/firewall-logs-to-different-datastream-by-type/377211.md?page=2)