# Field is of the wrong type **URL:** \ **Category:** Logstash\ **Created:** [December 6, 2023, 9:05pm UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762 "2023-12-06T21:05:28Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![soad20000](https://avatars.discourse-cdn.com/v4/letter/s/c6cbf5/32.png) [@soad20000](https://discuss.elastic.co/u/soad20000)\ **Post date:** [December 6, 2023, 9:05pm UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762/1 "2023-12-06T21:05:28Z")
Hello, I am getting an error when trying to use some fields that apparently aren't being mapped correctly. The fields are source.ip, source.port, destination.ip, and destination.port ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/1/a1c1ffd3b58d9aac9a8b98091b0bbb41262ef505.png) I have checked the mapping of those fields with two different indices. One of them is an index for syslogs, and the other is for fortigate firewall logs. It looks like the fields are being mapped correctly in the fortigate index (even though the screenshots above are from that index), but the syslog index has them mapped differently. ```auto [user@server ~]$ curl -X GET -H "Authorization: ApiKey " "https://localhost:9200/.ds-logs-fortinet_fortigate.log-default-2023.12.02-000002/_mapping/field/source.ip/" --insecure | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 145 100 145 0 0 4677 0 --:--:-- --:--:-- --:--:-- 4677 { ".ds-logs-fortinet_fortigate.log-default-2023.12.02-000002": { "mappings": { "source.ip": { "full_name": "source.ip", "mapping": { "ip": { "type": "ip" } } } } } } [user@server ~]$ curl -X GET -H "Authorization: ApiKey " "https://localhost:9200/.ds-logs-fortinet_fortigate.log-default-2023.12.02-000002/_mapping/field/source.port/" --insecure | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 153 100 153 0 0 4371 0 --:--:-- --:--:-- --:--:-- 4371 { ".ds-logs-fortinet_fortigate.log-default-2023.12.02-000002": { "mappings": { "source.port": { "full_name": "source.port", "mapping": { "port": { "type": "long" } } } } } } [user@server ~]$ curl -X GET -H "Authorization: ApiKey " "https://localhost:9200/.ds-logs-system.syslog-default-2023.11.16-000001/_mapping/f ield/source.ip/" --insecure | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 161 100 161 0 0 5031 0 --:--:-- --:--:-- --:--:-- 5031 { ".ds-logs-system.syslog-default-2023.11.16-000001": { "mappings": { "source.ip": { "full_name": "source.ip", "mapping": { "ip": { "type": "keyword", "ignore_above": 1024 } } } } } } [user@server ~]$ curl -X GET -H "Authorization: ApiKey " "https://localhost:9200/.ds-logs-system.syslog-default-2023.11.16-000001/_mapping/field/source.port/" --insecure | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 167 100 167 0 0 5566 0 --:--:-- --:--:-- --:--:-- 5566 { ".ds-logs-system.syslog-default-2023.11.16-000001": { "mappings": { "source.port": { "full_name": "source.port", "mapping": { "port": { "type": "keyword", "ignore_above": 1024 } } } } } } ``` The issue began this morning, or at least that's when I noticed it. Yesterday I added a new log path to the system integration. I wanted to start ingesting logs from /var/log/firewalld so I added that path to the integration, deployed it to my agents, and made a custom pipeline that uses Grok to parse the log entries. The new fields that were created are named differently, and have a "firewalld" prefix so that they don't get confused with other fields. That worked out fine, and I managed to see the data that I wanted and it was being mapped correctly. Any idea how I can fix this? Thanks for reading. ---
**Author:** ![yago82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yago82/32/97755_2.png) [@yago82](https://discuss.elastic.co/u/yago82)\ **Post date:** [December 7, 2023, 8:55am UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762/2 "2023-12-07T08:55:27Z")
Hi, Here's a general outline of the steps you'll need to take: 1. Create a new index with the correct mappings. For example, if you want `source.ip` to be of type `ip` and `source.port` to be of type `long` in all indices, you should specify these types in the mappings when creating the new index. 2. Reindex your data from the old index to the new index. This will copy all the documents from the old index to the new index, while applying the new mappings. 3. Delete the old index. Regards ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 7, 2023, 12:31pm UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762/3 "2023-12-07T12:31:24Z")
> [@soad20000](#): > > It looks like the fields are being mapped correctly in the fortigate index (even though the screenshots above are from that index), but the syslog index has them mapped differently. This is expected as the syslog dataset of the system integration does not have any mapping for source or destination fields. You can check the list of exported fields for each dataset in the [documentation](https://docs.elastic.co/integrations/system#syslog), if you want to add a field that it is not on this list, you will need to create it mapping first in the `@custom` component template of the integration and force a rollover before indexing any data with the new fields. > [@soad20000](#): > > The new fields that were created are named differently, and have a "firewalld" prefix so that they don't get confused with other fields. That worked out fine, and I managed to see the data that I wanted and it was being mapped correctly. Did you created the mapping for the `firewalld` fields? ---
**Author:** ![soad20000](https://avatars.discourse-cdn.com/v4/letter/s/c6cbf5/32.png) [@soad20000](https://discuss.elastic.co/u/soad20000)\ **Post date:** [December 7, 2023, 4:15pm UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762/4 "2023-12-07T16:15:16Z")
Hey, Ok I will try this out but I just wanted some clarification on something. > [@yago82](#): > > 1. Create a new index with the correct mappings. For example, if you want `source.ip` to be of type `ip` and `source.port` to be of type `long` in all indices, you should specify these types in the mappings when creating the new index. So mappings have to match across all indices? If I want source.ip to be of type ip like it was before, I have to add that mapping to every index? Or just the indexes where source.ip is being mapped explicitly? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 7, 2023, 4:27pm UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762/5 "2023-12-07T16:27:32Z")
> [@soad20000](#): > > So mappings have to match across all indices? If I want source.ip to be of type ip like it was before, I have to add that mapping to every index? Or just the indexes where source.ip is being mapped explicitly? If you are using Elastic Agent and integrations, not all integrations will have the same fields mapped, for example the Fortinet integration has the mapping for the `source.*` and `destination.*` fields, but the System integration does not have mappings for those fields because they are not normally present. If you want to add a field to an integration where there is no mapping for this field, you need to create a custom component template with the correct mapping, then after that you need to force a rollover on the datastream of this integration so a new index with the mapping is created, after that you can index new documents with the specific field you created the mapping. ---
**Author:** ![soad20000](https://avatars.discourse-cdn.com/v4/letter/s/c6cbf5/32.png) [@soad20000](https://discuss.elastic.co/u/soad20000)\ **Post date:** [December 7, 2023, 4:56pm UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762/6 "2023-12-07T16:56:28Z")
Ok I have added the mappings to the custom syslog component template like so: ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/a/4a98b12959f888bc6a390635156eb5f7617c077c.png) And the `logs-fortinet_fortigate.log@package` component template appears to be mapping destination.ip to type ip. ```auto "destination": { "properties": { "geo": { "properties": { "continent_name": { "ignore_above": 1024, "type": "keyword" }, "region_iso_code": { "ignore_above": 1024, "type": "keyword" }, "city_name": { "ignore_above": 1024, "type": "keyword" }, "country_iso_code": { "ignore_above": 1024, "type": "keyword" }, "country_name": { "ignore_above": 1024, "type": "keyword" }, "name": { "ignore_above": 1024, "type": "keyword" }, "location": { "type": "geo_point" }, "region_name": { "ignore_above": 1024, "type": "keyword" } } }, "nat": { "properties": { "port": { "type": "long" }, "ip": { "type": "ip" } } }, "as": { "properties": { "number": { "type": "long" }, "organization": { "properties": { "name": { "ignore_above": 1024, "type": "keyword", "fields": { "text": { "type": "match_only_text" } } } } } } }, "address": { "ignore_above": 1024, "type": "keyword" }, "port": { "type": "long" }, "bytes": { "type": "long" }, "domain": { "ignore_above": 1024, "type": "keyword" }, "ip": { "type": "ip" ``` I'm still trying to understand why the change I made to the syslog integration affected the fortigate integration though. If I understand correctly, now I just need to create a new index for syslog after forcing the rollover on its datastream? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 7, 2023, 5:00pm UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762/7 "2023-12-07T17:00:20Z")
> [@soad20000](#): > > I'm still trying to understand why the change I made to the syslog integration affected the fortigate integration though. Do you have any evidence of this? What you shared in the first post shows that the type of the source.ip and source.port in the Fortinet datastream are correct. It is not clear where is the screenshot you shared, if this is from a dashboard that queries both the Fortinet and System datastream, then the fact that the System datastream has the wrong mapping could lead to this error. > [@soad20000](#): > > If I understand correctly, now I just need to create a new index for syslog after forcing the rollover on its datastream? No, you do not create anything, forcing a rollover will automatically do that creating a new backing index, you just need to add the mapping to the field in the custom template. ---
**Author:** ![soad20000](https://avatars.discourse-cdn.com/v4/letter/s/c6cbf5/32.png) [@soad20000](https://discuss.elastic.co/u/soad20000)\ **Post date:** [December 7, 2023, 5:28pm UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762/8 "2023-12-07T17:28:36Z")
Yeah I am sorry about that, I'll try to be more clear. Here is one of the dashboards that was affected. ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/2/a24ad7aaf0a23e9e0b19ae820b8db086ef52afca.png) So this dashboard was created before I made that change to the system integration and it was working fine. I am guessing that my mistake is that the dashboard is querying both the fortinet and the system datastream. In fact, I can't see source.ip/port or destination.ip/port as fields in forti, they must have come from the system datastream. This is one of the grok expressions I added to the the custom syslog pipeline I made. Specifically to parse logs from firewalld. ```auto %{WORD:firewalld.action}: IN=%{DATA:firewalld.in_interface} OUT=%{DATA:firewalld.out_interface} MAC=%{DATA:firewalld.mac_src} SRC=%{IP:firewalld.src_ip} DST=%{IP:firewalld.dst_ip} LEN=%{INT:firewalld.len} TOS=%{DATA:firewalld.tos} PREC=%{DATA:firewalld.prec} TTL=%{INT:firewalld.ttl} ID=%{INT:firewalld.id} PROTO=%{WORD:firewalld.proto} SPT=%{INT:firewalld.spt} DPT=%{INT:firewalld.dpt} ``` I think the second mistake I made was thinking that the grok expression was going to take care of the mapping. The datastream `logs-system.syslog-default` is using the template `so-logs-system.syslog` This template doesn't have any mappings defined, but in the "preview" section I can see what looks like mappings to me. Where does it get these mappings from? I can see that the mapping I just made with the `firewalld` values appears in there. ```auto { "template": { "settings": { "index": { "lifecycle": { "name": "so-logs-system.syslog-logs" }, "codec": "best_compression", "routing": { "allocation": { "include": { "_tier_preference": "data_hot" } } }, "mapping": { "total_fields": { "limit": "10000" }, "ignore_malformed": "true" }, "final_pipeline": ".fleet_final_pipeline-1", "query": { "default_field": [ "cloud.account.id", "cloud.availability_zone", "cloud.instance.id", "cloud.instance.name", "cloud.machine.type", "cloud.provider", "cloud.region", "cloud.project.id", "cloud.image.id", "container.id", "container.image.name", "container.name", "host.os.version", "host.os.build", "host.os.codename", "host.os.family", "host.os.full", "host.os.kernel", "host.os.name", "host.os.platform", "host.type", "host.architecture", "host.domain", "host.hostname", "host.id", "host.mac", "host.name", "event.action", "event.category", "event.code", "event.kind", "event.outcome", "event.provider", "event.type", "ecs.version", "message", "process.name", "tags" ] }, "default_pipeline": "logs-system.syslog-1.43.0", "number_of_replicas": "0" } }, "mappings": { "_meta": { "managed_by": "security_onion", "managed": true }, "dynamic_templates": [ { "container.labels": { "path_match": "container.labels.*", "match_mapping_type": "string", "mapping": { "type": "keyword" } } }, { "strings_as_keyword": { "match_mapping_type": "string", "mapping": { "ignore_above": 1024, "type": "keyword" } } } ], "date_detection": false, "properties": { "@timestamp": { "type": "date", "ignore_malformed": false }, "cloud": { "properties": { "account": { "properties": { "id": { "type": "keyword", "ignore_above": 1024 } } }, "availability_zone": { "type": "keyword", "ignore_above": 1024 }, "image": { "properties": { "id": { "type": "keyword", "ignore_above": 1024 } } }, "instance": { "properties": { "id": { "type": "keyword", "ignore_above": 1024 }, "name": { "type": "keyword", "ignore_above": 1024 } } }, "machine": { "properties": { "type": { "type": "keyword", "ignore_above": 1024 } } }, "project": { "properties": { "id": { "type": "keyword", "ignore_above": 1024 } } }, "provider": { "type": "keyword", "ignore_above": 1024 }, "region": { "type": "keyword", "ignore_above": 1024 } } }, "container": { "properties": { "id": { "type": "keyword", "ignore_above": 1024 }, "image": { "properties": { "name": { "type": "keyword", "ignore_above": 1024 } } }, "name": { "type": "keyword", "ignore_above": 1024 } } }, "data_stream": { "properties": { "dataset": { "type": "constant_keyword" }, "namespace": { "type": "constant_keyword" }, "type": { "type": "constant_keyword", "value": "logs" } } }, "destination": { "properties": { "ipv6": { "type": "keyword", "ignore_above": 1024 } } }, "ecs": { "properties": { "version": { "type": "keyword", "ignore_above": 1024 } } }, "event": { "properties": { "action": { "type": "keyword", "ignore_above": 1024 }, "agent_id": { "type": "keyword", "ignore_above": 1024 }, "agent_id_status": { "type": "keyword", "ignore_above": 1024 }, "category": { "type": "keyword", "ignore_above": 1024 }, "code": { "type": "keyword", "ignore_above": 1024 }, "created": { "type": "date" }, "dataset": { "type": "constant_keyword", "value": "system.syslog" }, "duration": { "type": "long" }, "end": { "type": "date" }, "hash": { "type": "keyword", "ignore_above": 1024 }, "id": { "type": "keyword", "ignore_above": 1024 }, "ingested": { "type": "date", "format": "strict_date_time_no_millis||strict_date_optional_time||epoch_millis" }, "kind": { "type": "keyword", "ignore_above": 1024 }, "module": { "type": "constant_keyword", "value": "system" }, "original": { "type": "keyword", "index": false, "doc_values": false }, "outcome": { "type": "keyword", "ignore_above": 1024 }, "provider": { "type": "keyword", "ignore_above": 1024 }, "reason": { "type": "keyword", "ignore_above": 1024 }, "reference": { "type": "keyword", "ignore_above": 1024 }, "risk_score": { "type": "float" }, "risk_score_norm": { "type": "float" }, "sequence": { "type": "long" }, "severity": { "type": "long" }, "start": { "type": "date" }, "timezone": { "type": "keyword", "ignore_above": 1024 }, "type": { "type": "keyword", "ignore_above": 1024 }, "url": { "type": "keyword", "ignore_above": 1024 } } }, "firewalld": { "properties": { "dpt": { "type": "long" }, "dst_ip": { "type": "ip" }, "spt": { "type": "long" }, "src_ip": { "type": "ip" } } }, "host": { "properties": { "architecture": { "type": "keyword", "ignore_above": 1024 }, "containerized": { "type": "boolean" }, "domain": { "type": "keyword", "ignore_above": 1024 }, "hostname": { "type": "keyword", "ignore_above": 1024 }, "id": { "type": "keyword", "ignore_above": 1024 }, "ip": { "type": "ip" }, "mac": { "type": "keyword", "ignore_above": 1024 }, "name": { "type": "keyword", "ignore_above": 1024 }, "os": { "properties": { "build": { "type": "keyword", "ignore_above": 1024 }, "codename": { "type": "keyword", "ignore_above": 1024 }, "family": { "type": "keyword", "ignore_above": 1024 }, "full": { "type": "keyword", "ignore_above": 1024, "fields": { "text": { "type": "match_only_text" } } }, "kernel": { "type": "keyword", "ignore_above": 1024 }, "name": { "type": "keyword", "ignore_above": 1024, "fields": { "text": { "type": "match_only_text" } } }, "platform": { "type": "keyword", "ignore_above": 1024 }, "version": { "type": "keyword", "ignore_above": 1024 } } }, "type": { "type": "keyword", "ignore_above": 1024 } } }, "message": { "type": "match_only_text" }, "network": { "properties": { "initiated": { "type": "keyword", "ignore_above": 1024 } } }, "process": { "properties": { "name": { "type": "keyword", "ignore_above": 1024, "fields": { "text": { "type": "match_only_text" } } }, "pid": { "type": "long" } } }, "tags": { "type": "keyword", "ignore_above": 1024 } } }, "aliases": {} } } ``` Thank you for your patience, I have a lot to learn still. ---
**Author:** ![soad20000](https://avatars.discourse-cdn.com/v4/letter/s/c6cbf5/32.png) [@soad20000](https://discuss.elastic.co/u/soad20000)\ **Post date:** [December 7, 2023, 9:09pm UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762/9 "2023-12-07T21:09:16Z")
So again, I've checked the mapping of the index ` _index .ds-logs-fortinet_fortigate.log-default-2023.12.02-000002` ```auto { ".ds-logs-fortinet_fortigate.log-default-2023.12.02-000002": { "mappings": { "source.ip": { "full_name": "source.ip", "mapping": { "ip": { "type": "ip" } } } } } } ``` And it says right there that it's being mapped correctly. Same with this index `_index .ds-logs-system.security-default-2023.11.17-000001` ```auto { ".ds-logs-system.security-default-2023.11.17-000001": { "mappings": { "source.ip": { "full_name": "source.ip", "mapping": { "ip": { "type": "ip" } } } } } } ``` But I still get the same error. ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/8/c862948ed827f1ee1084383ba6ae56d554e7dc36.png) I have around 150 different indices, do I have to configure the mapping for each one? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 7, 2023, 9:45pm UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762/10 "2023-12-07T21:45:17Z")
> [@soad20000](#): > > I have around 150 different indices, do I have to configure the mapping for each one? Yes, a mapping changing will only affects **new** indices, if you have the field `source.ip` in multiple indices that are used by the same data view in Kibana, then you need to map the field `source.ip` as an `ip` in every one of those indices, normally you would change the template that applies to each index to fix the mapping. For data that was indexed with the wrong mapping, you would need to create a runtime field to fix the mapping and make kibana stone complaining about mapping conflicts. You can find the index that have conflicting mappings going to Advanced Settings -\> Data Views -\> Your data view, and filtering by the conflict type, this will show you the index where the `source.ip` is not an ip field. Then you can create a runtime field for each one of those indices: ```auto PUT index_name/_mapping { "runtime": { "source.ip": { "type": "ip" }, "source.port": { "type": "long" }, } } ``` ---
**Author:** ![soad20000](https://avatars.discourse-cdn.com/v4/letter/s/c6cbf5/32.png) [@soad20000](https://discuss.elastic.co/u/soad20000)\ **Post date:** [December 7, 2023, 10:57pm UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762/11 "2023-12-07T22:57:50Z")
> [@leandrojmp](#): > > > [@soad20000](#): > > > > I have around 150 different indices, do I have to configure the mapping for each one? > > You can find the index that have conflicting mappings going to Advanced Settings -\> Data Views -\> Your data view, and filtering by the conflict type, this will show you the index where the `source.ip` is not an ip field. > > Then you can create a runtime field for each one of those indices: > > ```auto > PUT index_name/_mapping > { > "runtime": { > "source.ip": { "type": "ip" }, > "source.port": { "type": "long" }, > } > } > > ``` Thank you SO much. Being able to see which indices had mapping conflicts was a huge help, I thought I was going to have to go through each one manually. I created the runtime field as indicated and it works perfectly now. As suspected, the conflict was in one of the system indices. Seriously can't thank you enough. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 4, 2024, 10:57pm UTC](https://discuss.elastic.co/t/field-is-of-the-wrong-type/348762/12 "2024-01-04T22:57:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.