Hello,
I am getting an error when trying to use some fields that apparently aren't being mapped correctly. The fields are source.ip, source.port, destination.ip, and destination.port
I have checked the mapping of those fields with two different indices. One of them is an index for syslogs, and the other is for fortigate firewall logs. It looks like the fields are being mapped correctly in the fortigate index (even though the screenshots above are from that index), but the syslog index has them mapped differently.
[user@server ~]$ curl -X GET -H "Authorization: ApiKey <redacted>" "https://localhost:9200/.ds-logs-fortinet_fortigate.log-default-2023.12.02-000002/_mapping/field/source.ip/" --insecure | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 145 100 145 0 0 4677 0 --:--:-- --:--:-- --:--:-- 4677
{
".ds-logs-fortinet_fortigate.log-default-2023.12.02-000002": {
"mappings": {
"source.ip": {
"full_name": "source.ip",
"mapping": {
"ip": {
"type": "ip"
}
}
}
}
}
}
[user@server ~]$ curl -X GET -H "Authorization: ApiKey <redacted>" "https://localhost:9200/.ds-logs-fortinet_fortigate.log-default-2023.12.02-000002/_mapping/field/source.port/" --insecure | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 153 100 153 0 0 4371 0 --:--:-- --:--:-- --:--:-- 4371
{
".ds-logs-fortinet_fortigate.log-default-2023.12.02-000002": {
"mappings": {
"source.port": {
"full_name": "source.port",
"mapping": {
"port": {
"type": "long"
}
}
}
}
}
}
[user@server ~]$ curl -X GET -H "Authorization: ApiKey <redacted>" "https://localhost:9200/.ds-logs-system.syslog-default-2023.11.16-000001/_mapping/f
ield/source.ip/" --insecure | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 161 100 161 0 0 5031 0 --:--:-- --:--:-- --:--:-- 5031
{
".ds-logs-system.syslog-default-2023.11.16-000001": {
"mappings": {
"source.ip": {
"full_name": "source.ip",
"mapping": {
"ip": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
}
}
[user@server ~]$ curl -X GET -H "Authorization: ApiKey <redacted>" "https://localhost:9200/.ds-logs-system.syslog-default-2023.11.16-000001/_mapping/field/source.port/" --insecure | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 167 100 167 0 0 5566 0 --:--:-- --:--:-- --:--:-- 5566
{
".ds-logs-system.syslog-default-2023.11.16-000001": {
"mappings": {
"source.port": {
"full_name": "source.port",
"mapping": {
"port": {
"type": "keyword",
"ignore_above": 1024
}
}
}
}
}
}
The issue began this morning, or at least that's when I noticed it. Yesterday I added a new log path to the system integration. I wanted to start ingesting logs from /var/log/firewalld so I added that path to the integration, deployed it to my agents, and made a custom pipeline that uses Grok to parse the log entries. The new fields that were created are named differently, and have a "firewalld" prefix so that they don't get confused with other fields. That worked out fine, and I managed to see the data that I wanted and it was being mapped correctly.
Any idea how I can fix this?
Thanks for reading.
