# Some of the Fortigate fields are not searchable?
**URL:** https://discuss.elastic.co/t/some-of-the-fortigate-fields-are-not-searchable/205371
**Category:** Elasticsearch
**Created:** [October 27, 2019, 12:59am UTC](https://discuss.elastic.co/t/some-of-the-fortigate-fields-are-not-searchable/205371 "2019-10-27T00:59:45Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Long\_Thai](https://discuss.elastic.co/u/Long_Thai)
#### Post date: [October 27, 2019, 12:59am UTC](https://discuss.elastic.co/t/some-of-the-fortigate-fields-are-not-searchable/205371/1 "2019-10-27T00:59:45Z")
I am using kv() to identify key/value pairs. The stream of inputs are from /var/log/syslog and filbeat.
I rename some of the fields like type, subtype, user, group, service to fgt\_type, fgt\_subtype, fgt\_user, fgt\_group, fgt\_service.
\<
kv {
source =\> "syslog\_message"
field\_split =\> ","
value\_split =\> "="
exclude\_keys =\> ["devid", "devname", "logid", "vd", "poluuid", "sessionid", "sentpkt","rcvdpkt", "crscore", "crlevel"]
}
if [syslog\_hostname] == "a.b.c.d" {
mutate {
rename =\> { "type" =\> "fgt\_type" }
rename =\> { "user" =\> "fgt\_user" }
rename =\> { "subtype" =\> "fgt\_subtype" }
add\_field =\> { "type" =\> "fortigate" }
rename =\> { "service" =\> "fgt\_service" }
rename =\> { "group" =\> "fgt\_group" }
rename =\> { "url" =\> "fgt\_url" }
remove\_field =\>["date","time","devname","devid","logid","vd","eventtime","sentpkt","rcvdpkt","poluuid","host.id"]
}
geoip {
source =\> "srcip"
}
}
/\>
But in Elasticsearch, we find that some of the Fortigate fields are not searchable such as user, group, etc.
When we look at the mappings between searchable fields and non-searchable fields, they look the same.
fgt\_type is searchable:
\<
GET /logstash-filebeat-rsyslog\_data-2019.10.25/\_mapping/field/fgt\_type
{
"logstash-filebeat-rsyslog\_data-2019.10.25" : {
"mappings" : {
"fgt\_type" : {
"full\_name" : "fgt\_type",
"mapping" : {
"fgt\_type" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore\_above" : 256
}
}
}
}
}
}
}
}
/\>
fgt\_user is not searchable
\<
GET /logstash-filebeat-rsyslog\_data-2019.10.25/\_mapping/field/fgt\_user
{
"logstash-filebeat-rsyslog\_data-2019.10.25" : {
"mappings" : {
"fgt\_user" : {
"full\_name" : "fgt\_user",
"mapping" : {
"fgt\_user" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore\_above" : 256
}
}
}
}
}
}
}
}
/\>
What should we do in order for fgt\_user or user becomes searchable in Elasticsearch ?
Thanks
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 27, 2019, 9:26am UTC](https://discuss.elastic.co/t/some-of-the-fortigate-fields-are-not-searchable/205371/2 "2019-10-27T09:26:27Z")
What is the query you run to determine that the field is not searchable? Can you show a document from that index that you would expect to match? Are you getting any error message?
---
### Author:  [@Long\_Thai](https://discuss.elastic.co/u/Long_Thai)
#### Post date: [October 27, 2019, 12:38pm UTC](https://discuss.elastic.co/t/some-of-the-fortigate-fields-are-not-searchable/205371/3 "2019-10-27T12:38:33Z")
Hi,
In the KQL box, I tried to search for fgt\_user:\* and it gives the results in the attached fgt\_user\_table file.
If I look through all the fields, it indicates that key fg\_user as "?" which says "unable to filter for presence of meta fields"
I list here the message where all key value pairs are in son format:
\<
{
"\_index": "logstash-filebeat-rsyslog\_data-2019.10.27" ,
"\_type": "\_doc" ,
"\_id": "074mDW4BuJS9LxnKYIIb" ,
"\_version": 1 ,
"\_score": null ,
"\_source" : {
"policytype": "policy" ,
"@timestamp": "2019-10-27T12:17:28.010Z" ,
"fgt\_type": "traffic" ,
"type": "fortigate" ,
"rcvdbyte": "226529" ,
"message": "Oct 27 12:17:27 10.120.120.1 date=2019-10-27,time=08: 17:27,devname="Bell-DC-Forti501E-Firewall",devid="FG5H1E5818903643",logid="0000000013",type="traffic",subtype="forward",level="notice",vd="root",eventtime=1572178647,srcip=x,srcport=64407,srcintf="ssl.root",srcintfrole="undefined",dstip=x,dstport=5601,dstintf="port9",dstintfrole="lan",poluuid="cf8a78dc-ce38-51e8-f16c-30a888cd6da6",sessionid=2190243146,proto=6,action="close",user="lthai",group="UDN-Montreal-Radius",authserver="Radius server",policyid=3,policytype="policy",service="tcp/5601",dstcountry="Reserved",srccountry="Reserved",trandisp="snat",transip=x,transport=64407,duration=126,sentbyte=16251,rcvdbyte=226529,sentpkt=148,rcvdpkt=185,appcat="unscanned",sentdelta=0,rcvddelta=0,dstdevtype="Router/NAT Device",dstdevcategory="None",masterdstmac="2c:23:3a:35:61:22",dstmac="2c:23:3a:35:61:22",dstserver=0" ,
"received\_at": "2019-10-27T12:17:28.010Z" ,
"received\_from": "{"name":"d0940d72625b","hostname":"d0940d72625b","os":{"name":"CentOS Linux","family":"redhat","version":"7 (Core)","platform":"centos","kernel":"4.4.0-166-generic","codename":"Core"},"containerized":true,"architecture":"x86\_64"}" ,
"srcport": "64407" ,
"log" : {
"offset": 123797723 ,
"file" : {
"path": "/mnt/log/syslog"
}
},
"dstport": "5601" ,
"srccountry": "Reserved" ,
"syslog\_timestamp": "Oct 27 12:17:27" ,
"srcip": "x" ,
"proto": "6" ,
"srcintf": "ssl.root" ,
"fgt\_user": "lthai" ,
"authserver": "Radius server" ,
"level": "notice" ,
"trandisp": "snat" ,
"dstintfrole": "lan" ,
"appcat": "unscanned" ,
"dstmac": "2c:23:3a:35:61:22" ,
"syslog\_message": "17:27,devname="Bell-DC-Forti501E-Firewall",devid="FG5H1E5818903643",logid="0000000013",type="traffic",subtype="forward",level="notice",vd="root",eventtime=1572178647,srcip=x,srcport=64407,srcintf="ssl.root",srcintfrole="undefined",dstip=x,dstport=5601,dstintf="port9",dstintfrole="lan",poluuid="cf8a78dc-ce38-51e8-f16c-30a888cd6da6",sessionid=2190243146,proto=6,action="close",user="lthai",group="yyy",authserver="Radius server",policyid=3,policytype="policy",service="tcp/5601",dstcountry="Reserved",srccountry="Reserved",trandisp="snat",transip=x,transport=64407,duration=126,sentbyte=16251,rcvdbyte=226529,sentpkt=148,rcvdpkt=185,appcat="unscanned",sentdelta=0,rcvddelta=0,dstdevtype="Router/NAT Device",dstdevcategory="None",masterdstmac="2c:23:3a:35:61:22",dstmac="2c:23:3a:35:61:22",dstserver=0" ,
"@version": "1" ,
"transip": "x" ,
"policyid": "3" ,
"dstdevcategory": "None" ,
"dstserver": "0" ,
"dstdevtype": "Router/NAT Device" ,
"dstintf": "port9" ,
"tags" : [
"fortigate\_fb" ,
"beats\_input\_codec\_plain\_applied" ,
"\_geoip\_lookup\_failure"
],
"geoip" : {},
"agent" : {
"hostname": "d0940d72625b" ,
"type": "filebeat" ,
"ephemeral\_id": "3b86d83f-d3b4-4f28-8b63-7a2454732121" ,
"id": "bdd69604-f34d-4d04-86ec-7e8721d2bf9e" ,
"version": "7.4.0"
},
"ecs" : {
"version": "1.1.0"
},
"input" : {
"type": "log"
},
"action": "close" ,
"sentbyte": "16251" ,
"fgt\_group": "UDN-Montreal-Radius" ,
"host" : {
"hostname": "d0940d72625b" ,
"name": "d0940d72625b" ,
"os" : {
"platform": "centos" ,
"kernel": "4.4.0-166-generic" ,
"family": "redhat" ,
"codename": "Core" ,
"version": "7 (Core)" ,
"name": "CentOS Linux"
},
"containerized": true ,
"architecture": "x86\_64"
},
"syslog\_hostname": "x" ,
"dstip": "x" ,
"dstcountry": "Reserved" ,
"srcintfrole": "undefined" ,
"transport": "64407" ,
"duration": "126" ,
"syslog\_program": "date=2019-10-27,time=08" ,
"sentdelta": "0" ,
"rcvddelta": "0" ,
"fgt\_subtype": "forward" ,
"fgt\_service": "tcp/5601" ,
"masterdstmac": "2c:23:3a:35:61:22"
},
"fields" : {
"received\_at" : [
"2019-10-27T12:17:28.010Z"
],
"@timestamp" : [
"2019-10-27T12:17:28.010Z"
]
},
"sort" : [
1572178648010
]
}
/\>
Thanks
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 27, 2019, 1:05pm UTC](https://discuss.elastic.co/t/some-of-the-fortigate-fields-are-not-searchable/205371/4 "2019-10-27T13:05:24Z")
Have you refreshed the index pattern in Kibana?
---
### Author:  [@Long\_Thai](https://discuss.elastic.co/u/Long_Thai)
#### Post date: [October 27, 2019, 1:22pm UTC](https://discuss.elastic.co/t/some-of-the-fortigate-fields-are-not-searchable/205371/5 "2019-10-27T13:22:15Z")
Yes I did and those fields that I mentioned remained non-searchable ?
---
### Author:  [@Long\_Thai](https://discuss.elastic.co/u/Long_Thai)
#### Post date: [October 27, 2019, 3:43pm UTC](https://discuss.elastic.co/t/some-of-the-fortigate-fields-are-not-searchable/205371/6 "2019-10-27T15:43:02Z")
I forgot to put this block after the grok
\<
date {
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
}
/\>
So everything is working now.
```
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
}
```
Thanks
---
### Author:  [@mandakh](https://discuss.elastic.co/u/mandakh)
#### Post date: [November 20, 2019, 10:49am UTC](https://discuss.elastic.co/t/some-of-the-fortigate-fields-are-not-searchable/205371/7 "2019-11-20T10:49:51Z")
ISSUE\>\>
I have error with following script. Because of Elasticsearch 7 mapping types deprecated.
Could you guys help me to convert script to work on ES7
curl -X PUT "127.0.0.1:9200/\_template/template\_forti" -H 'Content-Type: application/json' -d'
{
"version" : 50002,
"order" : 1,
"template" : "fortinet-_",
"settings" : {
"index" : {
"refresh\_interval" : "5s"
}
},
"mappings" : {
"default" : {
"\_all" : {
"enabled" : true,
"norms" : false
},
"dynamic\_templates" : [
{
"message\_field" : {
"path\_match" : "message",
"match\_mapping\_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
},
{
"string\_fields" : {
"match" : "_",
"match\_mapping\_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore\_above" : 256
}
}
}
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date",
"include\_in\_all" : false
},
"@version" : {
"type" : "keyword",
"include\_in\_all" : false
},
"geoip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"location" : {
"type" : "geo\_point"
},
"latitude" : {
"type" : "half\_float"
},
"longitude" : {
"type" : "half\_float"
}
}
},
"geodstip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"location" : {
"type" : "geo\_point"
},
"latitude" : {
"type" : "half\_float"
},
"longitude" : {
"type" : "half\_float"
}
}
},
"geosrcip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"location" : {
"type" : "geo\_point"
},
"latitude" : {
"type" : "half\_float"
},
"longitude" : {
"type" : "half\_float"
}
}
}
}
}
},
"aliases" : { }
}
'
SOURCE\>\>
> **[Fortigate Firewall ELK Configuration](https://webcache.googleusercontent.com/search?q=cache%3A1JG4NBDuRMEJ%3Ahttps%3A%2F%2Fblog.joshellis.nz%2Ffortigate-firewall-elk-configuration%2F%2B&cd=1&hl=en&ct=clnk&gl=nz)**
>
> I was interested to visualize some of the traffic flows from my Fortigate
> Firewall at home. FortiAnalyzer has some of this functionality but I wanted to
> learn a bit about the ELK Stack so I thought this would be a fun little project
> to get me...
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 18, 2019, 10:49am UTC](https://discuss.elastic.co/t/some-of-the-fortigate-fields-are-not-searchable/205371/8 "2019-12-18T10:49:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.