Mappings are "not_analyzed" Can it be set with manage_template

Elastic Stack Logstash
1

I am importing logs from FireSight(cisco) into Logstash and outputting to Elasticsearch. Here is my conf file from Logstash.

input {
udp {
port => 5545
type => "firesight"
}
}

filter {
grok {
patterns_dir => ["./patterns"]
match => { "message" => "%{SYSLOGBASE} Protocol: %{WORD:protocol}, SrcIP: %{IP:srcIP}, DstIP: %{IP:dstIP}, SrcPort: %{NUMBER:srcPort}, DstPort: %{NUMBER:dstPort}, TCPFlags: %{WORD:tcpFlags}, IngressInterface: %{WORD:ingressINT}, EgressInterface: %{WORD:egressINT}, IngressZone: %{WORD:ingressZone}, EgressZone: %{WORD:egressZone}, DE: %{DATA:primaryDE}, Policy: %{WORD:policy}, ConnectType: %{WORD:connectType}, AccessControlRuleName: %{DATA:aclRuleName}, AccessControlRuleAction: %{WORD:aclRuleAction}, UserName: %{WORD:username}, UserAgent: %{DATA:userAgent}, Client: %{WORD:client}, ClientVersion: %{DATA:clientVer}, ApplicationProtocol: %{WORD:appProtocol}, InitiatorPackets: %{NUMBER:initPackets}, ResponderPackets: %{NUMBER:responderPackets}, InitiatorBytes: %{NUMBER:initBytes}, ResponderBytes: %{NUMBER:respondBytes}, NAPPolicy: %{DATA:NaPPolicy}, DNSResponseType: %{DATA:dnsResponseType}, Sinkhole: %{WORD:sinkhole}, HTTPReferer: %{URI:httpReferer}, ReferencedHost: %{DATA:referencedHost}, URLCategory: %{DATA:urlCategory}, URLReputation: %{DATA:urlReputation}, URL: %{URI:URL}"}
match => { "message" => "%{FIREPOWERBASE} ?(UserName: %{DATA:username})?,? ?(UserAgent: %{DATA:UserAgent})?,? Client: %{DATA:client}, ?(ClientVersion: %{DATA:ClientVersion})?,? ApplicationProtocol: %{WORD:appProtocol}, ?(WebApplication: %{DATA:WebApp})?,? %{FPPACKETRESPONSE} DNSResponseType: %{DATA:dnsResponseType}, Sinkhole: %{WORD:sinkhole}, ?(HTTPReferer: %{URI:httpReferer})?,? ?(ReferencedHost: %{DATA:ReferencedHost})?,? URLCategory: %{DATA:urlCategory}, URLReputation: %{DATA:urlReputation}, URL: %{URI:url}"}
match => { "message" => "%{FIREPOWERBASE} Client: %{DATA:client}, ApplicationProtocol: %{DATA:appProtocol}, %{FPPACKETRESPONSE} DNSResponseType: %{DATA:dnsResponseType}, Sinkhole: %{WORD:sinkhole}, URLCategory: %{DATA:urlCategory}, URLReputation: %{DATA:urlRep}"}
}
}

output {
elasticsearch {
hosts => ["10.50.184.119", "10.50.184.120", "10.50.184.121"]
index => "firesight-%{+YYYY.MM.dd}"
}
stdout {}
}

I am trying to use the Visualize tab in Kibana and its coming back with the "not analyzed". I know i need to setup some mappings for the strings in Elastic. My question is can I use the "managed_template" field in Logstash to achieve this?

I would switch the conf file in Logstash output to add this:

managed_template => true

Would this force it to use the default template of Logstash, which adds the .raw, or do I need to create my own template with custom mapping and point the output to that?

2

That's a pretty horrible grok pattern, why not use the KV filter - Kv filter plugin | Logstash Reference [8.11] | Elastic

That aside;

Yes that should do what you want :slight_smile:

3

I am trying to use the Visualize tab in Kibana and its coming back with the "not analyzed".

Really? I'm pretty sure ES defaults to analyzed strings.

I would switch the conf file in Logstash output to add this:

managed_template => true

It's manage_template not managed_template and it already defaults to true.

Would this force it to use the default template of Logstash, which adds the .raw,

Logstash's default template only applies to logstash-* indexes, but your indexes are firesight-*.

or do I need to create my own template with custom mapping and point the output to that?

You need to create your own template and either point Logstash to it (with manage_template => true) or manage the template yourself outside of Logstash (with manage_template => false) .

4

Their went a month of my life. The KV filter is light years better than what I was doing. Thanks. I ended having to use

filter {
kv {
value_split => ":"
}
}

5

Thanks. I'll have to end up creating my own template. The design we would like to have is a different index names for different network devices getting logged.