Field missing in Elasticsearch/kibana

Elastic Stack Logstash
1

Hi !

So I made a huge logstash config for my wifi hotspots.
I managed to create nearly all the filters but I have a problem with one of them.

I receive this log:

<134>1 1473769601.251850936 APVDA001 flows allow src=172.27.7.238 dst=64.233.166.188 mac=4C:34:88:01:2B:5B protocol=tcp sport=55267 dport=5228

And I put this grok filter:

%{SYSLOG5424PRI}%{INT} %{NUMBER:timestamp_unix} %{WORD:borne} flows %{WORD:flow_type} src=%{IP:ip_source} dst=%{IP:ip_dest} mac=%{MAC:mac} protocol=%{WORD:protocol} sport=%{NUMBER:sport} dport=%{NUMBER:dport}

It works well with a grok debugger but some fields do not appear in Kibana: the fields ip_source, ip_dest, mac, protocol, sport and dport.

Why ? It parses well the others !

Thanks

2

Have you confirmed that Logstash is extracting the fields properly? So it's just Kibana which is not displaying them?

3

Well logstash extracts borne and flow_type so I think that it extracts fields properly.

How do I check if it works well ?

If logtash works well, then yes, it's a problem with Kibana (or maybe with elasticsearch ?).
On kibana I have the field borne and the field flow_type so why not the others ?

4

Did you try to refresh the field lists in kibana?
In kibana, Go to settings >> Indices
Select which index you are having trouble with and use the Yellow colored 'Refresh' button to refresh the fields list.

Check if this helps.

5

Yes I did it and I still have the problem...

I'll try a logtash config with just the grok for this log and see if it works.

EDIT: with just this grok pattern, it works. I have all the fields.
Maybe another grok pattern matches first ? (I checked my config file and I see nothing..)

EDIT 2: Here are the two config files I use, I hope it'll help (the cisco one is huge)

10-cisco.conf:

10-dhcp.conf:

EDIT 3: I FOUND THE ERROR.

In fact there is a sense of priority in grok patterns. I commented 2 patterns and it works again.
Those 2 patterns had the same begin pattern and nothinf after.
Now I'll see how to have those 2 patterns back. I don't want any grokparse failure

EDIT 4:
Everything work thanks :slight_smile:

1 Like