Fields in Index but not in Discover or JSON output

Elastic Stack Kibana
1

Hi all

ELK version: 6.2.4
Platform: Windows Server 2012

I am using the IIS module which is working perfectly. I added 2 headers to my log file in IIS and they are showing in the log file.
I then updated my IIS grok patterns to pull those 2 new fields into elasticsearch.
I proceeded to update the Index patters and I can see those 2 fields under my filebeat-* index patterns.

But, when I query the filebeat-2019.09.13 pattern, those fields are not in the JSON response. I also checked the Discover page in Kibana, and they are not there.

Filebeat is definitely sending those fields up. (I confirmed this by changing the output to a file, and observing what is being logged to the file).
I checked the Kibana verbose logs as well as the Elasticsearch logs (and filebeat logs), but there is nothing in the logs, no Errors and no Warnings.

Here is an example of a log in my file (IP addresses X'ed out):
2019-09-13 13:54:38 xxx.xxx.xxx.26 GET / - 80 - xxx.xxx.xxx.71 Mozilla/5.0+(compatible;+CloudFlare-AlwaysOnline/1.0;++http://www.cloudflare.com/always-online)+AppleWebKit/534.34 - 200 0 0 265 xxx.xxx.xxx.134 xxx.xxx.xxx.134

Here is my Grok pattern (I tried different data types including IP, IPHOST, Greedydata,etc, but no luck). I also made these 2 new fields optional and not optional, but also no luck.
"%{TIMESTAMP_ISO8601:iis.access.time} %{IPORHOST:iis.access.server_ip} %{WORD:iis.access.method} %{URIPATH:iis.access.url} %{NOTSPACE:iis.access.query_string} %{NUMBER:iis.access.port} %{NOTSPACE:iis.access.user_name} %{IPORHOST:iis.access.remote_ip} %{NOTSPACE:iis.access.agent} %{NOTSPACE:iis.access.referrer} %{NUMBER:iis.access.response_code} %{NUMBER:iis.access.sub_status} %{NUMBER:iis.access.win32_status} %{NUMBER:iis.access.request_time_ms} (%{DATA:iis.access.orig_ip_from_cloudflare})? (%{DATA:iis.access.x_forwarded_ip})?",

All fields are in the JSON output, except for these two:
(%{DATA:iis.access.orig_ip_from_cloudflare})? (%{DATA:iis.access.x_forwarded_ip})

I also deleted all my indexes (its a test instance so it is OK) and started from scratch, but alas, no luck with this either.

Your help will be greatly appreciated.

2

So if you do a search query in the Dev Console in kibana, you see the fields in the documents there? If not, this is a problem lower down the line, where the fields don't get ingested in Elasticsearch.

3

Hi Marius

Thanks for the reply.

The 2 fields show in my filebeat index, but the fields don't show up in the json document when querying via the Dev console (neither do they show up in the "discover" tab in kibana). In the Discover tab under "available fields" these 2 fields also do not show up.

4

They can show in the index template cause that can be defined without the fields actually having value or working. If it doesn't show up in the dev console, then they aren't in the documents in ES so I would check if that Grok pattern works properly.
You could use the integrated Grok debugger from Kibana, that should help.

5

I tested it out using https://grokconstructor.appspot.com/do/match
and it was all good. But let me try the kibana tester as well tomorrow morning.

6

I logged on real quick and see I dont have the Grok Debugger in my Kibana. I'm still on 6.2.4. I think it was introduced with 6.8?
I tried again using the online debugger I provided in my previous post, and its all good. I just replaced the masked IPs with something that looks like a real IP address like the below:
2019-09-13 13:54:38 111.111.111.26 GET / - 80 - 111.111.111.71 Mozilla/5.0+(compatible;+CloudFlare-AlwaysOnline/1.0;++http://www.cloudflare.com/always-online)+AppleWebKit/534.34 - 200 0 0 265 111.111.111.134 111.111.111.134

7

OK, so I figured it out (It's been quite a while since I last work on ELK). I never pushed my new json pipeline file up to ES so it can actually load the new config...
Using these 2 commands everything started working. For those reading this, replace your versions and file locations etc with your values:

Save old ingest: Invoke-WebRequest -uri http://xxx.xxx.xxx.213:9200/_ingest/pipeline/filebeat-6.4.2-iis-access-default | set-content old.json
PUT the new ingest: Invoke-WebRequest -uri http://xxx.xxx.xxx.213:9200/_ingest/pipeline/filebeat-6.4.2-iis-access-default -ContentType "application/json" -InFile 'd:\filebeat642\module\iis\access\ingest\default.json' -Method PUT

Kudos to this guy for the above commands: https://www.frakkingsweet.com/filebeat-and-iis-with-multiple-sites/

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.