Adding .raw to winlogbeat messages

Elastic Stack Beats
1

Hi - I'm using the default winlogbeat.template.json file that downloaded with winlogbeat.

I've made some modifications to the mapping in an attempt to get .raw not-analyzed results to show up for key string fields in Windows Event Viewer, specifically: category and source_name.

Below is a copy of my template file. The lines I added are rows 31-50. I also:

  • Stopped the winlogbeat service on my client
  • Deleted my winlogbeat indexes on Elasticsearch
  • Deleted .winlogbeat.yml in C:\ProgramData\winlogbeat
  • Made the edits below to winlogbeat.template.json
  • Restarted the winlogbeat service
  • When data started showing up, deleted and recreated the index mapping in Kibana.

What am I doing wrong? .raw are still not showing up in Kibana in either Visualize or Explore.

{
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true,
        "norms": {
          "enabled": false
        }
      },
      "dynamic_templates": [
        {
          "template1": {
            "mapping": {
              "doc_values": true,
              "ignore_above": 1024,
              "index": "not_analyzed",
              "type": "{dynamic_type}"
            },
            "match": "*"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "index": "analyzed",
          "type": "string"
        },		
		"category": {
                "analyzer": "standard",
                "fields": {
                    "raw": {
                        "index": "not_analyzed",
                        "type": "string"
                    }
                },
                "type": "string"
            },
		"source_name": {
                "analyzer": "standard",
                "fields": {
                    "raw": {
                        "index": "not_analyzed",
                        "type": "string"
                    }
                },
                "type": "string"
            }
      }
    }
  },
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "winlogbeat-*"
}
2

Hi @joshablett, first off these modification should not be necessary. The provided index template should be setting the type of these fields to be not_analyzed strings. What version of Winlogbeat are you using?

I did not see you mention in your steps that you actually installed the index template to Elasticsearch. Before you index any data in Elasticsearch you need to HTTP PUT the index template into ES unless you have configured Winlogbeat to automatically install the template (if you are changing an already installed template then you need also overwrite: true). https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html

You can check that the template is installed with curl http://localhost:9200/_template/winlogbeat?pretty.

3

Hi Andrew - thanks for the reply!

Just to be sure, I deleted, re-downloaded winlogbeat, and did a full reinstall. This wiped away the changes I had previously made to the winlogbeat.template.json file.

I am using version 1.2.3 from this page, downloaded today: https://www.elastic.co/downloads/beats/winlogbeat

Here is the template setting from my winlogbeat.yml file:

# A template is used to set the mapping in Elasticsearch
# By default template loading is disabled and no template is loaded.
# These settings can be adjusted to load your own template or overwrite existing ones
template:

  # Template name. By default the template name is winlogbeat.
  #name: "winlogbeat"

  # Path to template file
  path: "winlogbeat.template.json"

  # Overwrite existing template
  overwrite: true

When I check that the template is installed with curl http://localhost:9200/_template/winlogbeat?pretty, all I get in response is an empty bracket {}.

Any other things I can check?

Thanks,
Josh

4

Can you please post the full config file. I want to make sure that the indentation is correct.

5

You bet! Here it is:

It was too large to fit, and the file uploader only accepts images.
Thanks!

6

Everything appears to correct. Can you try running it with debug turned on to see if there is anything relevant about the template being loaded.

You have to modify the config file. Change level from info to debug and set selectors to *. Then restart Winlogbeat.

  level: debug
  selectors: ['*']
7

Nope. Searched for "template" and didn't see anything.

8

There should definitely be some log lines related the template loading coming from v1.2.3. You should be seeing some of the log lines coming from here:


9

Try uncommenting the #name: "winlogbeat" line in the template options.

10

Hi Andrew - sorry for the delay. I've actually updated to the 5.0.0 winlogbeat alpha to get the parsed fields. With a clean install, I'm still not seeing the .raw option. Should I follow the same steps to troubleshoot this version?

11

There fields are not named as .raw but all the strings (with the exception of message) are not_analyzed strings (or keywords in ES 5.x). Here's the template file used with ES 2.x.

12

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.