Custom field format doesn't get applied, some field types get changed

[DPattee]

I am editing the mybeat/_meta/fields.yml file to include a section like:

  fields:
    - name: resulturl
      type: text
      format: url
      required: true
      description: >
        PLEASE UPDATE DOCUMENTATION
    - name: packetloss
      type: integer
      required: true
      description: >
        PLEASE UPDATE DOCUMENTATION

I run make update and see that the 'format' line gotta merged in to the mybeat/fields.yml file as expected, and the 'type' of both those fields got pulled over as expected.

I delete the index and template from my previous run through the kibana console

DELETE _template/mybeat-8.0.0
DELETE mybeat-8.0.0*

I build and run my beat via

mage build && ./mybeat -e -d "*"

And then I go look at the data in kibana's discover view and the fields aren't properly formatted and they just say 'no cached mapping'

No Index Pattern gets made, which I thought should happen automatically when setting up a new beat. (even explicitly running ./mybeat -e -d "*" setup doesn't do it)

And if I look at the template through the kibana console via

GET _template/mybeat*

I just see that the integer field actually has its type set to long, and the text/url field just has its type set to text with no format information.

Of course I can manually create an index pattern and manually set the Format of a field through Kibana's index management UI, but that isn't how a nice beat works. (Looking at the automatically-created metricbeat-* index pattern I see that there are many fields that correctly kept their Format)

[shaunak (Shaunak Kashyap)]

Hmmm, it looks like something might be breaking in between the make update step and the step where the generated index template gets loaded into Elasticsearch. I suspect if we can figure this out we'll sort out the Kibana index pattern issue as well; in any case let's deal with that one later once we've figured out the index template issue.

When you run mage build && ./mybeat -e -d "*" do you see any errors between the time that your beat starts up and the time the first event is published to Elasticsearch?

Also, if you run ./mybeat export template do you see your custom fields in there?

Shaunak

[DPattee]

I mucked around a bit more over the weekend, trying to figure out how to get dashboards added too. (The docs skipped a bit, I finally found some examples in the official metricbeat repo)

So, first I'm going to delete everything through the console:

DELETE _template/speedbeat-8.0.0
DELETE speedbeat-8.0.0*

And double checked the Index Management and Index Patterns management UI to make sure I didn't miss anything.

Then a fresh make & build

make update
// a quick jump to my text editor to set the kibana and elastic hosts since I don't dev against localhost
mage build && ./speedbeat -e -d "*"

From here the log lines between startup and first event that seem relevant are where it properly connects and creates a new index:

2020-09-14T17:27:58.156-0700	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'speedbeat-8.0.0' as ILM is enabled.
2020-09-14T17:27:58.156-0700	INFO	eslegclient/connection.go:99	elasticsearch url: http://elasticservername:9200

Then it sends an event (and the payload debug output right above it shows the correct blob)

2020-09-14T17:28:46.091-0700	INFO	beater/Speedbeat.go:85	Event sent
2020-09-14T17:28:47.096-0700	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(elasticsearch(http://elasticservername:9200))

Then the ILM policy looks like it is attempting to create it fine, but I'd forgotten to manually delete the old policy

|2020-09-14T17:28:47.265-0700|INFO|[index-management]|idxmgmt/std.go:261|Auto ILM enable success.|
|2020-09-14T17:28:47.268-0700|INFO|[index-management.ilm]|ilm/std.go:139|do not generate ilm policy: exists=true, overwrite=false|
|2020-09-14T17:28:47.268-0700|INFO|[index-management]|idxmgmt/std.go:274|ILM policy successfully loaded.|
|2020-09-14T17:28:47.268-0700|INFO|[index-management]|idxmgmt/std.go:407|Set setup.template.name to '{speedbeat-8.0.0 {now/d}-000001}' as ILM is enabled.|
|2020-09-14T17:28:47.268-0700|INFO|[index-management]|idxmgmt/std.go:412|Set setup.template.pattern to 'speedbeat-8.0.0-*' as ILM is enabled.|
|2020-09-14T17:28:47.268-0700|INFO|[index-management]|idxmgmt/std.go:446|Set settings.index.lifecycle.rollover_alias in template to {speedbeat-8.0.0 {now/d}-000001} as ILM is enabled.|
|2020-09-14T17:28:47.268-0700|INFO|[index-management]|idxmgmt/std.go:450|Set settings.index.lifecycle.name in template to {speedbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.|

Then it tries to load the template

2020-09-14T17:28:47.329-0700	INFO	[template_loader]	template/load.go:121	Try loading template speedbeat-8.0.0 to Elasticsearch
|2020-09-14T17:28:47.840-0700|INFO|[template_loader]|template/load.go:113|template with name 'speedbeat-8.0.0' loaded.|
|2020-09-14T17:28:47.840-0700|INFO|[index-management]|idxmgmt/std.go:298|Loaded index template.|

At this point I switch to the kibana dev console and do a GET _template/speedbeat* and copy that output in to a text editor, find one of my fields, and see it isn't defined as expected.

I.e. the console output says (and appears to match the output of ./speedbeat export template > foo.txt )

"properties" : {
            "elapsed" : {
              "type" : "long"
            },

but my fields.yml has elapsed defined as

- name: download.elapsed
      type: integer
      format: duration
      required: true
      description: foo

The beat continues to run and moves on to the next event send. No additional elastic or kibana "loading stuff" type things occur. At this point no new entry is visible through the Index Patterns UI, and finding one of the documents in the * index shows all of the fields with the yellow exclamation point.