Custom Unpack for configuration

[tomkirk]

I'm working on a beat where I have currently in my configuration an array of string like this:

fields:
  - FieldName
  - AnotherFieldName

What I would like is the ability to optionally provide a type tag, like so:

fields:
    - FieldName
    - name: AnotherFieldName
    - type: int

Essentially I want to define my configuration structure so that if it is a string it uses that as the name parameter of my struct, otherwise it reads the name and type tags. A field is defined like this:

type Field struct {
    Name string 'config:"name"`
    Type string 'config:"type"`
}

and the config like this:

type Config struct {
    Fields []Field `config:"fields"`
}

I had considered implementing the Unpacker interface for a Field, checking if it was a string and setting the name, but then I wanted to call the default Unpack for the struct if it wasn't a string but I can't work out how to do this.

Is there a way of either

1. Defining the conversion with tags
2. calling the default Unpack for the field

Thanks

[marc.guasch]

Hello @tomkirk,

As you mention, you can implement the Unpacker interface. If I understood what you try to achieve correctly, I think you might not need to fallback to the original Unpack method with something like:

type cfg struct {
    Fields []field `config:"fields"`
}

type field struct {
    Name string `config:"name"`
    Type string `config:"type"`
}

func (f *field) Unpack(v interface{}) error {
	switch tv := v.(type) {
	case string:
		*f = field{
			Name: tv,
		}
	case map[string]interface{}:
		n, _ := tv["name"].(string)
		t, _ := tv["type"].(string)
        // additional checks if needed
		*f = field{
			Name: n,
			Type: t,
		}
	default:
		return errors.New("unexpected type")
	}
	return nil
}

Hope it helps!

[tomkirk]

Thanks for your reply.

The reason why I wanted to fallback to the original Unpack was to prevent the need to explicitly check each field. I was hoping that I could just use the config tags to do it for me. That way if I want another parameter in the future I can define it like the other configuration settings. Is that not possible?

[marc.guasch]

It is possible, in that case you will need to embed your field type in another one used just to do the Unpack, otherwise you will end up having a stack overflow from recursive Unpack calls:

type cfg struct {
	Fields []cfgfield `config:"fields"`
}

type cfgfield struct {
	field
}
    
type field struct {
	Name string `config:"name"`
	Type string `config:"type"`
}

func (f *cfgfield) Unpack(v interface{}) error {
	switch tv := v.(type) {
	case string:
		*f = cfgfield{
			field: field{
				Name: tv,
			},
		}
	default:
		cfg, err := ucfg.NewFrom(v)
		if err != nil {
			return err
		}
		field := field{}
		if err := cfg.Unpack(&field); err != nil {
			return err
		}
		*f = cfgfield{field: field}
	}
	return nil
}

Hope it helps

[tomkirk]

That looks like what I need. How would I go about getting access to the ucfg package inside my beat? The one I am working on isn't using go modules, but glide. I currently have my glide.yaml as:

package: .
import:
- package: github.com/elastic/beats
  version: v7.1.1
  subpackages:
  - /libbeat/beat
  - libbeat/cfgfile
  - libbeat/common
  - libbeat/logp

Is there a way to use the ucfg that is used inside that version of libbeat?

Thanks

[marc.guasch]

I am not sure if that is what you mean, but ucfg refers to elastic/go-ucfg which is a module of its own https://github.com/elastic/go-ucfg so I guess by adding it as another imported package should do the trick.