Header transform concern need help

jkaufmanlr · April 1, 2021, 5:09pm

I have an API POST that I want filebeat to contact. This API calls for having in the header the value:

x-api-key : apiToken

I have attempted the following >

  request.transforms:
    - set:
        target: header
        value: 'x-api-key token'

  request.transforms:
    - set:
        target: header.x-api-key
        value: 'token'

  request.transforms:
    - set:
        target: header.Authorization
        value: 'x-api-key token' (plus with equals or colon)

None appear to work, any suggestions?

legoguy1000 · April 1, 2021, 8:25pm

It should be the 2nd one. HTTP JSON input | Filebeat Reference [7.12] | Elastic. What response do u get back?

jkaufmanlr · April 2, 2021, 2:48am

Thank you, ya the second wound up working.

I now have an issue with the body. It needs something like:

- set:
    target: body.service
    value: ['directory']

Sadly no matter the format it doesn't allow the format

legoguy1000 · April 2, 2021, 3:02am

What does the entire request look like? Is the value meant to be a list?

jkaufmanlr · April 2, 2021, 2:18pm

To my knowledge, yes it should be a list. I feel that the body.service can only take strings? Either way, here is what the API call's body looks like normally:

{
    "service": ["directory"],
    "start_time": "2021-03-25T14:00:00Z"
}

This is what I have attempted:

 request.transforms:
    - set:
        target: header.x-api-key
        value: 'mytoken'
    - set:
        target: body.service
        value: ['directory']
    - set:
        target: body.start_time
        value: '2021-03-17T14:00:00Z'

Eventually I will need the timestamp to be today's date but last five minutes too. For now I just want to get the service value passed over.

legoguy1000 · April 4, 2021, 12:25am

Are u getting an error from filebeat? Or from the api ur hitting?

jkaufmanlr · April 6, 2021, 2:14pm

Chances are I think it is coming from the api, but I could have sworn it was coming from Filebeat. Spinning up an instance to retest and will reply again shortly.

Wanted to also mention in order for the api to work via postman I have to send over the body in raw format. Also researching if that is possible for input.

jkaufmanlr · April 6, 2021, 2:26pm

confirmed it is filebeat:

error if I try:
value: "['directory']"
(Also for without \ on the single)

ERROR        [input.httpjson-stateless]        v2/input.go:129        Error while processing http request: failed to execute http client.Do: server responded with status code 400: {"message":"Unable to parse JSON request body: Invalid service: \u0026[34 91 39 100 105 114 101 99 116 111 114 121 39 93 34]: json: cannot unmarshal string into Go value of type []string"}

if I try >
value: [directory]

filebeat[6184]: Exiting: Failed to start crawler: starting input failed: Error while initializing input: fail to unpack the set configuration: type mismatch accessing 'filebeat.inputs.0.request.transforms.1.set.value' (source:'/etc/filebeat/filebeat.yml') accessing 'filebeat.inputs.0.request' (source:'/etc/filebeat/filebeat.yml')

legoguy1000 · April 6, 2021, 2:54pm

what if u try

- set:
     target: body.service
     value: 
     - 'directory'

or use double quotes instead

- set:
     target: body.service
     value: ["directory"]

jkaufmanlr · April 6, 2021, 3:04pm

First:

      ERROR        instance/beat.go:971        Exiting: Failed to start crawler: starting input failed: Error while initializing input: fail to unpack the set configuration: type mismatch accessing 'filebeat.inputs.0.request.transforms.1.set.value' (source:'/etc/filebeat/filebeat.yml') accessing 'filebeat.inputs.0.request' (source:'/etc/filebeat/filebeat.yml')

second:

       ERROR        instance/beat.go:971        Exiting: Failed to start crawler: starting input failed: Error while initializing input: fail to unpack the set configuration: type mismatch accessing 'filebeat.inputs.0.request.transforms.1.set.value' (source:'/etc/filebeat/filebeat.yml') accessing 'filebeat.inputs.0.request' (source:'/etc/filebeat/filebeat.yml')

legoguy1000 · April 6, 2021, 3:23pm

Ya, looking atht ecode for the httpjson v2 input, its casting the value to a string

func setToCommonMap(m common.MapStr, key, val string) error {
	if val == "" {
		return nil
	}
	if _, err := m.Put(key, val); err != nil {
		return err
	}
	return nil
}

github.com

elastic/beats/blob/master/x-pack/filebeat/input/httpjson/internal/v2/transform_set.go#L129


	value, err := set.value.Execute(ctx, tr, set.defaultValue, set.log)
	if err != nil && set.failOnTemplateError {
		return transformable{}, err
	}
	if err := set.runFunc(ctx, tr, set.targetInfo.Name, value); err != nil {
		return transformable{}, err
	}
	return tr, nil
}

func setToCommonMap(m common.MapStr, key, val string) error {
	if val == "" {
		return nil
	}
	if _, err := m.Put(key, val); err != nil {
		return err
	}
	return nil
}

func setBody(ctx *transformContext, transformable transformable, key, value string) error {

jkaufmanlr · April 6, 2021, 3:28pm

Yeah based on the error I would agree that is the case. Is it possible this an be submitted as a code update or an issue?

jkaufmanlr · April 6, 2021, 3:47pm

PS - I also did try this within request.body as well. Outcome was the same.
Our work around is to just do file .log collection, sadly we just dont want to bother having an extra piece that our clients will need to cronjob calling the api.

legoguy1000 · April 6, 2021, 4:05pm

I definitely think it could be. I'd submit a github issue, https://github.com/elastic/beats/issues/new/choose. Is this a bug or enhancement, i'd leave that to you.

jkaufmanlr · April 6, 2021, 4:54pm

Thanks for the suggestion, I did find something that tricks it into becoming an array:

append:
set: body.service
value: 'value1'
append:
set: body.service
value: 'value2'

I did notice at this point it made the API all with service in the body as an array. Sadly the product's api only allows 1 value in the array? I'm certain at one point they wanted multiple values but then changed it to accept just one :\

Still, this feels like something that set should just allow anyways. Thanks for the assistance, I will reply with bug number once I am done.

legoguy1000 · April 7, 2021, 12:24pm

Saw the github issue. Another question, are you able to remove the body.service field and then do the append so it always has only 1 value?

jkaufmanlr · April 7, 2021, 2:03pm

I understand what you are asking sadly I am not certain how to go about removing service with the remove command. Sadly I feel this wouldn't work because append adds the value as a string, you have to append another string to make it an array. Is there a way to say remove: ... value: 'value2'?

legoguy1000 · April 7, 2021, 2:19pm

HTTP JSON input | Filebeat Reference [7.12] | Elastic. ANd I see what you are saying, HTTP JSON input | Filebeat Reference [7.12] | Elastic

jkaufmanlr · April 7, 2021, 4:05pm

Okay we are on the same page, I feel like that wont work, if you delete the body.service that would remove the service: ["directory" "value2"], so append would only append a string "directory" because the service would have been deleted right?

Meaning the outcome would result in >

service: "directory"

which that would need to be an array.

legoguy1000 · April 7, 2021, 4:17pm

Ya. So i see either a change to allow you to set an array or have append create an array if it doesn't exist already instead of a string.

Topic		Replies	Views
Filebeat httpjson with content type plus beat block Beats filebeat	1	265	December 28, 2020
Httpjson Input Config Issue Beats fleet , filebeat	2	499	April 15, 2022
[Filebeat 8.6.1 - httpjson] chain response.transform question Beats filebeat	3	515	April 10, 2023
HTTPJson Input Request Transforms Beats filebeat	1	348	September 12, 2021
[filebeat] - Json processor Beats filebeat	2	224	June 2, 2022

Header transform concern need help

Related topics