Header transform concern need help

Elastic Stack Beats
#1

I have an API POST that I want filebeat to contact. This API calls for having in the header the value:

x-api-key : apiToken

I have attempted the following >

  request.transforms:
    - set:
        target: header
        value: 'x-api-key token'

  request.transforms:
    - set:
        target: header.x-api-key
        value: 'token'

  request.transforms:
    - set:
        target: header.Authorization
        value: 'x-api-key token' (plus with equals or colon)

None appear to work, any suggestions?

#2

It should be the 2nd one. HTTP JSON input | Filebeat Reference [7.12] | Elastic. What response do u get back?

#3

Thank you, ya the second wound up working.

I now have an issue with the body. It needs something like:

- set:
    target: body.service
    value: ['directory']

Sadly no matter the format it doesn't allow the format

#4

What does the entire request look like? Is the value meant to be a list?

#5

To my knowledge, yes it should be a list. I feel that the body.service can only take strings? Either way, here is what the API call's body looks like normally:

{
    "service": ["directory"],
    "start_time": "2021-03-25T14:00:00Z"
}

This is what I have attempted:

 request.transforms:
    - set:
        target: header.x-api-key
        value: 'mytoken'
    - set:
        target: body.service
        value: ['directory']
    - set:
        target: body.start_time
        value: '2021-03-17T14:00:00Z'

Eventually I will need the timestamp to be today's date but last five minutes too. For now I just want to get the service value passed over.

#6

Are u getting an error from filebeat? Or from the api ur hitting?

#7

Chances are I think it is coming from the api, but I could have sworn it was coming from Filebeat. Spinning up an instance to retest and will reply again shortly.

Wanted to also mention in order for the api to work via postman I have to send over the body in raw format. Also researching if that is possible for input.

#8

confirmed it is filebeat:

error if I try:
value: "['directory']"
(Also for without \ on the single)

ERROR        [input.httpjson-stateless]        v2/input.go:129        Error while processing http request: failed to execute http client.Do: server responded with status code 400: {"message":"Unable to parse JSON request body: Invalid service: \u0026[34 91 39 100 105 114 101 99 116 111 114 121 39 93 34]: json: cannot unmarshal string into Go value of type []string"}

if I try >
value: [directory]

filebeat[6184]: Exiting: Failed to start crawler: starting input failed: Error while initializing input: fail to unpack the set configuration: type mismatch accessing 'filebeat.inputs.0.request.transforms.1.set.value' (source:'/etc/filebeat/filebeat.yml') accessing 'filebeat.inputs.0.request' (source:'/etc/filebeat/filebeat.yml')
#9

what if u try

- set:
     target: body.service
     value: 
     - 'directory'

or use double quotes instead

- set:
     target: body.service
     value: ["directory"]
#10

First:

      ERROR        instance/beat.go:971        Exiting: Failed to start crawler: starting input failed: Error while initializing input: fail to unpack the set configuration: type mismatch accessing 'filebeat.inputs.0.request.transforms.1.set.value' (source:'/etc/filebeat/filebeat.yml') accessing 'filebeat.inputs.0.request' (source:'/etc/filebeat/filebeat.yml')

second:

       ERROR        instance/beat.go:971        Exiting: Failed to start crawler: starting input failed: Error while initializing input: fail to unpack the set configuration: type mismatch accessing 'filebeat.inputs.0.request.transforms.1.set.value' (source:'/etc/filebeat/filebeat.yml') accessing 'filebeat.inputs.0.request' (source:'/etc/filebeat/filebeat.yml')
#11

Ya, looking atht ecode for the httpjson v2 input, its casting the value to a string

func setToCommonMap(m common.MapStr, key, val string) error {
	if val == "" {
		return nil
	}
	if _, err := m.Put(key, val); err != nil {
		return err
	}
	return nil
}
#12

Yeah based on the error I would agree that is the case. Is it possible this an be submitted as a code update or an issue?

#13

PS - I also did try this within request.body as well. Outcome was the same.
Our work around is to just do file .log collection, sadly we just dont want to bother having an extra piece that our clients will need to cronjob calling the api.

#14

I definitely think it could be. I'd submit a github issue, https://github.com/elastic/beats/issues/new/choose. Is this a bug or enhancement, i'd leave that to you.

#15

Thanks for the suggestion, I did find something that tricks it into becoming an array:

  • append:
    set: body.service
    value: 'value1'
  • append:
    set: body.service
    value: 'value2'

I did notice at this point it made the API all with service in the body as an array. Sadly the product's api only allows 1 value in the array? I'm certain at one point they wanted multiple values but then changed it to accept just one :\

Still, this feels like something that set should just allow anyways. Thanks for the assistance, I will reply with bug number once I am done.

#16

Saw the github issue. Another question, are you able to remove the body.service field and then do the append so it always has only 1 value?

#17

I understand what you are asking sadly I am not certain how to go about removing service with the remove command. Sadly I feel this wouldn't work because append adds the value as a string, you have to append another string to make it an array. Is there a way to say remove: ... value: 'value2'?

#18

HTTP JSON input | Filebeat Reference [7.12] | Elastic. ANd I see what you are saying, HTTP JSON input | Filebeat Reference [7.12] | Elastic

#19

Okay we are on the same page, I feel like that wont work, if you delete the body.service that would remove the service: ["directory" "value2"], so append would only append a string "directory" because the service would have been deleted right?

Meaning the outcome would result in >

service: "directory"

which that would need to be an array.

#20

Ya. So i see either a change to allow you to set an array or have append create an array if it doesn't exist already instead of a string.