# Header transform concern need help

**URL:** https://discuss.elastic.co/t/header-transform-concern-need-help/269064
**Category:** Beats
**Tags:** filebeat
**Created:** [April 1, 2021, 5:09pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064 "2021-04-01T17:09:20Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![jkaufmanlr](https://avatars.discourse-cdn.com/v4/letter/j/2bfe46/32.png) [@jkaufmanlr](https://discuss.elastic.co/u/jkaufmanlr)
#### Post date: [April 1, 2021, 5:09pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/1 "2021-04-01T17:09:20Z")

</div>

I have an API POST that I want filebeat to contact. This API calls for having in the header the value:

```auto
x-api-key : apiToken

```

I have attempted the following \>

```auto
  request.transforms:
    - set:
        target: header
        value: 'x-api-key token'

  request.transforms:
    - set:
        target: header.x-api-key
        value: 'token'

  request.transforms:
    - set:
        target: header.Authorization
        value: 'x-api-key token' (plus with equals or colon)

```

None appear to work, any suggestions?

---

<div class="post-metadata">

### Author: ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [April 1, 2021, 8:25pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/2 "2021-04-01T20:25:37Z")

</div>

It should be the 2nd one. [HTTP JSON input | Filebeat Reference [7.12] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html#request-transforms). What response do u get back?

---

<div class="post-metadata">

### Author: ![jkaufmanlr](https://avatars.discourse-cdn.com/v4/letter/j/2bfe46/32.png) [@jkaufmanlr](https://discuss.elastic.co/u/jkaufmanlr)
#### Post date: [April 2, 2021, 2:48am UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/3 "2021-04-02T02:48:41Z")

</div>

Thank you, ya the second wound up working.

I now have an issue with the body. It needs something like:

```
- set:
    target: body.service
    value: ['directory']

```

Sadly no matter the format it doesn't allow the format

---

<div class="post-metadata">

### Author: ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [April 2, 2021, 3:02am UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/4 "2021-04-02T03:02:33Z")

</div>

What does the entire request look like? Is the value meant to be a list?

---

<div class="post-metadata">

### Author: ![jkaufmanlr](https://avatars.discourse-cdn.com/v4/letter/j/2bfe46/32.png) [@jkaufmanlr](https://discuss.elastic.co/u/jkaufmanlr)
#### Post date: [April 2, 2021, 2:18pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/5 "2021-04-02T14:18:31Z")

</div>

To my knowledge, yes it should be a list. I feel that the body.service can only take strings? Either way, here is what the API call's body looks like normally:

```auto
{
    "service": ["directory"],
    "start_time": "2021-03-25T14:00:00Z"
}

```

This is what I have attempted:

```auto
 request.transforms:
    - set:
        target: header.x-api-key
        value: 'mytoken'
    - set:
        target: body.service
        value: ['directory']
    - set:
        target: body.start_time
        value: '2021-03-17T14:00:00Z'

```

Eventually I will need the timestamp to be today's date but last five minutes too. For now I just want to get the service value passed over.

---

<div class="post-metadata">

### Author: ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [April 4, 2021, 12:25am UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/6 "2021-04-04T00:25:04Z")

</div>

Are u getting an error from filebeat? Or from the api ur hitting?

---

<div class="post-metadata">

### Author: ![jkaufmanlr](https://avatars.discourse-cdn.com/v4/letter/j/2bfe46/32.png) [@jkaufmanlr](https://discuss.elastic.co/u/jkaufmanlr)
#### Post date: [April 6, 2021, 2:14pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/7 "2021-04-06T14:14:32Z")

</div>

Chances are I think it is coming from the api, but I could have sworn it was coming from Filebeat. Spinning up an instance to retest and will reply again shortly.

Wanted to also mention in order for the api to work via postman I have to send over the body in raw format. Also researching if that is possible for input.

---

<div class="post-metadata">

### Author: ![jkaufmanlr](https://avatars.discourse-cdn.com/v4/letter/j/2bfe46/32.png) [@jkaufmanlr](https://discuss.elastic.co/u/jkaufmanlr)
#### Post date: [April 6, 2021, 2:26pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/8 "2021-04-06T14:26:54Z")

</div>

confirmed it is filebeat:

error if I try:  
value: "['directory']"  
(Also for without \ on the single)

```auto
ERROR [input.httpjson-stateless] v2/input.go:129 Error while processing http request: failed to execute http client.Do: server responded with status code 400: {"message":"Unable to parse JSON request body: Invalid service: \u0026[34 91 39 100 105 114 101 99 116 111 114 121 39 93 34]: json: cannot unmarshal string into Go value of type []string"}

```

if I try \>  
value: [directory]

```auto
filebeat[6184]: Exiting: Failed to start crawler: starting input failed: Error while initializing input: fail to unpack the set configuration: type mismatch accessing 'filebeat.inputs.0.request.transforms.1.set.value' (source:'/etc/filebeat/filebeat.yml') accessing 'filebeat.inputs.0.request' (source:'/etc/filebeat/filebeat.yml')

```

---

<div class="post-metadata">

### Author: ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [April 6, 2021, 2:54pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/9 "2021-04-06T14:54:58Z")

</div>

what if u try

```auto
- set:
     target: body.service
     value: 
     - 'directory'

```

or use double quotes instead

```auto
- set:
     target: body.service
     value: ["directory"]

```

---

<div class="post-metadata">

### Author: ![jkaufmanlr](https://avatars.discourse-cdn.com/v4/letter/j/2bfe46/32.png) [@jkaufmanlr](https://discuss.elastic.co/u/jkaufmanlr)
#### Post date: [April 6, 2021, 3:04pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/10 "2021-04-06T15:04:39Z")

</div>

First:

```auto
      ERROR instance/beat.go:971 Exiting: Failed to start crawler: starting input failed: Error while initializing input: fail to unpack the set configuration: type mismatch accessing 'filebeat.inputs.0.request.transforms.1.set.value' (source:'/etc/filebeat/filebeat.yml') accessing 'filebeat.inputs.0.request' (source:'/etc/filebeat/filebeat.yml')

```

second:

```auto
       ERROR instance/beat.go:971 Exiting: Failed to start crawler: starting input failed: Error while initializing input: fail to unpack the set configuration: type mismatch accessing 'filebeat.inputs.0.request.transforms.1.set.value' (source:'/etc/filebeat/filebeat.yml') accessing 'filebeat.inputs.0.request' (source:'/etc/filebeat/filebeat.yml')

```

---

<div class="post-metadata">

### Author: ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [April 6, 2021, 3:23pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/11 "2021-04-06T15:23:00Z")

</div>

Ya, looking atht ecode for the httpjson v2 input, its casting the value to a string

```auto
func setToCommonMap(m common.MapStr, key, val string) error {
	if val == "" {
		return nil
	}
	if _, err := m.Put(key, val); err != nil {
		return err
	}
	return nil
}

```

> <https://github.com/elastic/beats/blob/master/x-pack/filebeat/input/httpjson/internal/v2/transform_set.go#L129>

---

<div class="post-metadata">

### Author: ![jkaufmanlr](https://avatars.discourse-cdn.com/v4/letter/j/2bfe46/32.png) [@jkaufmanlr](https://discuss.elastic.co/u/jkaufmanlr)
#### Post date: [April 6, 2021, 3:28pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/12 "2021-04-06T15:28:37Z")

</div>

Yeah based on the error I would agree that is the case. Is it possible this an be submitted as a code update or an issue?

---

<div class="post-metadata">

### Author: ![jkaufmanlr](https://avatars.discourse-cdn.com/v4/letter/j/2bfe46/32.png) [@jkaufmanlr](https://discuss.elastic.co/u/jkaufmanlr)
#### Post date: [April 6, 2021, 3:47pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/13 "2021-04-06T15:47:26Z")

</div>

PS - I also did try this within request.body as well. Outcome was the same.  
Our work around is to just do file .log collection, sadly we just dont want to bother having an extra piece that our clients will need to cronjob calling the api.

---

<div class="post-metadata">

### Author: ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [April 6, 2021, 4:05pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/14 "2021-04-06T16:05:48Z")

</div>

I definitely think it could be. I'd submit a github issue, [https://github.com/elastic/beats/issues/new/choose](https://github.com/elastic/beats/issues/new/choose). Is this a bug or enhancement, i'd leave that to you.

---

<div class="post-metadata">

### Author: ![jkaufmanlr](https://avatars.discourse-cdn.com/v4/letter/j/2bfe46/32.png) [@jkaufmanlr](https://discuss.elastic.co/u/jkaufmanlr)
#### Post date: [April 6, 2021, 4:54pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/15 "2021-04-06T16:54:19Z")

</div>

Thanks for the suggestion, I did find something that tricks it into becoming an array:

- append:  
set: body.service  
value: 'value1'
- append:  
set: body.service  
value: 'value2'

I did notice at this point it made the API all with service in the body as an array. Sadly the product's api only allows 1 value in the array? I'm certain at one point they wanted multiple values but then changed it to accept just one :\

Still, this feels like something that set should just allow anyways. Thanks for the assistance, I will reply with bug number once I am done.

---

<div class="post-metadata">

### Author: ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [April 7, 2021, 12:24pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/16 "2021-04-07T12:24:11Z")

</div>

Saw the github issue. Another question, are you able to remove the `body.service` field and then do the append so it always has only 1 value?

---

<div class="post-metadata">

### Author: ![jkaufmanlr](https://avatars.discourse-cdn.com/v4/letter/j/2bfe46/32.png) [@jkaufmanlr](https://discuss.elastic.co/u/jkaufmanlr)
#### Post date: [April 7, 2021, 2:03pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/17 "2021-04-07T14:03:07Z")

</div>

I understand what you are asking sadly I am not certain how to go about removing service with the remove command. Sadly I feel this wouldn't work because append adds the value as a string, you have to append another string to make it an array. Is there a way to say remove: ... value: 'value2'?

---

<div class="post-metadata">

### Author: ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [April 7, 2021, 2:19pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/18 "2021-04-07T14:19:15Z")

</div>

[HTTP JSON input | Filebeat Reference [7.12] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html#_delete). ANd I see what you are saying, [HTTP JSON input | Filebeat Reference [7.12] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html#_append)

---

<div class="post-metadata">

### Author: ![jkaufmanlr](https://avatars.discourse-cdn.com/v4/letter/j/2bfe46/32.png) [@jkaufmanlr](https://discuss.elastic.co/u/jkaufmanlr)
#### Post date: [April 7, 2021, 4:05pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/19 "2021-04-07T16:05:47Z")

</div>

Okay we are on the same page, I feel like that wont work, if you delete the body.service that would remove the service: ["directory" "value2"], so append would only append a string "directory" because the service would have been deleted right?

Meaning the outcome would result in \>

```auto
service: "directory"

```

which that would need to be an array.

---

<div class="post-metadata">

### Author: ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [April 7, 2021, 4:17pm UTC](https://discuss.elastic.co/t/header-transform-concern-need-help/269064/20 "2021-04-07T16:17:26Z")

</div>

Ya. So i see either a change to allow you to `set` an array or have `append` create an array if it doesn't exist already instead of a string.

[Next page](https://discuss.elastic.co/t/header-transform-concern-need-help/269064.md?page=2)
