# Bad request response when adding WAF logs to Elasticsearch
**URL:** https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093
**Category:** Elasticsearch
**Created:** [October 17, 2019, 3:48pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093 "2019-10-17T15:48:06Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)
#### Post date: [October 17, 2019, 3:48pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/1 "2019-10-17T15:48:07Z")
Hi there,
Due to some custom requirements, I am attempting to ship my WAF logs into Elasticsearch using lambda.
The function is fairly simple Python, as follows:
```auto
import re
import requests
from requests_aws4auth import AWS4Auth
region = 'us-east-1' # e.g. us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
host = '' # the Amazon ES domain, including https://
index = 'awswaf-'
type = 'waflog'
url = host + '/' + index + '/' + type
headers = { "Content-Type": "application/json" }
s3 = boto3.client('s3')
# Lambda execution starts here
def handler(event, context):
for record in event['Records']:
# Get the bucket name and key for the new file
bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']
##Troubleshooting: Log results in console
#print("Bucket :", bucket)
#print("Key :", key)
# Get, read, and split the file into lines
obj = s3.get_object(Bucket=bucket, Key=key)
body = obj['Body'].read()
lines = body.splitlines()
#Build and send the request
document = lines
r = requests.post(url, auth=awsauth, json=document, headers=headers)
#Log response in console
print("Response: ", r)
```
This is passing a WAF log which is just JSON (can provide an example if it would help) to the Elasticsearch domain, however I am getting a 400 Bad Request response for it.
I thought that this could be due to an issue with the indexing of the parameters, but I still received the same error after adding the following index:
```auto
PUT awswaf-
{
"mappings" : {
"properties" : {
"action" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"formatVersion" : {
"type" : "long"
},
"httpRequest" : {
"properties" : {
"args" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"clientIp" : {
"type" : "keyword",
"fields" : {
"keyword" : {
"type" : "ip"
}
}
},
"country" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"headers" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"value" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"httpMethod" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"httpVersion" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"uri" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"httpSourceId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"httpSourceName" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"rateBasedRuleList" : {
"properties" : {
"limitKey" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"maxRateAllowed" : {
"type" : "long"
},
"rateBasedRuleId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"terminatingRuleId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"terminatingRuleType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"timestamp" : {
"type" : "date",
"format" : "epoch_millis"
},
"webaclId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
```
Could you please help me to understand why the request is being rejected?
---
### Author:  [@lukas](https://discuss.elastic.co/u/lukas)
#### Post date: [October 17, 2019, 10:24pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/2 "2019-10-17T22:24:27Z")
Does this belong in the Elasticsearch discuss forums? I'm not sure I understand where Kibana fits into this question.
---
### Author:  [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)
#### Post date: [October 18, 2019, 9:25am UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/3 "2019-10-18T09:25:17Z")
You're right, sorry about that. Updated.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 18, 2019, 1:55pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/4 "2019-10-18T13:55:48Z")
What version of Elasticsearch are we talking about here? What is the error response? I tried the request you quoted above and it worked:
```auto
# curl -XPUT 'http://localhost:9200/awswaf-' -H 'Content-type: application/json' --data-binary $'{"mappings":{"properties":{"terminatingRuleId":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"rateBasedRuleList":{"properties":{"limitKey":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"rateBasedRuleId":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"maxRateAllowed":{"type":"long"}}},"terminatingRuleType":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"webaclId":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"action":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"httpRequest":{"properties":{"httpMethod":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"args":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"httpVersion":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"country":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"uri":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"headers":{"properties":{"value":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"name":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}}}},"clientIp":{"type":"keyword","fields":{"keyword":{"type":"ip"}}}}},"formatVersion":{"type":"long"},"httpSourceName":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"httpSourceId":{"type":"text","fields":{"keyword":{"ignore_above":256,"type":"keyword"}}},"timestamp":{"format":"epoch_millis","type":"date"}}}}'
# ----------------------------------------
# Response:
# 200 OK
# {
# "shards_acknowledged": true,
# "acknowledged": true,
# "index": "awswaf-"
# }
# at 2019-10-18T13:55:02.720Z
# (0.459298s elapsed)
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 18, 2019, 1:57pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/5 "2019-10-18T13:57:52Z")
Sorry, misread the post, you're not having problems with the index creation but with something else. Can you share the exact request that's being sent to Elasticsearch as well as the exact response?
---
### Author:  [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)
#### Post date: [October 18, 2019, 4:26pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/6 "2019-10-18T16:26:36Z")
The request being sent is:
```auto
{
"timestamp": 1571137876737,
"formatVersion": 1,
"webaclId": "1950425e-9278-4c05-8c67-4d500b9659be",
"terminatingRuleId": "Default_Action",
"terminatingRuleType": "REGULAR",
"action": "ALLOW",
"httpSourceName": "ALB",
"httpSourceId": "639917743964-app/XXXXXXXXXX/2d9c1970fc485a41",
"ruleGroupList": [],
"rateBasedRuleList": [
{
"rateBasedRuleId": "ea04e5d3-bcc1-499b-ab7a-583ef849d48a",
"limitKey": "IP",
"maxRateAllowed": 100
}
],
"nonTerminatingMatchingRules": [],
"httpRequest": {
"clientIp": "XXX.XXX.XXX.XXX",
"country": "JP",
"headers": [
{
"name": "Host",
"value": "X.X.XXX.XXX"
},
{
"name": "User-Agent",
"value": "HTTP Banner Detection (https://security.ipip.net)"
}
],
"uri": "/",
"args": "",
"httpVersion": "HTTP/1.1",
"httpMethod": "GET",
"requestId": null
}
}
```
And the response I am getting is:
```auto
{
"error": {
"root_cause": [
{
"type": "not_x_content_exception",
"reason": "not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"
}
],
"type": "mapper_parsing_exception",
"reason": "failed to parse",
"caused_by": {
"type": "not_x_content_exception",
"reason": "not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"
}
},
"status": 400
}
```
Thanks 🙂
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 18, 2019, 4:45pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/7 "2019-10-18T16:45:40Z")
Can you reproduce this outside your Python script? For instance I tried creating this document with `curl` and it worked:
```auto
curl 'http://localhost:9200/awswaf-/_doc' -H 'Content-type: application/json' --data-binary $'{"terminatingRuleId":"Default_Action","rateBasedRuleList":[{"limitKey":"IP","rateBasedRuleId":"ea04e5d3-bcc1-499b-ab7a-583ef849d48a","maxRateAllowed":100}],"terminatingRuleType":"REGULAR","webaclId":"1950425e-9278-4c05-8c67-4d500b9659be","action":"ALLOW","httpRequest":{"requestId":null,"httpMethod":"GET","args":"","httpVersion":"HTTP/1.1","country":"JP","uri":"/","headers":[{"value":"1.2.3.4","name":"Host"},{"value":"HTTP Banner Detection (https://security.ipip.net)","name":"User-Agent"}],"clientIp":"1.2.3.4"},"formatVersion":1,"httpSourceName":"ALB","nonTerminatingMatchingRules":[],"httpSourceId":"639917743964-app/XXXXXXXXXX/2d9c1970fc485a41","ruleGroupList":[],"timestamp":1571137876737}'
{"_index":"awswaf-","_type":"_doc","_id":"CljA320BEfZQotkPq6Kt","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":2}
```
If you can do it with `curl` and not with your Python script then that helps us narrow down the issue.
---
### Author:  [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)
#### Post date: [October 21, 2019, 8:54am UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/8 "2019-10-21T08:54:14Z")
I used Postman to make the same request outside of the python script and was able to successfully create the document:
```auto
{
"_index": "testindex-",
"_type": "waflog",
"_id": "4m2B7W0BpWxPtAhBShPK",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1
}
```
I changed the index in the Postman request, as there was a mapping clash using the "awswaf-" index, and made this change to the python script too, but still got the same error in the python script with a new index.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 21, 2019, 9:16am UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/9 "2019-10-21T09:16:31Z")
Ok, that tells us the issue lies in how your Python code is sending its request. I think you can [enable some logging to see the exact request](https://stackoverflow.com/questions/10588644/how-can-i-see-the-entire-http-request-thats-being-sent-by-my-python-application) so you can compare it to what you're sending via Postman.
---
### Author:  [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)
#### Post date: [October 21, 2019, 2:14pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/10 "2019-10-21T14:14:43Z")
I think the only difference I can see between the requests is that my Python script is sending the JSON with a [' '] envelope. Could this be causing the error?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 21, 2019, 2:18pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/11 "2019-10-21T14:18:41Z")
Yes, a surrounding pair of quotes would indeed result in the exception you're getting:
```auto
$ curl 'http://localhost:9200/awswaf-/_doc' -H 'Content-type: application/json' --data-binary $'\'{"terminatingRuleId":"Default_Action","rateBasedRuleList":[{"limitKey":"IP","rateBasedRuleId":"ea04e5d3-bcc1-499b-ab7a-583ef849d48a","maxRateAllowed":100}],"terminatingRuleType":"REGULAR","webaclId":"1950425e-9278-4c05-8c67-4d500b9659be","action":"ALLOW","httpRequest":{"requestId":null,"httpMethod":"GET","args":"","httpVersion":"HTTP/1.1","country":"JP","uri":"/","headers":[{"value":"1.2.3.4","name":"Host"},{"value":"HTTP Banner Detection (https://security.ipip.net)","name":"User-Agent"}],"clientIp":"1.2.3.4"},"formatVersion":1,"httpSourceName":"ALB","nonTerminatingMatchingRules":[],"httpSourceId":"639917743964-app/XXXXXXXXXX/2d9c1970fc485a41","ruleGroupList":[],"timestamp":1571137876737}\''
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}},"status":400}
```
---
### Author:  [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)
#### Post date: [October 21, 2019, 2:38pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/12 "2019-10-21T14:38:36Z")
Would that apply for double quotes as well?
This is the raw value that I'm now passing:
```auto
"{\"timestamp\":1571649558020,\"formatVersion\":1,\"webaclId\":\"1950425e-9278-4c05-8c67-4d500b9659be\",\"terminatingRuleId\":\"Default_Action\",\"terminatingRuleType\":\"REGULAR\",\"action\":\"ALLOW\",\"httpSourceName\":\"ALB\",\"httpSourceId\":\"639917743964-app/XXXXXXXXX/2d9c1970fc485a41\",\"ruleGroupList\":[],\"rateBasedRuleList\":[{\"rateBasedRuleId\":\"ea04e5d3-bcc1-499b-ab7a-583ef849d48a\",\"limitKey\":\"IP\",\"maxRateAllowed\":100}],\"nonTerminatingMatchingRules\":[],\"httpRequest\":{\"clientIp\":\"xxx.xxx.xxx.xxx\",\"country\":\"FR\",\"headers\":[{\"name\":\"Host\",\"value\":\"xxx.xxx.xxx.xxx\"},{\"name\":\"User-Agent\",\"value\":\"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36\"}],\"uri\":\"/\",\"args\":\"\",\"httpVersion\":\"HTTP/1.1\",\"httpMethod\":\"GET\",\"requestId\":null}}\n"
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 21, 2019, 2:59pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/13 "2019-10-21T14:59:59Z")
Unfortunately it's not clear what you mean. It definitely won't work if Elasticsearch sees the exact message you have shared there, quotes and all, but maybe something will remove those quotes first?
---
### Author:  [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)
#### Post date: [October 21, 2019, 4:30pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/14 "2019-10-21T16:30:30Z")
Hi David,
Thanks for all your assistance, I've managed to get this working now.
I am, however, struggling to get my index mappings to apply properly, for example my epoch millis timestamp field is being displayed as a number, which I can't change in the dev console or in the index patterns menu.
Also, my nested options don't seem to be working properly, despite the console telling me that the index is created successfully.
Could you help me to understand what I'm doing wrong there?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 18, 2019, 4:30pm UTC](https://discuss.elastic.co/t/bad-request-response-when-adding-waf-logs-to-elasticsearch/204093/15 "2019-11-18T16:30:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.