# Detections API cant work with Unicode characters
**URL:** https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781
**Category:** Elastic Security
**Created:** [January 25, 2024, 10:10am UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781 "2024-01-25T10:10:07Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@gbL2k](https://discuss.elastic.co/u/gbL2k)
#### Post date: [January 25, 2024, 10:10am UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/1 "2024-01-25T10:10:07Z")
Hello everyone!
I would like to to create detections with unicode characters in the description but I cant seem to find a way to make the API work with unicode characters.
Anyone had the same problem?
Thanks in advance!
---
### Author:  [@Juan\_Pablo\_Djeredjia](https://discuss.elastic.co/u/Juan_Pablo_Djeredjia)
#### Post date: [January 25, 2024, 11:49am UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/2 "2024-01-25T11:49:32Z")
Hi @gbL2k, thanks for reaching out!
Could you post here an example of the request that you are making (endpoint, payload, etc), and the error that you are getting back?
---
### Author:  [@gbL2k](https://discuss.elastic.co/u/gbL2k)
#### Post date: [January 25, 2024, 1:29pm UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/3 "2024-01-25T13:29:32Z")
I can ,sure.
The endpoint is /api/detection\_engine/rules with a POST.
Here is an example payload:
```auto
{
"author": ["test"],
"description": "test",
"from": "now-6m",
"name": "test íú",
"risk_score": 100,
"severity": "medium",
"index": ["logs-*"],
"type": "query",
"rule_id": "00000000-0000-0000-0000-00000000000f",
"language": "lucene",
"query": "(process.executable:*) ",
"false_positives": ["test"],
"note": "test: áéő ",
"threat": [{"framework": "MITRE ATT&CK", "tactic": {"id": "TA0006", "name": "Credential Access", "reference": "https://attack.mitre.org/tactics/TA0006/"}, "technique": [{"id": "T1003", "name": "OS Credential Dumping", "reference": "https://attack.mitre.org/techniques/T1003/", "subtechnique": [{"id": "T1003.001", "name": "OS Credential Dumping: LSASS Memory", "reference": "https://attack.mitre.org/techniques/T1003/001/"}]}]}],
"enabled": true
}
```
It only fails if I have Unicode characters in the note or description field.
The response is 400.
---
### Author:  [@Juan\_Pablo\_Djeredjia](https://discuss.elastic.co/u/Juan_Pablo_Djeredjia)
#### Post date: [January 25, 2024, 5:54pm UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/4 "2024-01-25T17:54:30Z")
Hi @gbL2k . I just ran that request to create a rule with your exact same payload and the rule was created succesfully:
I think we need to evaluate if the 400 response is being cause by some other issue in the body payload or headers.
A couple questions:
- do you get an error message along with the 400 status code? How are you running this request? curl, Postman, a script, etc?
- what does your Elastic setup look like? Are you self-managed or Cloud?
- How are you authenticating your request?
---
### Author:  [@gbL2k](https://discuss.elastic.co/u/gbL2k)
#### Post date: [January 26, 2024, 10:34am UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/5 "2024-01-26T10:34:52Z")
Thanks for the answer it helps a lot,now I know I have to look somewhere else.
We have tested and its working from postman.
However if we want to make the request from python it will not work.
The elastic setup is self-managed.
We use API key authentication.
The request in python looks as the following:
` elastic_data=requests.post(url,headers=headers, data=data,verify=False)`
where the data is the one seen above and the headers are the following:
```auto
headers = {
'Content-Type': 'application/json;charset=UTF-8',
'kbn-xsrf': 'true',
'Authorization': 'ApiKey ' + api_key
}
```
---
### Author:  [@Juan\_Pablo\_Djeredjia](https://discuss.elastic.co/u/Juan_Pablo_Djeredjia)
#### Post date: [January 26, 2024, 11:31am UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/6 "2024-01-26T11:31:29Z")
So I can help you better, would you mind pasting the whole code snippet of the request? Of course, redact the API key.
Also, what error messages do you see? Can you add some logging?
---
### Author:  [@gbL2k](https://discuss.elastic.co/u/gbL2k)
#### Post date: [January 26, 2024, 11:50am UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/7 "2024-01-26T11:50:21Z")
Sure,its quite simple and short, most of the scripts is only building a valid json object.Which it does perfectly.Unfortunately I only see a 400 response and nothing else.
```auto
import requests
import os
import tomllib
import sys
import json
#example usage: update_alert.py Detections https://127.0.0.1:5601/api/detection_engine/rules API_KEY
url = sys.argv[2]
api_key = sys.argv[3]
headers = {
'Content-Type': 'application/json;charset=UTF-8',
'kbn-xsrf': 'true',
'Authorization': 'ApiKey ' + api_key
}
directory=sys.argv[1]
files = os.listdir(directory)
for file_name in files:
file_path=os.path.join(directory,file_name)
if file_path.endswith(".toml"):
with open(file_path,"rb") as toml:
alert=tomllib.load(toml)
data=""
data+="{\n"
for field in alert['rule']:
if type(alert['rule'][field]) == list:
data += " " + "\"" + field + "\": " + str(alert['rule'][field]).replace("'","\"") + "," + "\n"
elif type(alert['rule'][field]) == str:
if field == 'description':
data += " " + "\"" + field + "\": \"" + str(alert['rule'][field]).replace("\n"," ").replace("\"","\\\"").replace("\\","\\\\") + "\"," + "\n"
elif field == 'query':
data += " " + "\"" + field + "\": \"" + str(alert['rule'][field]).replace("\\","\\\\").replace("\"","\\\"").replace("\n"," ") + "\"," + "\n"
else:
data += " " + "\"" + field + "\": \"" + str(alert['rule'][field]).replace("\n"," ").replace("\"","\\\"") + "\"," + "\n"
elif type(alert['rule'][field]) == int:
data += " " + "\"" + field + "\": " + str(alert['rule'][field]) + "," + "\n"
elif type(alert['rule'][field]) == dict:
data += " " + "\"" + field + "\": " + str(alert['rule'][field]).replace("'","\"") + "," + "\n"
data+=" \"enabled\": true\n}\n"
print(data)
elastic_data=requests.post(url,headers=headers, data=data,verify=False)
print(elastic_data)
```
---
### Author:  [@Juan\_Pablo\_Djeredjia](https://discuss.elastic.co/u/Juan_Pablo_Djeredjia)
#### Post date: [January 26, 2024, 11:53am UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/8 "2024-01-26T11:53:59Z")
Can you post here the results from
`print(data)`
`print(elastic_data)`
?
---
### Author:  [@gbL2k](https://discuss.elastic.co/u/gbL2k)
#### Post date: [January 26, 2024, 11:56am UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/9 "2024-01-26T11:56:05Z")
The output of data is the json seen above and the output of elastic\_data is actually the response from elastic which is "\"
---
### Author:  [@gbL2k](https://discuss.elastic.co/u/gbL2k)
#### Post date: [January 26, 2024, 11:59am UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/10 "2024-01-26T11:59:47Z")
Maybe the problem is actually in python and the requests library?
---
### Author:  [@Juan\_Pablo\_Djeredjia](https://discuss.elastic.co/u/Juan_Pablo_Djeredjia)
#### Post date: [January 26, 2024, 12:07pm UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/11 "2024-01-26T12:07:12Z")
> Maybe the problem is actually in python and the requests library?
It's possible. The payload looks correctly formatted.
Can you try adding the following header?
`elastic-api-version: 2023-10-31`
Also, what about the URL? Can you print that out? Just to do a sanity check that we are pointing to the expected endpoint.
---
### Author:  [@gbL2k](https://discuss.elastic.co/u/gbL2k)
#### Post date: [January 26, 2024, 12:12pm UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/12 "2024-01-26T12:12:33Z")
I have added the header and nothing changed.
The uri is the following:
https://ELASTIC\_IP:5601/api/detection\_engine/rules
---
### Author:  [@Juan\_Pablo\_Djeredjia](https://discuss.elastic.co/u/Juan_Pablo_Djeredjia)
#### Post date: [January 26, 2024, 12:15pm UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/13 "2024-01-26T12:15:39Z")
> [@gbL2k](#):
>
> https://ELASTIC\_IP:5601/api/detection\_engine/rules
Try:
`https://ELASTIC_IP:5601/kbn/api/detection_engine/rules`
Does the response change to `404`?
---
### Author:  [@gbL2k](https://discuss.elastic.co/u/gbL2k)
#### Post date: [January 26, 2024, 12:20pm UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/14 "2024-01-26T12:20:51Z")
No, its remained the same.
---
### Author:  [@Juan\_Pablo\_Djeredjia](https://discuss.elastic.co/u/Juan_Pablo_Djeredjia)
#### Post date: [January 26, 2024, 12:24pm UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/15 "2024-01-26T12:24:39Z")
Ok, let's try to get more information from the response. Try adding:
```auto
try:
error_message = elastic_data.json()
print("Error Message:", error_message)
except json.JSONDecodeError:
print("Error decoding JSON in response body.")
```
---
### Author:  [@gbL2k](https://discuss.elastic.co/u/gbL2k)
#### Post date: [January 26, 2024, 12:29pm UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/16 "2024-01-26T12:29:55Z")
I did and got the answer:""Error decoding JSON in response body."
We have also tried to gather some more information why the request failed with this exact method but it seems we only get a response code back.
---
### Author:  [@Juan\_Pablo\_Djeredjia](https://discuss.elastic.co/u/Juan_Pablo_Djeredjia)
#### Post date: [January 26, 2024, 12:45pm UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/17 "2024-01-26T12:45:34Z")
Ok, so I created a Python script and replicated the issue.
It's a simpler issue 😅: you need to properly format you data as JSON before passing it to the `requests` library:
```auto
import json
# rest of code...
elastic_data = requests.post(url, headers=headers, data=json.dumps(data), verify=False)
```
I got 200 with this fix, and a properly created rule, with Unicode characters.
Also, for next time when debugging the response, use `print(elastic_data. __dict__ )`
---
### Author:  [@gbL2k](https://discuss.elastic.co/u/gbL2k)
#### Post date: [January 26, 2024, 1:10pm UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/18 "2024-01-26T13:10:45Z")
Then I will offically go insanse over why its still not working.
We have also tried this method before.
Would you mind pasting the code here?
---
### Author:  [@gbL2k](https://discuss.elastic.co/u/gbL2k)
#### Post date: [January 26, 2024, 1:15pm UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/19 "2024-01-26T13:15:28Z")
Either way thanks for the help Pablo.
---
### Author:  [@Juan\_Pablo\_Djeredjia](https://discuss.elastic.co/u/Juan_Pablo_Djeredjia)
#### Post date: [January 26, 2024, 1:19pm UTC](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781/20 "2024-01-26T13:19:06Z")
This works for me and I think should work for you as well:
```auto
import requests
import json
headers = {
'Content-Type': 'application/json;charset=UTF-8',
'kbn-xsrf': 'true',
'Authorization': "", # Your auth here
'elastic-api-version': '2023-10-31'
}
data = {
"author": ["test"],
"description": "test",
"from": "now-6m",
"name": "test íú",
"risk_score": 100,
"severity": "medium",
"index": ["logs-*"],
"type": "query",
"rule_id": "00000000-0000-0000-0000-0000adfa0000000f",
"language": "lucene",
"query": "(process.executable:*) ",
"false_positives": ["test"],
"note": "test: áéő ",
"threat": [
{
"framework": "MITRE ATT&CK",
"tactic": {
"id": "TA0006",
"name": "Credential Access",
"reference": "https://attack.mitre.org/tactics/TA0006/"
},
"technique": [
{
"id": "T1003",
"name": "OS Credential Dumping",
"reference": "https://attack.mitre.org/techniques/T1003/",
"subtechnique": [
{
"id": "T1003.001",
"name": "OS Credential Dumping: LSASS Memory",
"reference": "https://attack.mitre.org/techniques/T1003/001/"
}
]
}
]
}
],
"enabled": True
}
# I'm running Kibana locally, but use your own url
url = 'http://localhost:5601/kbn/api/detection_engine/rules'
elastic_data = requests.post(url, headers=headers, data=json.dumps(data), verify=False)
print(elastic_data. __dict__ )
```
[Next page](https://discuss.elastic.co/t/detections-api-cant-work-with-unicode-characters/351781.md?page=2)