# Failed to load resource: the server responded with a status of 500 (Internal Server Error)
**URL:** https://discuss.elastic.co/t/failed-to-load-resource-the-server-responded-with-a-status-of-500-internal-server-error/225612
**Category:** APM
**Tags:** ui
**Created:** [March 30, 2020, 5:15am UTC](https://discuss.elastic.co/t/failed-to-load-resource-the-server-responded-with-a-status-of-500-internal-server-error/225612 "2020-03-30T05:15:44Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Agzem](https://discuss.elastic.co/u/Agzem)
#### Post date: [March 30, 2020, 5:15am UTC](https://discuss.elastic.co/t/failed-to-load-resource-the-server-responded-with-a-status-of-500-internal-server-error/225612/1 "2020-03-30T05:15:44Z")
**Kibana version** : 7.6.1
**Elasticsearch version** : 7.6.1
**APM Server version** : 7.6.1
**APM Agent language and version** : Java Maven Agent, 1.14.0
**Browser version** : Chrome Latest
**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: Helm from elastic github repository
**Fresh install or upgraded from other version?** : Fresh install
**Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
No, only custom Index on APM-server that will dynamically create index based on Agent environment setting deployed on Kubernetes.
**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:
I've config APM-Server with custom Index pattern e.g. `foo-bar-apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}`, my Spring Java Agent is able to feed its log to APM-Server and log showed up in Kibana UI correctly, but for APM UI at first it's not showing any log since I did not set the indices in UI yet, after I set the indices it responsed with:
"Failed to load resource: the server responded with a status of 500 (Internal Server Error)"
Error :
Indices Configuration:
**Steps to reproduce** :
1. config custom index pattern in APM-Server configmap.
2. startup APM-Server.
3. startup APM Agent.
4. make HTTP Request.
5. create new APM index from elasticsearch, verify that data is correct.
6. in APM UI, config indices to `*-apm-*`
7. click apply return to main menu in the APM UI, verify errors.
**Errors in browser console (if relevant)**:
```auto
"Failed to load resource: the server responded with a status of 500 (Internal Server Error)"
```
```auto
Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
```
**Provide logs and/or server output (if relevant)**:\
```auto
{"type":"error","@timestamp":"2020-03-29T21:45:54Z","tags":[],"pid":6,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:63:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:165:34)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?start=2020-03-29T21%3A13%3A15.038Z&end=2020-03-29T21%3A28%3A15.039Z","query":{"start":"2020-03-29T21:13:15.038Z","end":"2020-03-29T21:28:15.039Z"},"pathname":"/api/apm/ui_filters/environments","path":"/api/apm/ui_filters/environments?start=2020-03-29T21%3A13%3A15.038Z&end=2020-03-29T21%3A28%3A15.039Z","href":"/api/apm/ui_filters/environments?start=2020-03-29T21%3A13%3A15.038Z&end=2020-03-29T21%3A28%3A15.039Z"},"message":"Internal Server Error"}
```
---
### Author:  [@caue.marcondes](https://discuss.elastic.co/u/caue.marcondes)
#### Post date: [March 30, 2020, 2:04pm UTC](https://discuss.elastic.co/t/failed-to-load-resource-the-server-responded-with-a-status-of-500-internal-server-error/225612/2 "2020-03-30T14:04:05Z")
> [@Agzem](#):
>
> foo-bar-apm
Hi @Agzem, thanks for your message.
Can you please run this command and let me know what's the result?
`GET _cat/indices`
---
### Author:  [@Agzem](https://discuss.elastic.co/u/Agzem)
#### Post date: [March 31, 2020, 3:03am UTC](https://discuss.elastic.co/t/failed-to-load-resource-the-server-responded-with-a-status-of-500-internal-server-error/225612/3 "2020-03-31T03:03:56Z")
Hi @caue.marcondes, here's the result from `cat indices `
---
### Author:  [@caue.marcondes](https://discuss.elastic.co/u/caue.marcondes)
#### Post date: [March 31, 2020, 10:11am UTC](https://discuss.elastic.co/t/failed-to-load-resource-the-server-responded-with-a-status-of-500-internal-server-error/225612/4 "2020-03-31T10:11:37Z")
@Agzem,
I've been trying to reproduce the problem but unfortunately, I couldn't.
I set up an APM-server with a custom index name. Check how it looks when I run the `_cat` API:
This is the indices page on APM:
Have you created a new index pattern on Kibana settings? If yes can you browse your transactions on Kibana `Discover`?
---
### Author:  [@Agzem](https://discuss.elastic.co/u/Agzem)
#### Post date: [March 31, 2020, 1:33pm UTC](https://discuss.elastic.co/t/failed-to-load-resource-the-server-responded-with-a-status-of-500-internal-server-error/225612/5 "2020-03-31T13:33:58Z")
@caue.marcondes
> Have you created a new index pattern on Kibana settings?
I did created one based on what apm-server send to elasticsearch
> If yes can you browse your transactions on Kibana `Discover` ?
the weird thing is, yes I can search all of my apm data normally with no issue at all in my apm index in discover. ☹
so I went into my Kibana pod and press refresh with indices setting while in apm UI to reproduce this error again and I found some error that I missed to included(this one came before the one I included in the first post).
it's a bit long so I cut and masked some parts.
```auto
{ Error: [illegal_argument_exception]
Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment]
in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting,
so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index.
Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting,
so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index.
Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting,
so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index.
Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting,
so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index.
Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting,
so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index.
Note that this can use significant memory.
at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js: 349: 15)
at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js: 306: 7)
at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js: 173: 7)
at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js: 4929: 19)
at IncomingMessage.emit (events.js: 203: 15)
at endReadableNT (_stream_readable.js: 1145: 12)
at process._tickCallback (internal/process/next_tick.js: 63: 19)
status: 400,
displayName: 'BadRequest',
message:
'[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead.
Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and)
[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead.
Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and)
[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead.
Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and)
[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead.
Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and)
[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead.
Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and)
[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead.
Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory.',
path: ' /*-apm-*%2C*-apm-*%2C*-apm-*/_search',
query: {
ignore_throttled: true
},
body: {
error: {
root_cause:
[Array],
type: 'search_phase_execution_exception',
reason: 'all shards failed',
phase: 'query',
grouped: true,
failed_shards: [Array],
caused_by: [Object]
},
status: 400
},
statusCode: 400,
response:
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
},
{
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
},
{
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
},
{
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
},
{
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
},
{
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "env-dev-apm-7.6.1-metric-2020.03.31",
"node": "b-61-4wJT3CG2aP2RgoGhQ",
"reason": {
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
},
{
"shard": 0,
"index": "foo-bar-env-apm-7.6.1-transaction-2020.03.31",
"node": "NkEVjGr2SCGi2Xlu_v41jA",
"reason": {
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
}
],
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory.",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
}
},
"status": 400
},
toString: [Function],
toJSON: [Function]
}
```
---
### Author:  [@Agzem](https://discuss.elastic.co/u/Agzem)
#### Post date: [March 31, 2020, 1:37pm UTC](https://discuss.elastic.co/t/failed-to-load-resource-the-server-responded-with-a-status-of-500-internal-server-error/225612/6 "2020-03-31T13:37:00Z")
this is my index output setting
```auto
index: "%{[service.environment]}-apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"
setup:
template:
name: "apm-server"
pattern: "apm-*"
```
---
### Author:  [@caue.marcondes](https://discuss.elastic.co/u/caue.marcondes)
#### Post date: [April 1, 2020, 7:41am UTC](https://discuss.elastic.co/t/failed-to-load-resource-the-server-responded-with-a-status-of-500-internal-server-error/225612/7 "2020-04-01T07:41:49Z")
@Agzem I can reproduce the error now. Will take a look and reach back to you. Thanks for your help.
---
### Author:  [@caue.marcondes](https://discuss.elastic.co/u/caue.marcondes)
#### Post date: [April 1, 2020, 9:30am UTC](https://discuss.elastic.co/t/failed-to-load-resource-the-server-responded-with-a-status-of-500-internal-server-error/225612/8 "2020-04-01T09:30:04Z")
@Agzem
I reproduced and fixed the error. It is a problem in the apm-server.yml file. First, you have to make sure `setup.template.name` , `setup.template.pattern` and `output.elasticsearch.index/indices` configs are set up correctly, the config must match between these three options.
In your config file you set up `setup.template.pattern`:`apm-*`, and you change the default index to `%{[service.environment]}-apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}` when apm-server creates the mappings it looks for the value set on `setup.template.pattern`, as they don't match, the mappings are created with the default multi-fileds.
```auto
"service" : {
"properties" : {
"environment" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
...
```
That's why you got this error:
> Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment]
Here is how you can fix the apm-server.yml:
```auto
setup.template.name: "apm-server"
setup.template.pattern: "*-apm-*" # here was the error, your pattern was set as apm-*
output.elasticsearch:
index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
indices:
- index: "%{[service.environment]}-apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"
```
You shouldn't specify `output.elasticsearch.index` with variable parts, if any docs are sent that do not have these variable part set (e.g. service.environment) these docs can’t be ingested and fill up the APM Server queue. That's why I kept `output.elasticsearch.index` as it is, and added my own index on `output.elasticsearch.indices.index`.
After the change here is how the mapping looks like:
```auto
"service" : {
"dynamic" : "false",
"properties" : {
"environment" : {
"type" : "keyword",
"ignore_above" : 1024
},
```
**Beware that to make it work you must delete the template and indices already created. Be careful you will lose your data if you delete it.**
=================================================================================
There's another solution, very similar, but you wouldn't have to be worried about matching all config properties.
```auto
setup.template.name: "apm-server"
setup.template.pattern: "apm-*" # Keep the default value here
output.elasticsearch:
index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
indices:
# place the service.environment after the apm-
- index: "apm-%{[service.environment]}-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"
```
With this setup, everything will keep working as it should and you'll have your indices created by each environment you have. And in the APM UI, you don't have to change the indices as well.
Please let me know if you have any further questions.
---
### Author:  [@Agzem](https://discuss.elastic.co/u/Agzem)
#### Post date: [April 1, 2020, 3:31pm UTC](https://discuss.elastic.co/t/failed-to-load-resource-the-server-responded-with-a-status-of-500-internal-server-error/225612/9 "2020-04-01T15:31:02Z")
@caue.marcondes
Thanks for very detailed walkthrough, so it really caused by my template pattern 😥 .
Also your comment cleared up alot of question in my head until now, especially the queue size, during some of my internal load testing my Java Agent are just getting `HTTP 503 : queue is full` error and I was trying to fix it by increase the `queue.mem`.
with your help, my APM UI is working now. Thanks again for your help 😃 .
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 22, 2020, 11:31am UTC](https://discuss.elastic.co/t/failed-to-load-resource-the-server-responded-with-a-status-of-500-internal-server-error/225612/10 "2020-04-22T11:31:05Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.