Failed to load resource: the server responded with a status of 500 (Internal Server Error)

Kibana version: 7.6.1

Elasticsearch version: 7.6.1

APM Server version: 7.6.1

APM Agent language and version: Java Maven Agent, 1.14.0

Browser version: Chrome Latest

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Helm from elastic github repository

Fresh install or upgraded from other version?: Fresh install

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

No, only custom Index on APM-server that will dynamically create index based on Agent environment setting deployed on Kubernetes.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I've config APM-Server with custom Index pattern e.g. foo-bar-apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}, my Spring Java Agent is able to feed its log to APM-Server and log showed up in Kibana UI correctly, but for APM UI at first it's not showing any log since I did not set the indices in UI yet, after I set the indices it responsed with:
"Failed to load resource: the server responded with a status of 500 (Internal Server Error)"

Error :

Indices Configuration:

Steps to reproduce:

  1. config custom index pattern in APM-Server configmap.
  2. startup APM-Server.
  3. startup APM Agent.
  4. make HTTP Request.
  5. create new APM index from elasticsearch, verify that data is correct.
  6. in APM UI, config indices to *-apm-*
  7. click apply return to main menu in the APM UI, verify errors.

Errors in browser console (if relevant):

 "Failed to load resource: the server responded with a status of 500 (Internal Server Error)"

Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.

Provide logs and/or server output (if relevant):\

{"type":"error","@timestamp":"2020-03-29T21:45:54Z","tags":[],"pid":6,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:63:19)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:165:34)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?start=2020-03-29T21%3A13%3A15.038Z&end=2020-03-29T21%3A28%3A15.039Z","query":{"start":"2020-03-29T21:13:15.038Z","end":"2020-03-29T21:28:15.039Z"},"pathname":"/api/apm/ui_filters/environments","path":"/api/apm/ui_filters/environments?start=2020-03-29T21%3A13%3A15.038Z&end=2020-03-29T21%3A28%3A15.039Z","href":"/api/apm/ui_filters/environments?start=2020-03-29T21%3A13%3A15.038Z&end=2020-03-29T21%3A28%3A15.039Z"},"message":"Internal Server Error"}

Hi @Agzem, thanks for your message.

Can you please run this command and let me know what's the result?

GET _cat/indices

Hi @caue.marcondes, here's the result from cat indices

@Agzem,

I've been trying to reproduce the problem but unfortunately, I couldn't.

I set up an APM-server with a custom index name. Check how it looks when I run the _cat API:

This is the indices page on APM:

Have you created a new index pattern on Kibana settings? If yes can you browse your transactions on Kibana Discover?

@caue.marcondes

Have you created a new index pattern on Kibana settings?

I did created one based on what apm-server send to elasticsearch

If yes can you browse your transactions on Kibana Discover ?

the weird thing is, yes I can search all of my apm data normally with no issue at all in my apm index in discover. :frowning:

so I went into my Kibana pod and press refresh with indices setting while in apm UI to reproduce this error again and I found some error that I missed to included(this one came before the one I included in the first post).

it's a bit long so I cut and masked some parts.

{ Error: [illegal_argument_exception] 
    Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] 
    in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, 
    so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. 
    Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, 
    so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. 
    Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, 
    so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. 
    Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, 
    so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. 
    Note that this can use significant memory. (and) [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, 
    so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. 
    Note that this can use significant memory.
    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js: 349: 15)
    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js: 306: 7)
    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js: 173: 7)
    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js: 4929: 19)
    at IncomingMessage.emit (events.js: 203: 15)
    at endReadableNT (_stream_readable.js: 1145: 12)
    at process._tickCallback (internal/process/next_tick.js: 63: 19)
  status: 400,
  displayName: 'BadRequest',
  message:
   '[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. 
   Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and)
    [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. 
    Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and) 
    [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. 
    Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and) 
    [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. 
    Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and) 
    [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. 
    Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (and)
    [illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. 
    Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory.',
  path: ' /*-apm-*%2C*-apm-*%2C*-apm-*/_search',
  query: { 
      ignore_throttled: true 
    },
  body: { 
      error: { 
        root_cause: 
            [Array],
            type: 'search_phase_execution_exception',
            reason: 'all shards failed',
            phase: 'query',
            grouped: true,
            failed_shards: [Array],
            caused_by: [Object]
        },
    status: 400
    },
  statusCode: 400,
  response:
   {
        "error": {
            "root_cause": [
                {
                    "type": "illegal_argument_exception",
                    "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
                },
                {
                    "type": "illegal_argument_exception",
                    "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
                },
                {
                    "type": "illegal_argument_exception",
                    "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
                },
                {
                    "type": "illegal_argument_exception",
                    "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
                },
                {
                    "type": "illegal_argument_exception",
                    "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
                },
                {
                    "type": "illegal_argument_exception",
                    "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
                }
            ],
            "type": "search_phase_execution_exception",
            "reason": "all shards failed",
            "phase": "query",
            "grouped": true,
            "failed_shards": [
                {
                    "shard": 0,
                    "index": "env-dev-apm-7.6.1-metric-2020.03.31",
                    "node": "b-61-4wJT3CG2aP2RgoGhQ",
                    "reason": {
                        "type": "illegal_argument_exception",
                        "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
                    }
                },
                {
                    "shard": 0,
                    "index": "foo-bar-env-apm-7.6.1-transaction-2020.03.31",
                    "node": "NkEVjGr2SCGi2Xlu_v41jA",
                    "reason": {
                        "type": "illegal_argument_exception",
                        "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
                    }
                }
            ],
            "caused_by": {
                "type": "illegal_argument_exception",
                "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory.",
                "caused_by": {
                    "type": "illegal_argument_exception",
                    "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
                }
            }
        },
        "status": 400
    },
  toString: [Function],
  toJSON: [Function]
}

this is my index output setting

 index: "%{[service.environment]}-apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"

    setup:
      template:
        name: "apm-server"
        pattern: "apm-*"

@Agzem I can reproduce the error now. Will take a look and reach back to you. Thanks for your help.

@Agzem

I reproduced and fixed the error. It is a problem in the apm-server.yml file. First, you have to make sure setup.template.name , setup.template.pattern and output.elasticsearch.index/indices configs are set up correctly, the config must match between these three options.

In your config file you set up setup.template.pattern:apm-*, and you change the default index to %{[service.environment]}-apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd} when apm-server creates the mappings it looks for the value set on setup.template.pattern, as they don't match, the mappings are created with the default multi-fileds.

"service" : {
          "properties" : {
            "environment" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
...

That's why you got this error:

Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment]

Here is how you can fix the apm-server.yml:

setup.template.name: "apm-server"
setup.template.pattern: "*-apm-*" # here was the error, your pattern was set as apm-*
output.elasticsearch:
index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
  indices:
    - index: "%{[service.environment]}-apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"

You shouldn't specify output.elasticsearch.index with variable parts, if any docs are sent that do not have these variable part set (e.g. service.environment) these docs can’t be ingested and fill up the APM Server queue. That's why I kept output.elasticsearch.index as it is, and added my own index on output.elasticsearch.indices.index.

After the change here is how the mapping looks like:

"service" : {
          "dynamic" : "false",
          "properties" : {
            "environment" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },

Beware that to make it work you must delete the template and indices already created. Be careful you will lose your data if you delete it.

=================================================================================

There's another solution, very similar, but you wouldn't have to be worried about matching all config properties.

setup.template.name: "apm-server"
setup.template.pattern: "apm-*" # Keep the default value here
output.elasticsearch:
index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
  indices:
    # place the service.environment after the apm-
    - index: "apm-%{[service.environment]}-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"

With this setup, everything will keep working as it should and you'll have your indices created by each environment you have. And in the APM UI, you don't have to change the indices as well.

Please let me know if you have any further questions.

2 Likes

@caue.marcondes

Thanks for very detailed walkthrough, so it really caused by my template pattern :disappointed_relieved: .
Also your comment cleared up alot of question in my head until now, especially the queue size, during some of my internal load testing my Java Agent are just getting HTTP 503 : queue is full error and I was trying to fix it by increase the queue.mem.

with your help, my APM UI is working now. Thanks again for your help :smiley: .

1 Like

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.