UI Filters in APM not Working

[Vladimiro (Vladimiro Corsi)]

Hi,

I am running Kibana 7.7.0. After upgrading to APM Server 7.7.1 (from 7.4.0) I noticed that UI filters in APM page do not work anymore: does not filter by environment, host etc.
If I look at my browser (Chrome or Firefox) dev tools I can see that requests like

`https://kibana.foo.fo/s/myspace/api/apm/services/MyService/transaction_groups?start=2020-06-09T05%3A58%3A58.424Z&end=2020-06-09T06%3A20%3A40.865Z&transactionType=custom&uiFilters=%7B%22kuery%22%3A%22host.hostname%20%3A%20%5C%22myhost%5C%22%22%2C%22environment%22%3A%22dev%22%7D`

return results which are not filtered according to the criteria. The strange thing is that filtering works for some time after the indexes are created for the first time: If I stop APM Server, delete all APM indexes and restart, then filtering works for "some time". Can you help me solve this?

Best

Vladimiro

[dgieselaar (Dario Gieselaar)]

Hello @Vladimiro! I cannot reproduce your issue right away. I might need some more info:

• What page are you looking at?
• What filters are you applying?
• What data do you expect to be filtered while it's not? does it apply to all pieces of data, or just one instance (e.g. the transaction group list)?
• Are you using the query bar (in the header) to filter, or the contextual filters?

[Vladimiro (Vladimiro Corsi)]

Hi Dario,

Any page of APM UI (so under /s/myspace/app/apm#/services) is affected.
Query bar is not working: all transactions are returned and metrics (TPM, latency percentiles) are calculated by aggregating as if no query was defined. Even if a filter with a non-existing label for example, all transaction data are returned.

Same for contextual filters: e.g. Environment, Host.

Data look stored correctly on the indices: service.environment is set, labels are set, host.hostname is set.

I have just switched back to APM Server 7.4.0 and for the moment everything works well.

Best

Vladimiro

[Vladimiro (Vladimiro Corsi)]

Hi,

Today (after an apm-transaction index rollover) it stopped working again.

best

vladimiro

[simitt (Silvia Mitter)]

Hi @Vladimiro,
sorry to hear that you are experiencing these issues.

Please check the APM Server logs for any errors.

It would also be interesting to check if any of the managed indices are unhealthy or show any problems. Navigating to Management/Index Management should give you more information about the healthiness of the indices. Alternatively you could run GET apm*/_ilm/explain and check for potential issues.

Additionally I'd be interested in seeing if the APM templates are set up properly - have you customized anything related to ILM or template setup in your apm-server.yml?
For checking if the apm templates are set up as expected, you can run GET _template/apm*. This should return you a template with a mapping containing amongst others these dynamic_templates:

        {
          "labels" : {
            "path_match" : "labels.*",
            "mapping" : {
              "type" : "keyword"
            },
            "match_mapping_type" : "string"
          }
        },
        {
          "labels" : {
            "path_match" : "labels.*",
            "mapping" : {
              "type" : "boolean"
            },
            "match_mapping_type" : "boolean"
          }
        },
        {
          "labels" : {
            "path_match" : "labels.*",
            "mapping" : {
              "scaling_factor" : 1000000,
              "type" : "scaled_float"
            },
            "match_mapping_type" : "*"
          }
        },

If everything mentioned looks ok, could you maybe provide us an example transaction document containing some of the fields you would like to query on? Please remove all sensitive information before providing a document here.

[Vladimiro (Vladimiro Corsi)]

Hi Silvia,
Thanks for your reply. Hereunder the information you requested.

APM Errors
Today I am experiencing occasional errors like this:
Jun 12 11:44:44 apm apm-server[32307]: 2020-06-12T13:44:44.949+0200#011ERROR#011[request]#011middleware/log_middleware.go:74#011data validatio n error#011{"request_id": "d9a0d7e9-7b4a-460b-b2ce-4b44da5f61b1", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_ address": "*", "user-agent": "elasticapm-java/1.12.0", "response_code": 400, "error": "read tcp *:8200->*:40094: i/o timeout"}

Also visible in the Monitoring graphs

Index Management Errors
Nothing here

ILM Explain
https://pastebin.com/A2pNCaFY

APM Templates
https://pastebin.com/gtuJuuSE

Sample Transaction
https://pastebin.com/zkx31CKS

I want to point out that my issue is not that I cannot retrieve the transaction I pasted. It is probably collected amidst all the others that are not filtered out by the filters.

Again many thanks for your help.

Best

Vladimiro

[simitt (Silvia Mitter)]

Hi @Vladimiro,
thanks for providing these additional details. I cannot see anything suspicious in the files you provided, so I am going to consult with some colleagues and hope we are able to come back to you soon.

[dgieselaar (Dario Gieselaar)]

@Vladimiro: can you share a screenshot of the APM UI, one where you have no filters and one where you have filters applied? Again please scrub out any sensitive information.

[Vladimiro (Vladimiro Corsi)]

Hi @dgieselaar,

Here are 2 screenshots: with and without filtering by environment. Data shown is the same, whilst in dev I have almost no data.

Best

Vladimiro