Duplicated transaction with different ip

Elastic Observability APM
1

Kibana version: 6.6.1

Elasticsearch version: 6.6.1

APM Server version: 6.6.1

APM Agent language and version: apm-agent-rum-js@3.0.0

Browser version: Android 9, Chrome/74.0.3729.136

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version?
Fresh install

Is there anything special in your setup?
noop

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

1
1.png2482×1308 365 KB

As screenshot shows, there are two almost the same 'page-load' transactions at the same time, except ip field. How did it happen?

Steps to reproduce:
Haven't reproduce it.

Errors in browser console (if relevant): noop

Provide logs and/or server output (if relevant): noop

2

Hi @zhangsh,

Thanks for reaching out and welcome to the forum.

Would you please send the full document for these transactions? Please redact any personal information that might exist on these documents.

Also, are you using any type of proxy or load balancer for APM server?

As a side note, the current version of the RUM agent is 4.4.2. we recommend to use the latest version since we have made many improvements since version 3.0.0.

Cheers,
Hamid

3

Firstly, thanks for your quickly reply!

Here are the two transactions' JSON:

{
  "_index": "apm-6.6.1-transaction-2019.08.23",
  "_type": "doc",
  "_id": "WKHRvGwBd9z6GtobQ1Ht",
  "_version": 1,
  "_score": null,
  "_source": {
    "trace": {
      "id": "eec9f724e68aecbee68e8485cd056b75"
    },
    "@timestamp": "2019-08-23T04:52:06.106Z",
    "beat": {
      "hostname": "apm-server10-001",
      "name": "apm-server10-001",
      "version": "6.6.1"
    },
    "host": {
      "name": "apm-server10-001"
    },
    "context": {
      "service": {
        "agent": {
          "name": "js-base",
          "version": "3.0.0"
        },
        "name": "name",
        "language": {
          "name": "javascript"
        }
      },
      "page": {
        "referer": "",
        "url": "url"
      },
      "user": {
        "ip": "121.227.106.100",
        "id": "id",
        "user-agent": "Mozilla/5.0 (Linux; Android 9; MI 9 Build/PKQ1.181121.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.136 Mobile Safari/537.36"
      }
    },
    "processor": {
      "name": "transaction",
      "event": "transaction"
    },
    "transaction": {
      "duration": {
        "us": 164300
      },
      "name": "name",
      "span_count": {
        "started": 11
      },
      "id": "ca9d513efa9589ff",
      "marks": {
        "agent": {
          "firstContentfulPaint": 147.69999999953143,
          "domInteractive": 133,
          "domComplete": 333,
          "timeToFirstByte": 1
        },
        "navigationTiming": {
          "responseEnd": 2,
          "responseStart": 1,
          "domInteractive": 133,
          "domainLookupEnd": 0,
          "domContentLoadedEventStart": 133,
          "domComplete": 333,
          "domainLookupStart": 0,
          "connectEnd": 0,
          "connectStart": 0,
          "loadEventStart": 333,
          "requestStart": 1,
          "fetchStart": 0,
          "domContentLoadedEventEnd": 133,
          "loadEventEnd": 333,
          "domLoading": 4
        }
      },
      "type": "page-load",
      "sampled": true
    },
    "timestamp": {
      "us": 1566535926106518
    }
  },
  "fields": {
    "@timestamp": [
      "2019-08-23T04:52:06.106Z"
    ],
    "error id icon": [
      null
    ]
  },
  "highlight": {
    "transaction.id": [
      "@kibana-highlighted-field@ca9d513efa9589ff@/kibana-highlighted-field@"
    ],
    "context.service.name": [
      "@kibana-highlighted-field@serviceName@/kibana-highlighted-field@"
    ],
    "transaction.type": [
      "@kibana-highlighted-field@page-load@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1566535926106
  ]
}

========================

{
  "_index": "apm-6.6.1-transaction-2019.08.23",
  "_type": "doc",
  "_id": "yBnRvGwBjBx0RsJRQ17g",
  "_version": 1,
  "_score": null,
  "_source": {
    "trace": {
      "id": "eec9f724e68aecbee68e8485cd056b75"
    },
    "@timestamp": "2019-08-23T04:52:05.864Z",
    "beat": {
      "hostname": "apm-server10-001",
      "name": "apm-server10-001",
      "version": "6.6.1"
    },
    "host": {
      "name": "apm-server10-001"
    },
    "context": {
      "service": {
        "agent": {
          "name": "js-base",
          "version": "3.0.0"
        },
        "name": "name",
        "language": {
          "name": "javascript"
        },
      },
      "page": {
        "referer": "",
        "url": "url"
      },
      "user": {
        "ip": "223.104.146.18",
        "id": "id",
        "user-agent": "Mozilla/5.0 (Linux; Android 9; MI 9 Build/PKQ1.181121.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.136 Mobile Safari/537.36"
      }
    },
    "processor": {
      "name": "transaction",
      "event": "transaction"
    },
    "transaction": {
      "duration": {
        "us": 164300
      },
      "name": "name",
      "span_count": {
        "started": 11
      },
      "id": "ca9d513efa9589ff",
      "marks": {
        "agent": {
          "firstContentfulPaint": 147.69999999953143,
          "domInteractive": 133,
          "domComplete": 333,
          "timeToFirstByte": 1
        },
        "navigationTiming": {
          "responseEnd": 2,
          "responseStart": 1,
          "domainLookupEnd": 0,
          "domInteractive": 133,
          "domContentLoadedEventStart": 133,
          "domComplete": 333,
          "domainLookupStart": 0,
          "connectEnd": 0,
          "connectStart": 0,
          "loadEventStart": 333,
          "requestStart": 1,
          "fetchStart": 0,
          "domContentLoadedEventEnd": 133,
          "loadEventEnd": 333,
          "domLoading": 4
        }
      },
      "type": "page-load",
      "sampled": true
    },
    "timestamp": {
      "us": 1566535925864622
    }
  },
  "fields": {
    "@timestamp": [
      "2019-08-23T04:52:05.864Z"
    ],
    "error id icon": [
      null
    ]
  },
  "highlight": {
    "transaction.id": [
      "@kibana-highlighted-field@ca9d513efa9589ff@/kibana-highlighted-field@"
    ],
    "context.service.name": [
      "@kibana-highlighted-field@serviceName@/kibana-highlighted-field@"
    ],
    "transaction.type": [
      "@kibana-highlighted-field@page-load@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1566535925864
  ]
}

And we're using load balancer - Nginx for APM server.

As for RUM agent version, we're about to upgrade it among several projects soon.

Very appreciate,
zhangsh

4

Anyone here can help?

5

Hi @zhangsh,

Sorry about the slow response.
We're investigating this issue. I will update this thread as we progress.

Cheers,
Hamid

6

OK, thanks!

7

Hi @zhangsh,
my colleague @Hamidreza and I had a look at this issue. You have mentioned that you are using a load balancer in front of APM Server. Any chance the load balancer is configured in a way that it retries failing POST events to the APM Server? The APM Server processes all incoming events without checking for duplicates, meaning in case the same event is sent up twice it will be processed and stored twice.
For parsing the IP addresses, the server first looks at different header fields and if all of them are empty, uses the request's remote address. Could you check whether the stored IP addresses reflect the address of where the load balancer is running?

8

Hi @simitt, sorry for the late response.
After consulting with my colleague, we've confirmed that Nginx has no retry configuration on fail requests :dizzy_face:.
Is there any other possibilities?

9

Interesting, so we need to further dig into this.

How often did this happen? The frequency might give an idea about whether this could be a general issue related to the setup or potentially uncover some kind of edge case issue.

The latest Elastic Stack version released is 7.3 and the latest RUM agent version is 4.4. Is it possible for you to upgrade to a newer version and see if this helps?

10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.