Duplicated transaction with different ip

Kibana version: 6.6.1

Elasticsearch version: 6.6.1

APM Server version: 6.6.1

APM Agent language and version: apm-agent-rum-js@3.0.0

Browser version: Android 9, Chrome/74.0.3729.136

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version?
Fresh install

Is there anything special in your setup?
noop

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

As screenshot shows, there are two almost the same 'page-load' transactions at the same time, except ip field. How did it happen?

Steps to reproduce:
Haven't reproduce it.

Errors in browser console (if relevant): noop

Provide logs and/or server output (if relevant): noop

Hi @zhangsh,

Thanks for reaching out and welcome to the forum.

Would you please send the full document for these transactions? Please redact any personal information that might exist on these documents.

Also, are you using any type of proxy or load balancer for APM server?

As a side note, the current version of the RUM agent is 4.4.2. we recommend to use the latest version since we have made many improvements since version 3.0.0.

Cheers,
Hamid

Firstly, thanks for your quickly reply!

Here are the two transactions' JSON:

{
  "_index": "apm-6.6.1-transaction-2019.08.23",
  "_type": "doc",
  "_id": "WKHRvGwBd9z6GtobQ1Ht",
  "_version": 1,
  "_score": null,
  "_source": {
    "trace": {
      "id": "eec9f724e68aecbee68e8485cd056b75"
    },
    "@timestamp": "2019-08-23T04:52:06.106Z",
    "beat": {
      "hostname": "apm-server10-001",
      "name": "apm-server10-001",
      "version": "6.6.1"
    },
    "host": {
      "name": "apm-server10-001"
    },
    "context": {
      "service": {
        "agent": {
          "name": "js-base",
          "version": "3.0.0"
        },
        "name": "name",
        "language": {
          "name": "javascript"
        }
      },
      "page": {
        "referer": "",
        "url": "url"
      },
      "user": {
        "ip": "121.227.106.100",
        "id": "id",
        "user-agent": "Mozilla/5.0 (Linux; Android 9; MI 9 Build/PKQ1.181121.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.136 Mobile Safari/537.36"
      }
    },
    "processor": {
      "name": "transaction",
      "event": "transaction"
    },
    "transaction": {
      "duration": {
        "us": 164300
      },
      "name": "name",
      "span_count": {
        "started": 11
      },
      "id": "ca9d513efa9589ff",
      "marks": {
        "agent": {
          "firstContentfulPaint": 147.69999999953143,
          "domInteractive": 133,
          "domComplete": 333,
          "timeToFirstByte": 1
        },
        "navigationTiming": {
          "responseEnd": 2,
          "responseStart": 1,
          "domInteractive": 133,
          "domainLookupEnd": 0,
          "domContentLoadedEventStart": 133,
          "domComplete": 333,
          "domainLookupStart": 0,
          "connectEnd": 0,
          "connectStart": 0,
          "loadEventStart": 333,
          "requestStart": 1,
          "fetchStart": 0,
          "domContentLoadedEventEnd": 133,
          "loadEventEnd": 333,
          "domLoading": 4
        }
      },
      "type": "page-load",
      "sampled": true
    },
    "timestamp": {
      "us": 1566535926106518
    }
  },
  "fields": {
    "@timestamp": [
      "2019-08-23T04:52:06.106Z"
    ],
    "error id icon": [
      null
    ]
  },
  "highlight": {
    "transaction.id": [
      "@kibana-highlighted-field@ca9d513efa9589ff@/kibana-highlighted-field@"
    ],
    "context.service.name": [
      "@kibana-highlighted-field@serviceName@/kibana-highlighted-field@"
    ],
    "transaction.type": [
      "@kibana-highlighted-field@page-load@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1566535926106
  ]
}

========================

{
  "_index": "apm-6.6.1-transaction-2019.08.23",
  "_type": "doc",
  "_id": "yBnRvGwBjBx0RsJRQ17g",
  "_version": 1,
  "_score": null,
  "_source": {
    "trace": {
      "id": "eec9f724e68aecbee68e8485cd056b75"
    },
    "@timestamp": "2019-08-23T04:52:05.864Z",
    "beat": {
      "hostname": "apm-server10-001",
      "name": "apm-server10-001",
      "version": "6.6.1"
    },
    "host": {
      "name": "apm-server10-001"
    },
    "context": {
      "service": {
        "agent": {
          "name": "js-base",
          "version": "3.0.0"
        },
        "name": "name",
        "language": {
          "name": "javascript"
        },
      },
      "page": {
        "referer": "",
        "url": "url"
      },
      "user": {
        "ip": "223.104.146.18",
        "id": "id",
        "user-agent": "Mozilla/5.0 (Linux; Android 9; MI 9 Build/PKQ1.181121.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.136 Mobile Safari/537.36"
      }
    },
    "processor": {
      "name": "transaction",
      "event": "transaction"
    },
    "transaction": {
      "duration": {
        "us": 164300
      },
      "name": "name",
      "span_count": {
        "started": 11
      },
      "id": "ca9d513efa9589ff",
      "marks": {
        "agent": {
          "firstContentfulPaint": 147.69999999953143,
          "domInteractive": 133,
          "domComplete": 333,
          "timeToFirstByte": 1
        },
        "navigationTiming": {
          "responseEnd": 2,
          "responseStart": 1,
          "domainLookupEnd": 0,
          "domInteractive": 133,
          "domContentLoadedEventStart": 133,
          "domComplete": 333,
          "domainLookupStart": 0,
          "connectEnd": 0,
          "connectStart": 0,
          "loadEventStart": 333,
          "requestStart": 1,
          "fetchStart": 0,
          "domContentLoadedEventEnd": 133,
          "loadEventEnd": 333,
          "domLoading": 4
        }
      },
      "type": "page-load",
      "sampled": true
    },
    "timestamp": {
      "us": 1566535925864622
    }
  },
  "fields": {
    "@timestamp": [
      "2019-08-23T04:52:05.864Z"
    ],
    "error id icon": [
      null
    ]
  },
  "highlight": {
    "transaction.id": [
      "@kibana-highlighted-field@ca9d513efa9589ff@/kibana-highlighted-field@"
    ],
    "context.service.name": [
      "@kibana-highlighted-field@serviceName@/kibana-highlighted-field@"
    ],
    "transaction.type": [
      "@kibana-highlighted-field@page-load@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1566535925864
  ]
}

And we're using load balancer - Nginx for APM server.

As for RUM agent version, we're about to upgrade it among several projects soon.

Very appreciate,
zhangsh

Anyone here can help?

Hi @zhangsh,

Sorry about the slow response.
We're investigating this issue. I will update this thread as we progress.

Cheers,
Hamid

OK, thanks!

Hi @zhangsh,
my colleague @Hamidreza and I had a look at this issue. You have mentioned that you are using a load balancer in front of APM Server. Any chance the load balancer is configured in a way that it retries failing POST events to the APM Server? The APM Server processes all incoming events without checking for duplicates, meaning in case the same event is sent up twice it will be processed and stored twice.
For parsing the IP addresses, the server first looks at different header fields and if all of them are empty, uses the request's remote address. Could you check whether the stored IP addresses reflect the address of where the load balancer is running?