# APM index logs default to epoch

**URL:** <https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868>\
**Category:** APM\
**Created:** [September 17, 2018, 5:50pm UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868 "2018-09-17T17:50:10Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![ioannism](https://avatars.discourse-cdn.com/v4/letter/i/b2d939/32.png) [@ioannism](https://discuss.elastic.co/u/ioannism)\
**Post date:** [September 17, 2018, 5:50pm UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/1 "2018-09-17T17:50:10Z")

</div>

I'm having this difficult to replicate issue. Sometimes, seemingly at random, an apm index will be created with the index date set to epoch i.e. apm-6.3.0-transaction-1970.01.01  
I've turned on debugging in the logs but I don't see anything that would point to the root cause. Where else should I be looking? I'm using v0.7.0 of the Java agent.

TIA!

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 17, 2018, 6:41pm UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/2 "2018-09-17T18:41:06Z")

</div>

Hi and thanks for giving the Java agent a try! Could you post the documents inside that index?

---

<div class="post-metadata">

**Author:** ![ioannism](https://avatars.discourse-cdn.com/v4/letter/i/b2d939/32.png) [@ioannism](https://discuss.elastic.co/u/ioannism)\
**Post date:** [September 17, 2018, 7:34pm UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/3 "2018-09-17T19:34:07Z")

</div>

Hey Felix, will do, as soon as I stop wiping the damn thing in anger 🙂

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 17, 2018, 7:36pm UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/4 "2018-09-17T19:36:44Z")

</div>

Haha, sounds like a plan 😉

Some more questions: How often does that happen? Which frameworks and application server are you using? What are your settings? Which apm server version are you using?

---

<div class="post-metadata">

**Author:** ![ioannism](https://avatars.discourse-cdn.com/v4/letter/i/b2d939/32.png) [@ioannism](https://discuss.elastic.co/u/ioannism)\
**Post date:** [September 17, 2018, 7:40pm UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/5 "2018-09-17T19:40:58Z")

</div>

Hard to say. It's a custom spring boot app that I'm tracing with a Spring Hessian handled endpoint, but I've turned off _everything_ via the configuration and I'm tracing only inside a single method in our codebase.  
Having said that, before I decided to add custom traces, I did observe the same issue without using the APM library in the project. I'm using APM 6.3 and my settings are as follows

```
service_name=app-search-engine
server_urls=http://xxxxxxx:8200
application_packages=com.insight
service_version=9faf8110c20d0fe72ad84f36f1cdbcccbe7a936f
capture_body=transactions
application_packages=com.insight
ignore_urls=(?i)/app-search-engine/metrics*,(?i)/app-search-engine/health*
disable_instrumentations=jdbc,servlet-api,servlet-api-async,spring-mvc,http-client,apache-httpclient,spring-resttemplate,incubating

```

EDIT: just noticed I'm duplicating the application\_packages property, oops.

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 17, 2018, 7:43pm UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/6 "2018-09-17T19:43:14Z")

</div>

> [@ioannism](#):
>
> I'm tracing only inside a single method in our codebase.

What do you mean by that? How do you trace this method?

---

<div class="post-metadata">

**Author:** ![ioannism](https://avatars.discourse-cdn.com/v4/letter/i/b2d939/32.png) [@ioannism](https://discuss.elastic.co/u/ioannism)\
**Post date:** [September 17, 2018, 8:40pm UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/7 "2018-09-17T20:40:44Z")

</div>

Using apm-agent-api 🙂 v0.7

---

<div class="post-metadata">

**Author:** ![ioannism](https://avatars.discourse-cdn.com/v4/letter/i/b2d939/32.png) [@ioannism](https://discuss.elastic.co/u/ioannism)\
**Post date:** [September 17, 2018, 8:54pm UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/8 "2018-09-17T20:54:36Z")

</div>

to be precise

```
final Transaction transaction = ElasticApm.startTransaction();
transaction.setType(type);
transaction.setName(name);
// method body goes here
// Spans throughout, example below
Span span = transaction.createSpan();
span.setType(type);
span.setName(name);
// some more code to trace
span.end()
transaction.end()
```

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 18, 2018, 9:33am UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/9 "2018-09-18T09:33:16Z")

</div>

You should end the transactions and spans in a `finally` block as shown in the [documentation](https://www.elastic.co/guide/en/apm/agent/java/current/public-api.html#api-start-transaction). Otherwise, this is looking fine.

For further analysis, I'd have to have a look at the documents inside the `apm-6.3.0-transaction-1970.01.01` index.

---

<div class="post-metadata">

**Author:** ![ioannism](https://avatars.discourse-cdn.com/v4/letter/i/b2d939/32.png) [@ioannism](https://discuss.elastic.co/u/ioannism)\
**Post date:** [September 18, 2018, 10:06am UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/10 "2018-09-18T10:06:28Z")

</div>

Thanks, I already do, but elided for brevity. As soon as one of these nasties shows up again I'll post it here 🙂

---

<div class="post-metadata">

**Author:** ![ioannism](https://avatars.discourse-cdn.com/v4/letter/i/b2d939/32.png) [@ioannism](https://discuss.elastic.co/u/ioannism)\
**Post date:** [September 18, 2018, 11:43am UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/11 "2018-09-18T11:43:19Z")

</div>

Whilst I'm setting my traps for this bad boy, I had a general nature tracing question I was hoping you could shed some light on.

Given the following calling chain

```
A -> B -> D 
     C -> E

```

meaning: Class A within the context of a single method, calls methods from class B and C, and they in turn call methods from D and E respectively.

I want to create a single transaction that spans all the classes involved in A's method call. What's the best way to do this? I keep thinking that I have to pass the created **Transaction** object to B,C,D,E so that I can create **Span** instances from it, but that doesn't feel right at all.

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 19, 2018, 6:07am UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/12 "2018-09-19T06:07:01Z")

</div>

> [@ioannism](#):
>
> I keep thinking that I have to pass the created **Transaction** object to B,C,D,E so that I can create **Span** instances from it, but that doesn't feel right at all.

That would currently the way to go. At its current state, the public API does not yet offer a method which makes a span "active", so that you can retrieve it with `ElasticApm.currentSpan()`. The reason is that there is a lot which can go wrong in terms of leaking the spans in a `ThreadLocal`. So at the moment, you really have to pass along the Spans and Transactions. Is that a bummer for you?

---

<div class="post-metadata">

**Author:** ![ioannism](https://avatars.discourse-cdn.com/v4/letter/i/b2d939/32.png) [@ioannism](https://discuss.elastic.co/u/ioannism)\
**Post date:** [September 19, 2018, 6:24am UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/13 "2018-09-19T06:24:59Z")

</div>

I'd say so. For all given paths that my application will start a root Transaction, I have to modify all methods from participating classes to accept a `Span` as an argument. This forces an additional concern on the developer.  
So, given a class `Foo` that begins a `Transaction` and has a dependency on class `Bar` , that means for class `Baz` that uses `Bar` (and every other class) I have to "force" instrumentation upon them, even if `Baz` doesn't care about tracing. At best, this will create needless noise, at worst it will confuse the viewer, with rootless `Span`s appearing in the APM dashboard. This also forces me to make decisions that will be hard to change (because, recompile/release 😉 )

At this point, I'm considering writing a solution with AOP/Pointcuts and annotations, but it's somewhat hand-wavy in my head and I've yet to examine the finer points. I'm currently trying something out abstracting the APM API away in a Spring component with `ThreadLocal` but I've had mixed success..

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 19, 2018, 6:55am UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/14 "2018-09-19T06:55:04Z")

</div>

I've added two issues to provide support for that:

> <https://github.com/elastic/apm-agent-java/issues/211>

  

> <https://github.com/elastic/apm-agent-java/issues/212>

If you are interested, you can also help implementing this. We are using Byte Buddy, a byte code manipulation framework. It works in a similar way than Spring AOP but has the advantage that it's not tied to Spring.

If you are interested in collaborating (either by coding or by providing feedback), you can follow up in any of those issues.

---

<div class="post-metadata">

**Author:** ![ioannism](https://avatars.discourse-cdn.com/v4/letter/i/b2d939/32.png) [@ioannism](https://discuss.elastic.co/u/ioannism)\
**Post date:** [September 19, 2018, 7:03am UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/15 "2018-09-19T07:03:14Z")

</div>

Happy to contribute whichever way I can, thank you very much for following this up!

---

<div class="post-metadata">

**Author:** ![ioannism](https://avatars.discourse-cdn.com/v4/letter/i/b2d939/32.png) [@ioannism](https://discuss.elastic.co/u/ioannism)\
**Post date:** [September 19, 2018, 1:43pm UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/16 "2018-09-19T13:43:46Z")

</div>

Finally caught one

```auto
{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 3,
    "max_score": 1,
    "hits": [
      {
        "_index": "apm-6.3.0-transaction-1970.01.01",
        "_type": "doc",
        "_id": "bIPl8WUBobq0Xh-lDBVi",
        "_score": 1,
        "_source": {
          "@timestamp": "1970-01-01T00:00:00.000Z",
          "processor": {
            "name": "transaction",
            "event": "transaction"
          },
          "transaction": {
            "id": "00000000-0000-0000-0000-000000000000",
            "name": "DispatcherServlet#doPost",
            "duration": {
              "us": 2806324779308
            },
            "type": "request",
            "sampled": false,
            "span_count": {
              "dropped": {
                "total": 0
              }
            }
          },
          "context": {
            "service": {
              "name": "app-search-engine",
              "agent": {
                "version": "0.7.0",
                "name": "java"
              },
              "version": "9faf8110c20d0fe72ad84f36f1cdbcccbe7a936f",
              "language": {
                "name": "Java",
                "version": "1.8.0_162"
              },
              "runtime": {
                "name": "Java",
                "version": "1.8.0_162"
              }
            },
            "process": {
              "title": "/usr/java/jdk1.8.0_162/jre/bin/java",
              "pid": 15372
            },
            "system": {
              "hostname": "Linux",
              "architecture": "amd64",
              "platform": "the.agent.server.com",
              "ip": "xxx.xx.x.xxx"
            }
          },
          "host": {
            "name": "the.apm.server.com"
          },
          "beat": {
            "version": "6.3.0",
            "name": "the.apm.server.com",
            "hostname": "the.apm.server.com"
          }
        }
      },
      {
        "_index": "apm-6.3.0-transaction-1970.01.01",
        "_type": "doc",
        "_id": "2oPn8WUBobq0Xh-ls_rM",
        "_score": 1,
        "_source": {
          "@timestamp": "1970-01-01T00:00:00.000Z",
          "processor": {
            "name": "transaction",
            "event": "transaction"
          },
          "transaction": {
            "name": "HessianServiceExporter",
            "duration": {
              "us": 2806497785353
            },
            "type": "request",
            "sampled": false,
            "span_count": {
              "dropped": {
                "total": 0
              }
            },
            "id": "00000000-0000-0000-0000-000000000000"
          },
          "context": {
            "service": {
              "language": {
                "name": "Java",
                "version": "1.8.0_162"
              },
              "runtime": {
                "name": "Java",
                "version": "1.8.0_162"
              },
              "name": "app-search-engine",
              "agent": {
                "name": "java",
                "version": "0.7.0"
              },
              "version": "9faf8110c20d0fe72ad84f36f1cdbcccbe7a936f"
            },
            "process": {
              "pid": 15372,
              "title": "/usr/java/jdk1.8.0_162/jre/bin/java"
            },
            "system": {
              "hostname": "Linux",
              "architecture": "amd64",
              "platform": "the.agent.server.com",
              "ip": "xxx.xx.x.xxx"
            }
          },
          "beat": {
            "name": "the.apm.server.com",
            "hostname": "the.apm.server.com",
            "version": "6.3.0"
          },
          "host": {
            "name": "the.apm.server.com"
          }
        }
      },
      {
        "_index": "apm-6.3.0-transaction-1970.01.01",
        "_type": "doc",
        "_id": "cYTn8WUBobq0Xh-l0wUL",
        "_score": 1,
        "_source": {
          "@timestamp": "1970-01-01T00:00:00.000Z",
          "transaction": {
            "duration": {
              "us": 2806505777235
            },
            "type": "request",
            "sampled": false,
            "span_count": {
              "dropped": {
                "total": 0
              }
            },
            "id": "00000000-0000-0000-0000-000000000000",
            "name": "DispatcherServlet#doPost"
          },
          "context": {
            "service": {
              "name": "app-search-engine",
              "agent": {
                "name": "java",
                "version": "0.7.0"
              },
              "version": "9faf8110c20d0fe72ad84f36f1cdbcccbe7a936f",
              "language": {
                "name": "Java",
                "version": "1.8.0_162"
              },
              "runtime": {
                "name": "Java",
                "version": "1.8.0_162"
              }
            },
            "process": {
              "pid": 15372,
              "title": "/usr/java/jdk1.8.0_162/jre/bin/java"
            },
            "system": {
              "ip": "xxx.xx.x.xxx",
              "hostname": "Linux",
              "architecture": "amd64",
              "platform": "the.agent.server.com"
            }
          },
          "beat": {
            "hostname": "the.apm.server.com",
            "version": "6.3.0",
            "name": "the.apm.server.com"
          },
          "host": {
            "name": "the.apm.server.com"
          },
          "processor": {
            "name": "transaction",
            "event": "transaction"
          }
        }
      }
    ]
  }
}

```

It occurred as soon as I reverted `enable_instrumentations` to its default value.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 10, 2018, 9:44am UTC](https://discuss.elastic.co/t/apm-index-logs-default-to-epoch/148868/17 "2018-10-10T09:44:06Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.
