Apm-agent-java TraceParent ratelimit?

Elastic Observability APM
1

We are working on our traceparent header to make sure the agent would not blindly trust the flags in traceparent header and we have noticed that there is an todo on TraceParent.java

Just wondering/curious since this bit was last touched about 2 years ago. Is there any plan for this functionality to be implemented? or if elastic is happy to receive external contribution on this specific one?

2

Hi and welcome to the forum @marsblkm :wave:
Did you already implement something? What would you consider to be the ideal solution for that? What other alternatives can you think of?

3

Hi @Eyal_Koren ,

Nope we haven't implemented anything yet. Initially we were looking at load balancer level and wondering if applying a rate limiter on load balancer(modifying the sampled flag in the header) would work, and then somehow found out there is a todo in the code itself.

For the implementation itself, (sorry if this part does not make sense) I know there are some exisiting libraries(e.g Guava) that provide in memory rate limiting functionalities and maybe can start from there? Although I think that could be a little bit tricky since blindly apply a rate limiting on apm agents might gonna cause missing spans...

The other alternatives that I can think of is pretty much our initial approach right now (rate limiting on load balancer part).

4

I am not sure what is memory rate limiting and how it will be used. If we were to look into rate limiting, my intuition of what would be simplest and most robust would be something numeric, for example - "create up to 50 transactions per second", or anything else in this direction. Not as easy to get right as it sounds, but definitely doable.

However, it will still suffer from what you pointed out.
What about focusing on the trust part? For example - a configurable allow-list of domains that we can trust. If not configured - trust anything. If configured, only trust trace context headers coming alongside a Host header with a trusted domain. This requires more configuration, but it guarantees that you trust anything that comes from your env. Would that solve your issue?

1 Like
5

@Eyal_Koren
What about focusing on the trust part? For example - a configurable allow-list of domains that we can trust. If not configured - trust anything. If configured, only trust trace context headers coming alongside a Host header with a trusted domain. This requires more configuration, but it guarantees that you trust anything that comes from your env. Would that solve your issue?

Yes this was pretty much like the solution we were considering and we decided to park this for a while before we can get a little bit more information from the you guys, seeing if anything can be leveraged from the apm-agent itself.

Thanks for answering my questions!

6

Sure thing!

What I meant is to add such configurable allow list to the agent and rely on the Host header contents.
If you want to contribute that as a PR, here are some pointers:

  • Read our contributing documentation :slight_smile:
  • Add a configuration option to co.elastic.apm.agent.configuration.CoreConfiguration with:
    • a proper name
    • a proper description
    • probably typed ConfigurationOption<List<WildcardMatcher>>
  • Run ConfigurationExporterTest so that documentation gets updated with the new config option automatically
  • Read this config option where we create the child transaction. If nothing configured, continue as usual. Otherwise, this location gives you access to the http request object (carrier) and a way to extract headers from it (traceContextHeaderGetter), which you can use to obtain the value of the Host header. Compare that to the configured list and decide whether to allow or deny (and log a warning, so that you either know you should update your allow list, or be aware that unexpected requests arrive)
  • Add proper test cases

However, this will required some effort and time. The way to get it NOW is through your load balancer config :slight_smile:

1 Like
7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.