Nest - Nested Aggregation - Count max of 1 per parent


I have an index that has some nested documents.

I am doing some aggregations on the nested docs to determine the number of docs that match a query per day. My code looks like this:

var result = client.Search<EntityCases>(s => s
                .Query(q => q
                      .Nested(n => n
                    .Query(q2 => q2
                    .Terms(t => t
                        .Field(p => p.Notes.First().AgentName)
                        .Terms(from p in agents select p.userid)

                .Aggregations(z => z
                    .Nested("NotesAgg", n => n
                        .Aggregations(a => a
                            .DateHistogram("Date", h => h
                                .Field(p => p.Notes.First().LastModDate)
                                .Aggregations(childAggs => childAggs
                                    .Terms("User", st => st
                                        .Field(p => p.Notes.First().AgentName))

This successfully brings back the count of notes (per day) for each user. My question is; how can i limit it so that it only counts one note, per user for each parent document. So if a user adds multiple notes to a parent over different dates to only count it as one. We are trying to determine how many parents have been touched per user and at the moment the query tells me how many times each user touched the parent (per day).

Sorry if this doesnt make sense.