Using multiple dates for aggregation

[Ron_OG (Ron Ingraham)]

I'm trying to create a bar or line chart to compare a given month over several years, ie. I want to display all of the doc counts for the month of Jan. from 2005-2017. Is this possible to do?

Thanks ahead of time.

[timroes]

Hi Ron,

sorry for the late reply.

You have basically two options to do this. To achieve what you want, you need a field in your documents, with the month name (same would apply if you try to aggregate weekdays together).

You can create a scripted field, that extracts the month name from the actual date field, with the following painless script:

Calendar cal = Calendar.getInstance();
cal.setTimeInMillis(doc['@timestamp'].date.millis);
cal.getDisplayName(Calendar.MONTH, Calendar.LONG, Locale.getDefault());

(Replace @timestamp with your date field!)

You can now do a terms aggregation on that field, which would aggregate all documents in January together, all in February, and so on.

Since scripted fields can be slow, I would recommend adding the month name before indexing to your document.
If you know that you have to do visualizations on it, that is usually the best and fastest approach.

Cheers,
Tim

[Ron_OG (Ron Ingraham)]

Thanks Tim. If I'm understanding you correctly, the first option would actually take re-indexing the existing indices to include a new mapped field called 'month'. Given the number of docs we have per year spanning decades, the idea of reindexing all of these is not appealing. But I do understand what you are saying about the second option as well. Given that this Visualization will only be viewed once a year, I think the second option might work better. Assuming, of course, that it doesn't take multiple days to create the visualization for a single month!

Thanks again.

-Ron

[timroes]

Hi Ron,

I just think you need to try it out, how well the scripted field performs for you.
If it won't cause any problems, just go with it. It was more of the general warning, that these might be slower than actual indexing the "needed fields", and if you find your chart to slow, this is something you might want to consider (but that really depends on your data and clustersize, so best just try it out).

Cheers,
Tim