I have a terms filter (over multiple fields) and an equivalent boolean query; but the terms query is faster. Is some optimization performed when the terms filter is applied (which thereby allows it to perform better than the equivalent boolean query)?
Yep 
You can read more about it here: https://www.elastic.co/guide/en/elasticsearch/guide/current/filter-caching.html
Generally, filters are executed in a "non-scoring" mode which gives them two main performance advantages. Firstly, they can omit the actual scoring of the document. Scoring a doc is relatively quick (the summation of a bunch of multiplications), but even 1ns for a billion documents is 1 second of computation.
Secondly, non-scoring filters can be cached, meaning subsequent executions of the filter clause can leverage the cache instead of hitting the various data-structures.
Where possible, we encourage people to convert queries => filters for better performance (assuming you don't need the scoring aspect)