For anyone following, I did have to add my CustomTraceListener to the specialSources\allEvents\listeners:
<specialSources>
<allEvents switchValue="Information" name="All Events">
<listeners>
<add name="FileLogTraceListener" />
<add name="CustomTraceListener" />
</listeners>
</allEvents>
...