The (partial) solution

Traffic shaping under Linux is done in the kernel, but prioritization of packages is done in user space. There is much talk of a suite called iproute2, but its docs are meager and its links are dead, and on Debian, all that can be installed is the package iproute.

The documentation for traffic shaping is the LARTC HOWTO. It has a short section on limiting a single host or netmask, and this could be used for capping multicast (which could be caught in a netmask: the ranges are here). However, this example uses the cbq queueing discipline, and from the rest of the docs we gather that an htb discipline might be simpler and exhibit less complex behaviour. (It is well documented in a separate HTB User Guide by its author.)

More useful info in the tc manpage, the Packet-Shaping-HOWTO calls traffic shaping packet shaping, but it has some nice comments on the error messages of tc.

Before trying our hand at multicast, we first try to limit bandwidth for a single IP address, thusly: