When removing a pipe from the lb or fq component,
make sure we do not remove another pipe from the active set.