CRM-20441 Return early if we have removed all Activity IDs as got no access to them