Re-add the outside-of-the-SM-Tree handling that I somehow dropped