Calculate layers to keep only when necessary