Heat oven to 400°F. Grease a 9x13-inch baking dish.
Boil potatoes in a large pot of salted water until fork-tender, 12–15 minutes. Drain well and let steam off.
Warm 2 tbsp butter and the olive oil in a large skillet over medium heat. Soften the onion, about 3 minutes.
Stir in cabbage and 1/2 tsp salt. Cook, stirring, until wilted and lightly golden at edges, 8–10 minutes. Add garlic for the last minute.
Mash potatoes with remaining 2 tbsp butter and the warm milk until creamy. Season with remaining salt, pepper, and nutmeg.
Fold the cabbage mixture and 1 1/4 cups cheddar into the potatoes. Spread into the prepared dish.
Sprinkle with remaining 3/4 cup cheddar. Bake until bubbling and browned on top, 15–20 minutes.
Rest 5 minutes. Finish with scallions and parsley, then serve hot.