Whisk cocoa, brown sugar, pumpkin pie spice, and salt with 1/2 cup milk in a saucepan until smooth.
Pour in the remaining milk and warm over medium heat, whisking until steaming.
Whisk in pumpkin puree and simmer gently for 3–4 minutes to thicken and meld flavors.
Remove from heat and stir in vanilla.
Divide into mugs and top with whipped cream and a sprinkle of cinnamon, if desired.