Heat the oven to 340°F (170°C). Line a 9×13-inch metal baking pan with a parchment sling and lightly grease the sides.
In a small saucepan over low heat, warm the Guinness with the molasses just until steaming (do not boil). Whisk in the cocoa powder and instant coffee until completely smooth; set aside to cool until warm to the touch.
In a medium bowl, whisk together the flour, baking powder, baking soda, and fine sea salt.
In a large mixing bowl, beat the sugar and eggs on medium-high speed for 2–3 minutes until lighter in color and slightly thick. With the mixer running, slowly stream in the oil and vanilla until emulsified, then mix in the sour cream until smooth.
Add half of the dry ingredients to the bowl and mix on low just to combine. Pour in all of the warm cocoa–stout mixture and mix briefly. Add the remaining dry ingredients and mix until only a few streaks of flour remain; finish by folding with a spatula. Let the batter rest for 10 minutes to hydrate.
Scrape the batter into the prepared pan, smooth the top, and tap the pan firmly on the counter 2–3 times to release air bubbles. Bake for 32–36 minutes, or until a toothpick inserted near the center comes out with a few moist crumbs.
Cool in the pan for 15 minutes, then use the parchment to lift the cake onto a rack to cool completely before frosting.
Make the buttercream: In a stand mixer, whip the softened butter on medium-high for 3 minutes until pale and airy. Add half the powdered sugar along with the salt and vanilla; mix on low to combine, then medium for 1 minute. Beat in the remaining sugar, then slowly add the Baileys until the frosting is smooth and fluffy, 1–2 minutes more.
Spread the Baileys buttercream over the cooled cake in swoops and finish with a light sprinkle of flaky sea salt. Chill 15 minutes to set before slicing.