For soem reason the Bitterly Cold effect icon still shows up when you are in the milodan ritual cave, even though it doesnt actually take hold of you or anything.
Yes, while the status is active, it only ever does damage when outdoors and in other frozen terrain (damage varies depending on overall cold resistance). When in a cave, it just does 0 damage over time. For clarity, I made the status effect temporarily hide itself when in a cave or indoors.