It's never really bothered me, since I figure I'll get back to whatever it was eventually. I've beaten the final boss and am now slowly working my way through getting the rest of the treasures and the music coins (the latter of which I've never done before), and not remembering what most recently changed has not affected my playthrough at all, because yeah, I don't keep track of all of it. I just go where the mysterious voice tells me to if I can't remember where I wanted to go next.