The Goggles Do Nothing
Fortune Gear is optional insomuch as you can go to two distinct places, and you have your choice of which. The next update will be the other choice. But, ultimately, you have to go to both places eventually at around this point in the game to proceed. So, one way or another, you are fighting Janus at this point in the plot.