Actually, it means that you live in the middle of nowhere - in the boonies - out in the boondocks - in an extremely rural or out-of-the-way area. Destination has nothing to do with it.
If the nearest place to buy gas for your car is an hour's drive away, you live in bumfuckegypt!
Fogey/fogy /fougi/ sl. (early 18C+, orig. Scot) old-fashioned, stuck-in-the mud.
Person with old fashioned ideas which he is unwilling to change: Come to the disco and stop being such an old fogey!
You think me an old fogeyand an old tory, his thoughtful voice said. I saw three generations since O’Connel’s time. I remember the famine. Do you know that the orange lodges agitated for repeal of the union twenty years before O’Connel did or before the prelates of your communion denounced him as a demagogue? You fenians forget some things. (James Joyce, Ulysses. PenguinBooks,1992. p. 38)