Why? He asked for Sturgeon Bay. The places I'm thinking of are not online. On the west side of town (across the Bay, if you're coming from Fish Creek), go across the old covered bridge on Michigan Ave (not the one a few blocks away) and continue up Michigan for maybe 3-4 blocks. On your left, on a corner, is the place where I found Creme Soda and a bunch of private press LPs & cool local WI 50s/60s records. The last couple times I was there, there was very little vinyl left, and it was out in their warehouse. It may have to be dropped from the list, but it used to serve me very well. There used to be another antique shop next door with cool old record players and shit, but I think that's now a gift shop.
South of Sturgeon Bay, near Little Sturgeon out in the sticks, there are a couple shops that get odd stuff from local farmers. I've found crazy-good 45s for cheap at this place:
Ken's Granary
2931 County Road Mm
Sturgeon Bay, WI 54235
(920) 743-2173
...as well as some real oddball non-musical items. Haven't been there for a year or so, and stock changes over a lot, so who knows? They also have weird hours sometimes, so maybe call to confirm. Sturgeon Bay actually had a lot of national bands play back in the day, as well as bands further north who played to the vacationing hordes, so it wouldn't hurt to hit up all the thrift/antique spots I guess. I'd personally go driving around/hiking and get on a boat if possible, maybe hit Al Johnson's Swedish Restaurant with the goats on the roof and imported Swedish waitresses, or eat the DC cherry stuffed French Toast at White Gull Inn in Fish Creek (voted best breakfast in America), you're probably not going to find bonzers on this trip.