Out here in California there is a shop called the Wheelsmith that should have what you want. Click on the "smoothie" link on their home page. http://www.thewheelsmith.net/index.html
Call bob and Chris sage at wheelsmith. They recently relocated to corona,ca. And manufacture what you are looking for.
I'm using (fairly) modern Mopar 16" spare tire wheels, which are 4-1/2." Check out my avatar. You can find them in the salvage yards for cheap. None of mine cost over $15 (each) & none of them appear to have been used. There are two versions....you want the ones whose centers are slightly inset. Also, shoebox Fords had an optional 16" (4-1/2" width) wheel, which you see for sale here from time to time.
You would have to be quick to find them around this area as most of the local wrecking yards yank them out, bust the tire off and have the rim in the s**** bin before the car ever hits the yard. Great idea if you get the right tires to fit them.