Rain, i think the issue of scale is in regards to height of the surrounding buildings.... three storys and rooftop dining(on a tiny tiny lot) would look pretty big next to Regions, Mossfire, OBrothers, the Carter Pharmacy building, the condos across the street, etc.
Yes, it wouldnt be higher than 1661, but it would stick out like a sore thumb over the existing buildings adjacent to it. Especially with a design that makes the eye naturally look skyward when viewing the building.
The parking would be a completely seperate issue.