Downtown doesn't have 56,000 workers, and doesn't have 38,000 either. The actual figure is more like 8,000.
Does it actually have 12million SF of office space? Then it must have a 90% vacancy rate.
That figure is also bogus, half the buildings down there have been demolished or turned into single-tenant government offices, and a sizeable chunk of what's left are either gutted or otherwise not in marketable condition, yet the total ft. sq. numbers haven't changed in decades.
There are 25 existing buildings (the 25 largest between Northbank, Southbank, Riverside Ave) that contain between 9 and 10 million SF. It's not unreasonable to state that downtown has 12-15 million rentable SF. This includes office space in 1-5 floor buildings, as well. Yes, downtown is mostly surface parking. But it actually has a sizable built environment (albeit super spread out). And no that does not include City Hall or the City Hall Annex, but it does include privately owned buildings where the City happens to lease all or most of the space.
The type of employment can also make a difference. Tech office space often has a ridiculously low tenant/space ratio. I've heard of several deals that are sub 1:100 (1 employee per rentable SF). Usable is a whole 'nother story. Most of the office deals I have worked on recently have been tech deals where the floorplan of the space in question is designed for 1:110-1:150. There are still law firms where the ratio is north of 1:500, just to point out the difference.
Therefore, it can be concluded that the reason that many people feel or comment that San Francisco can feel even busier than a Chicago (which is much larger) is that in addition to having pretty high building density (73 million RSF in an area 30% the size of Atlanta's CBD, or probably not too dissimilar as Northbank + Southbank), there is lots of tech employment with high employment densities, AND a very low vacancy rate.