I'm not saying that it doesn't "look" very appealing, but I make the walk from the core to the mexican restaurant quite often. It takes me right through the middle of the group that congregates on state / main, but really they haven't asked for anything in the year and a half I've been doing this. After a few trips, we just nodded, said good morning and went on our way, no harm no foul.
And yes, I'm the same person who posted that I don't give shit to panhandlers, but these guys are hangin' out where they can and to date they haven't been any problems.