Here's an article outlining the argument of responsive web design being bad for performance (and debunking it).
I thought this was put to bed a year or two ago. I wasn't aware there was still anyone on the "RWD performs poorly" train. Obviously anything can perform poorly, but there's nothing inherent about responsive web design that makes it perform badly.
More to the point, have you ever visited a well-designed, mobile-specific site with good performance? They all seem dog-slow to me...
I guess that's what I get for largely listening to sane and rational people for advice on design.