Stack Overflow is how most Stack Exchange users seem to find any of the other sites at this point, so yeah I'd say it's safe to assume a good amount of UX.SE people are from or have used SO/SU/SF.
The thing is Stack Overflow was hugely popular and for the most part all the sites were born of users of SO. Stack Exchange is trying to grow to not just be a network of programmers that have other interests, but building up that sort of general brand is hard.
I wouldn't say we're all programmers rather than UX designers though; many designers that work with CSS or HTML end up on Stack Overflow even if they're not programmers by job title.
Actually, I found some extremely strong evidence from our Area51 commitment phase:
committers active in
83.0% Stack Overflow
80.4% User Experience
So, out of people who * committed* to the site, slightly more participated in SO than participated here...so that's our base audience. We've grown a LOT since then but a fair number of our active users came form SO.