This is Mike from Balsamiq.
I appreciate, and I guess should have expected the feedback. We admit to not working on the icon when we did this integration, but I completely agree with all the feedback. It's obvious that a smiley icon, while a legacy of the initial Mockups App UI isn't going to work in a rich text editor to suggest anything other than an emoticon widget.
So on to action. The main goal should be to design an icon that:
Is imediately recognizable and indicates that it's tied to the "add wireframe / mockup" behavior. Better scent and recognition, less mystery meat.
Fits into the toolbar dimensions. The toolbar has containers for icons of 20px, but actual icon height is max 12px.
This morning I started some sketches to see what iconography might work, starting with the initial concept using the sketchy Mockup image to more generic wireframe images.
We did a few versions showing these ideas in the toolbar.
At the box in the top is the progression of ideas as we iterated on the icon. We're thinking that for editor integrations like this, an icon like the two in the right are appropriate for signifying a wireframe.
We'd be happy to go with the one we depict in the toolbar image above, right next to the image icon. Please chime in and let us know what you think, and thanks for the honest criticism.
Edit: For those who might not read through all of the comments, here are the current proposals: