Self-evaluation - how do you feel your site is doing?

Question

Periodically, the Community Team will evaluate the quality of a beta site. In addition to examining general site traffic stats and Meta activity, we check the content quality by selecting 10 random open-and-answered questions from the past 30-40 days to see how their answers compare to other sites on the Internet. This all goes into our evaluation of when a site is ready to graduate.

So if it wasn't clear, yes, we are looking at when this site will be graduating. It's one of those things on the line.

As discussed in Podcast #30, we would like to explore options to get the communities directly involved in these evaluations. After all, you are the subject experts, and your evaluation would be more meaningful. We've posted a general version of this on Meta Stack Overflow, but I would also like to see what you, the User Experience community can do on your own personal level.

How can the community pull off a self-evaluation that would be meaningful? What would be the process? How would it be organized? Can we use meta and/or chat? The process and the results should be transparent so the community can learn from the experience.

Below is an excerpt of our most recent evaluation of your site. Take a gander at how we looked at your content quality to get an idea of what our own approach is.

• Symbol for "Swipe Left"
  Better. Our answer is a variation on a standard - it's going to be either a point or a pointing hand. Searching for "swipe" or "drag" or "fling" is actually very poor, it's searching for "touchscreen motion" or similar that yields icons and the like that you'd expect. So we're actually very advantageous on that respect.

• Intuitive vs. Efficient: Which is more important?
  Worse. We're a very condensed and consumable version of what information is out there. This is a hot topic in UX and so it's no surprise that there are presentations and even full books on the subject which you can find with a decent search. I'd go for Par if the answer had a lot more detail on the different approaches, as most results cover the area of our answer in just their opening paragraph.

• Usability - Testing Windows applications under different settings
  Worse. This may be up to interpretation based on how you interpret the author's degree of intent for usability and accessibility. While the accepted answer provides a couple of nice points, it only provides a couple sufficient for a particular angle of accessibility. Another answer provides a link to more resources, which can also be found with cursory searches.

• Correct cursor for "content cannot be interacted with"
  Par. We cover the good strategies succinctly yet directly with both the accepted answer and its follower. The advice itself is not too easy to find to begin with, but at the same time, neither is our question. The lack of ease of discovery is kinda hurting it.

• Why is there a header and logo at the top in a web application?
  Better. Most places and searches will tell you that this is a common practice. More in-depth places will go into the details of why, but you don't find those places as easily when searching this specific aspect - it is usually a part of a greater treatise on element placement. So we get a nice boost by addressing the "why" while also being discoverable on the lone search.

• Is “<Object name>: <quantity>” notation acceptable in most popular localization?
  Par. This is a hard thing to search for. It's hard to even find this question. This is yet again one of the things that is better covered in in-depth resources but not discoverable. Our answers do provide both examples and strategies. However, because it is also not very discoverable, it cannot be called "better" that much. Not even sure how that can be corrected.

• Is UX the same as Usability?
  Par. This is a common topic of discussion that is easily found. We have a fair amount of varied analyses in our answers on the subject, which also go and reference other resources for private study. External resources provide the gamut of better and worse analyses than what we offer, variant on how you search for the subject. In the end, we're one of several good answers on the subject. Sometimes, being par is a good thing.

• How to design a form that mostly is read-only, but sometimes editable?
  Better. Most search results on the subject will tell you how to save an editable form in a read-only format, or strategies of two separate forms. There are few resources that cover the union of these as desired in this question, and though the proposed strategies are identical to ours, the discoverability of ours is its boon.

• How can we communicate that UX work can't be done by just anyone?
  Par. Arguments can be said about whether there is danger to hosting this kind of loaded question, but that is irrelevant to this analysis. We have a usable if brief analogy of the subject matter, but quite frankly you can get results left and right in the camps of "Anyone can do UX" and "Not everyone can do UX". Kudos to the question for being able to show up on either side, but due to the subject matter it cannot be honestly stated that the stance given is "better" or "worse".

• What research is there suggesting modal dialogs are disruptive?
  ...what, why is Stack Overflow showing up when I start my search query for this?
  Worse. By nature, our existing answer is not actually providing an answer to the given question. It's just confirming what's already said in the question (which you can also find in rudimentary searches anyway) and saying that the user has no answer. A failure is terrible even if the competition isn't faring any better. [Update: Ben has since written a better answer. The original answer was deleted. See the comments below.]

End results: 3 Better, 4 Par, 3 Worse.

When we have strong answers, we have strong answers. There is a lot of good work in various answers, both those covered in this analysis as well as brief surfs of the site. Sometimes it is hurt by low discoverability, though, but that isn't a direct fault of the site (but it's something users could look to improving when they get a chance). However, there are some places where there are just... weak answers that are not befitting the expertise and quality demonstrated in other answers. Incomplete, very brief or curt, and otherwise just being "sufficient" without being excellent.

We can come up with the sample questions. How can we turn those questions and your efforts into a useful and authoritative evaluation of the site? How would you approach the task of a quality check?

Oh, and feel free to comment on any aspects of my review, if you'd like.

Answer 1

Your analysis is excellent. I'm not sure that someone within this community who knows more about the subject matter would be able to do a better job.

• You're an expert at evaluating questions and answers (and know enough about UX to understand what we're talking about).
• You're probably seeing the questions and answers for the first time, together, with votes, just as a visitor from Google would.
• You have an objective, outsider's point of view.

As suggested in the podcast, it's worth considering hiring a UX expert who isn't involved in the community to do evaluations. Another alternative would be to find a college student who is training to get into the UX field to do evaluations. It would be interesting to compare and contrast the results from the three different parties.

For now, though, I think your analysis is good and there's no reason to overcomplicate things. I think you should continue doing exactly what you did: conduct the analysis internally and post the results on meta. This feedback is invaluable. It gives us a clear, measurable goal: to find questions that don't have good answers, shore them up, and get a better score next time.

Answer 2

I think there are two points that I would make to this - as well as saying thank you and well done Grace Note for all of this work.

Firstly, UX does have a more significant level of subjectivity than some other ( more technical ) aspects. More explicitly, the answers for a specific situation may not always be applicable on a wider basis, because the set of UX criterion applied and weighted will be different per question. It means, I think, that there will always be questions where multiple opposing answers are all valid, which doesn't mean all correct. I find the site helpful in understanding some of the criteria I need to apply to a problem ,rather than necessarily expecting a clear definitive solution.

And a couple of questions I have posted on SO recently have been specific problems, with one defined answer. My UX issues with the same project have not been so easy to find solutions to.

Secondly, your point about citing sources is a good one. I think it would make a real difference if people could cite more sources. However, I am aware that I don't often do this, because my sources are often wide and varied - my search for answers to questions raised during my reading takes me into a whole lot of areas of psychology, and I tend to distill what I have read into a practical solution, so I may not even remember all of the sources, because I use them and progress from them.

I wonder if those who are better than me at remembering sources could make some wiki contributions to give us a core set of UX references, that would provide a good starting point. I might them be able to refer to more obscure material beyond that. While this is my problem, some of those I have worked with in UX also seem to have a more applied approach. Maybe it comes with the territory?