Should we run our own UX studies?

Question

While browsing through the questions in UX.SE, I noticed that that a lot of answers to questions don't have links to actual tests/studies. According to this answer that's OK, and we shouldn't do anything to stop it.

However, there was one answer to that question that said something about a lack of good studies to use as reference for these questions. I'm not sure how true that is, but it got me thinking. It would be interesting if the UX.SE community got together and ran a few UX studies and tests of our own, preferably to provide real data to answer a popular question on UX.SE.

UPDATE: Here is a more detailed proposal (feel free to comment):

• Every week (or every two weeks, or every month, depending on how many people are interested), a question on the UX meta would be created asking people to submit proposals for possible UX studies that we could run. The proposals would detail what data the studies would collect, how that data would be collected, and why it would be useful. The person who submitted the proposal would be responsible for managing the entire study, though he or she could get other people to help.
• The proposal with the most up votes will be selected to be used as our study of the week (or whatever time interval we decide on).
• Once the study has been completed, we will post the results on a blog (see this post)

An example proposal of a possible UX study:

Please Note: This is just a draft of an example proposal.

Are links with images more likely to be clicked on?

Based off of this question (please read it so you will understand the proposal): Are links with images more likely to be clicked on?

• This test would consist of three webpages: one with a link with a small thumbnail (as described in the question), one with the link described in the above, but with a large thumbnail; and one with the same link described above, only without the thumbnail.
• There would be some javascript on every page that would count what portion of the visitors clicked the link, and if they did, how long it took them to do so.
• The data would show whether having a thumbnail improves click through rates, and whether the size of the thumbnail matters.

This is the more old proposal: (only it's not much of a proposal)

If people are interested, then what we need to do is:

• We need to know how we are going to run these studies. Ideally, then should let anyone do them from a computer at home, so that everybody can participate (unfortunately, that means no fancy equipment).
• We are going to need to decide who is going to make, or program, the studies. Perhaps alternate every week?
• Decide if we are going to do studies to answer popular questions (on UX.SE), or do studies to find out random statistics.
  • If we are going to do studies to answer popular questions, we need to decide which questions we are going to answer.
• We are going to need a chatroom or some other sort of hq (I'll create it once I know people are interested).
• I was thinking that we would publish the studies as answered to popular questions, but if we decide to just do studies about random but interesting UX related things, then that wont work. We could use a blog (see this answer on the stack overflow meta).
  • If we are going to studies to answer popular questions (on UX.SE), then a blog might still be a cool "supplementary" publishing platform.

Let me know what you think :) Questions and Comments are welcome.

Answer 1

I think it's an interesting idea and you should press on. Don't ask for permission. Don't form a committee. Just go ahead and put together a study and let us know where you need help.

Answer 2

A UX study is possibly something Stack Exchange, Inc could underwrite.

We do have some nascent community grant programs underway, so feel free to hammer this into a detailed proposal!

Answer 3

I'm kind of wary of this as I feel like it raises more questions than it answers.

What exactly would we study? What would be the goal? Just providing backup to certain answers to questions? Which questions? How would we pick which to pursue? Would it be worth it, given that the most popular of our questions so far only have 3k views? Wouldn't it be better to invest resources in increasing awareness of StackUX before we start doing research (or would doing the research be a way of achieving that)?

How would we do the research in practice? Who would do it and how would we decide? How would we ensure that it's reported accurately and objectively?

I don't mean to be a party pooper but right now it feels like this would be hard to get off the ground. Perhaps we need a more concrete plan? I agree that we should gauge interest before making a decision either way, though.

Answer 4

I'm 1000% in for bringing forward more "tangible" studies on UX. You are right when you say right now most answers are pure top-of-the-head opinion, which in some cases can be based on experience or on real knowledge with no inmediatelly available links to sources, which is acceptable, but I've got the feeling this is the exception.

Nielsen Norman Group have some good studies, but they go at $300+ a pop. We've got some good UX/UI Guidelines (Yahoo, Apple, Android, Microsoft), but following them is an exercise of faith, because they don't publish sources to what they assert.

And then we can do a search on the uncountable analytics, marketing, design, etc. firms that publish their investigations.

Every extra "serious" investigations on UX will always be good and I will participate.

I am sure most of us have sites/apps that could be used in "experiments".

That said, I feel it's a bit naive to have it organized as a community, this things tend to work when there's an interest behind, individually or as a company, and interest normally translates in money. Sad but true. Thus many questions rise as other answers state.

Nevertheless:

A proposal

1. Have some kind of election to choose which UX:SE questions should be studied, like the Community Promotion Ads posts here in meta are elected.

2. A very concise type of study should be set, like Google Analytics' A/B Testings (since most of us use GA) and which sites/apps qualify for testing.

3. After picking some winners all interested users should run the tests in their own and come back with the results.

4. Results should be peer-review for quality. Then published in some neutral site. Since it is in UX.SE's interest, having a special section for these studies would be a good thing, it would also give legitimacy and notoriety.

Some bonus Amazon vouchers would also go a long way to reclute volunteers ;)