Are questions on testing methodology on-topic?

Question

Are questions about testing technique (study methods, testing pitfalls, costings and the like) technically on-topic? Testing is obviously a part of UX practice, and it's difficult to imagine where else such questions might go in the SE ecosystem, but they aren't actually usability or product experience queries - so do they really belong?

For instance, which of the following five examples would fall within the scope of UX:SE?

1. "How many users should I test with?"
2. "How much should I pay users to test?"
3. "What's the value of exploratory user testing as opposed to task-focused studies?"
4. "Is [Online Testing Service] worth the money?"
5. "How do I spot post-hoc justifications in user tests?"

Answer 1

Usability testing is certainly on topic (though our tagging is a bit of a mess).

User Experience - Stack Exchange is for User Experience Designers, Information Architects, and Human Computer Interaction researchers.

If it's a problem one could reasonably encounter as a UXD, HCI researcher (!), IA etc, it's pretty much on topic here. That definitely includes testing. What it excludes implimentation problems like programming/scripting (try Stack Overflow), general workplace issues try The Workplace or purely graphic design problems like using Photoshop (try Graphic Design).

Testing is a big part of design, and methodology is a big part of doing testing right. If it's important to UX design specifically (or extremely related fields of design), we should (and generally do) cover it. I see testing as something both important and specific to design; unlike coding, which is off topic because you should ask programmers, testing is a situation of design where you want to ask designers and researchers. Topicality is really as much about who you want to ask as it is about where.

However with your example questions (granted, they're just examples without clarifying body text) I note some possible problems with constructiveness; particularly #4 is more opiniony and hard to support with facts/practical experience. Say one person had a great experience with Usaura, one person had a terrible one. Both post answers, but do either really help?

As long as these questions are about solving problems I don't see a problem with them. I do see a problem with "should I use X" or issues comparing services rather than techniques. But that's not an intrinsic problem with testing questions; we also get "Is X software okay for wireframing" questions which have similar problems.