1995 Experiment Comments

Comments before the presentation

• The experiment, design, time, training, etc.

  • There was not enough time to absorb the new PBR technique
  • There was a learning effect due to just realize "how to look for things"
  • They seemed to prefer the university site - doing it in the office would be harder, with distractions
  
  

• The documents

  • The generic documents were harder because of coherence and you had to read everything from first to last page. The NASA documents were easier because of familiar application domain and less "scope" of defects (said by designer)
  • The same person also mentioned the generic documents could be improved if they were more concrete and specific
  • About half of the group seemed to find the NASA documents to be the hardest, while the other half found the generic documents to be hardest
  • There were no apparent differences between the NASA A and NASA B documents, neiter in size, nature nor complexity
  • The documents were easier to read the second time (said by tester)
  • The documents seemed a good size for the time given.
  
  

• The technique

  • PBR was assumed to be an advantage for the NASA documents while they did not expect it to lead to better results for the generic documents (said by designer, others seemed to agree)
  • The experiment has had little influence on current practice, but one subject reported to think of the concept of perspectives in the review process without applying scenarios or questions
  • The tester was perceived to be the easisest role to step into
  • One reviewer who said he had difficulty applying the perspective (tester) fell back on his usual approach
  
  

Comments after the presentation

• What may explain the results?

  • An imposed perspective may constrain the resources you feel you can apply because your experience is much broader (for NASA documents). Thus a perspective can feel restrictive
  • NASA documents seem to be written for developers while the generics are less technically detailed and more directed towards users
  • The same person also mentioned that it was difficult to see where the users fit in to the NASA documents
  • When assuming your usual perspective, you do not use the questions, you rely on your experience

• The advantage of PBR

  • The advantage of PBR is that you get more focused and feel more responsibility because you are one of very few persons on a team responsible for a particular perspective
  
  

• Suggestions for improvement

  • The experiment should be split up in more shorter sessions - e.g. one session each day for four days. This would allow for more training and would reduce effects due to boredom, fatigue etc. at the end of the day
  • If the list if questions is too long then people won't use it
  • Other people would be encouraged to participate in a replication. Applying PBR in a real project (case study) would be preferred
  
  

Web Accessibility