Questions about “Foveated 3D Graphics (Microsoft)” User Study

Problem1: They did test for only one scene.
1. The first problem is that foveation level is highly depentent on scene. They may get totally different parameters if they change to another scene. Of course, this is the problem of all the user studies. Till now, only NVIDIA mentiond the multliple factors affecting vision. However, they don’t have good ways to deal with this.
2. The second problem is about data analysis. They avoid the problem of one parameter ->multiple result by testing only one scene.
Problem2: I don’t believe that their result is monotone.
1. They just said:
  1. Ramp Test: For the ramp test, we identified this threshold as the lowest quality index for which each subject incorrectly labeled the ramp direction or reported that quality did not change over the ramp.
  2. Pair Test: for the pair test, we identified a foveation quality threshold for each subject as the lowest variable index j
    
    he or she reported as equal to or better in quality than the non-foveated reference.
2. Suppose their quality level is 11,12,13,14,15. What if they get result of 1,1,1,0,1 ? Is their final quality level 13 or 15?
  1. I don’t believe this situation did happen in their user study.
  2. If it happens, what should we do? Of course we should test for multiple scenes for many participants, and get the average. So we go back to problem 1.