JAFROC SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

================================================================================

JAFROC Version 4.2.1

Last Compilation Dates
	JAFROC_CORE.DLL : Aug  7 2014 15:10:56
	JAFROC.EXE      : Aug  7 2014 15:11:24

	Today's date: 5/15/2020 10:48:53



FOM selected         :     ANALYSIS_METHOD_JAFROC_WEIGHTED
Input  Data Filename :     \\vmware-host\Shared Folders\VmWareShared\FedRoc.xlsx
Output Data Filename :     \\vmware-host\Shared Folders\VmWareShared\FedRoc_JAFROC_WEIGHTED.txt

================================================================================
 Analysis method:         :  WEIGHTED JAFROC FOM DBM-MRMC SIGNIFICANCE TESTING
 Number of Readers        :  4
 Number of Treatments     :  5
 Number of Normal Cases   :  100
 Number of Abnormal Cases :  100
 fraction normal cases    :  0.500
 min    LESIONS_PER_IMAGE :  1
 max    LESIONS_PER_IMAGE :  1
 mean   LESIONS_PER_IMAGE :  1
 Total  LESIONS           :  100
 Inc. Loc. Frac.          :  0.000

================================================================================

For TRT = 1         , and RDR = 1         , max FPF =  1.000.
For TRT = 1         , and RDR = 3         , max FPF =  1.000.
For TRT = 1         , and RDR = 4         , max FPF =  1.000.
For TRT = 1         , and RDR = 5         , max FPF =  1.000.
For TRT = 2         , and RDR = 1         , max FPF =  1.000.
For TRT = 2         , and RDR = 3         , max FPF =  1.000.
For TRT = 2         , and RDR = 4         , max FPF =  1.000.
For TRT = 2         , and RDR = 5         , max FPF =  1.000.
For TRT = 3         , and RDR = 1         , max FPF =  1.000.
For TRT = 3         , and RDR = 3         , max FPF =  1.000.
For TRT = 3         , and RDR = 4         , max FPF =  1.000.
For TRT = 3         , and RDR = 5         , max FPF =  1.000.
For TRT = 4         , and RDR = 1         , max FPF =  1.000.
For TRT = 4         , and RDR = 3         , max FPF =  1.000.
For TRT = 4         , and RDR = 4         , max FPF =  1.000.
For TRT = 4         , and RDR = 5         , max FPF =  1.000.
For TRT = 5         , and RDR = 1         , max FPF =  1.000.
For TRT = 5         , and RDR = 3         , max FPF =  1.000.
For TRT = 5         , and RDR = 4         , max FPF =  1.000.
For TRT = 5         , and RDR = 5         , max FPF =  1.000.

================================================================================

 Avg. number of non-lesion localization marks per reader on normal images:  1.000.
 Avg. number of non-lesion localization marks per reader on abnormal images:  0.000.
 Avg. number of lesion localization marks per reader :  1.000.

================================================================================

 ====================================================================
 *****                        Overview                          *****
 ====================================================================
 Three analyses are presented: 
 (1) Analysis 1 treats both readers and cases as random samples
     --results apply to the reader and case populations;
 (2) Analysis 2 treats only cases as a random sample
     --results apply to the population of cases but only for the
     readers used in this study; and
 (3) Analysis 3 treats only readers as a random sample
     --results apply to the population of readers but only for the
     cases used in this study.
 
 For all three analyses, the null hypothesis of equal treatments is
 tested in part (a), treatment difference 95% confidence intervals
 are given in part (b), and treatment 95% confidence intervals are
 given in part (c).  Parts (a) and (b) are based on the treatment x
 reader x case ANOVA while part (c) is based on the reader x case
 ANOVA for the specified treatment; these ANOVA tables are displayed
 before the analyses.  Different error terms are used as indicated
 for parts (a), (b), and (c) according to whether readers and cases
 are treated as fixed or random factors.  Note that the treatment
 confidence intervals in part (c) are based only on the data for the
 specified treatment, rather than the pooled data.  Treatment
 difference 95% confidence intervals for each reader are presented
 in part (d) of Analysis 2; each interval is based on the treatment
 x case ANOVA table (not included) for the specified reader.
 ===========================================================================
 *****                            Estimates                            *****
 ===========================================================================

                        TREATMENT
              --------------------------------------------------------------
  READER      1            2            3            4            5         
----------    ----------   ----------   ----------   ----------   ----------
1             0.90425003   0.86425000   0.81295002   0.90235001   0.84140003
3             0.79820001   0.84469998   0.81634998   0.83149999   0.77300000
4             0.81174999   0.82050002   0.75274998   0.78864998   0.77114999
5             0.86645001   0.87159997   0.85729998   0.87980002   0.84799999
 
 
 TREATMENT MEANS (averaged across readers)
----------    -----------------------------
1             0.84516251
2             0.85026246
3             0.80983752
4             0.85057503
5             0.80838752
 
 

 TREATMENT MEAN DIFFERENCES
----------   ----------    -----------
1          - 2             -0.00509995
1          - 3             0.03532499
1          - 4             -0.00541252
1          - 5             0.03677499
2          - 3             0.04042494
2          - 4             -0.00031257
2          - 5             0.04187495
3          - 4             -0.04073751
3          - 5             0.00145000
4          - 5             0.04218751
 
 
 
 
 ===========================================================================
 *****                          ANOVA Tables                           *****
 ===========================================================================
 
 TREATMENT X READER X CASE ANOVA
 
Source            SS               DF             MS        
------   --------------------    ------   ------------------
     T             1.51756515         4           0.37939129
     R             4.37650833         3           1.45883611
     C           207.50892145       199           1.04275840
    TR             1.10929619        12           0.09244135
    TC            88.81370124       796           0.11157500
    RC           105.84777518       597           0.17729946
   TRC           225.96734996      2388           0.09462619
 TOTAL           635.14111749      3999
 
 
 
 READER X CASE ANOVAs for each treatment
 
 
                        Mean Squares
 Source     df   1            2            3            4            5         
 ------    ---   ----------   ----------   ----------   ----------   ----------   
      R      3   0.48442428   0.10451247   0.37091821   0.51555470   0.35319185
      C    199   0.26881670   0.31010601   0.32304942   0.27755950   0.30952677
     RC    597   0.09986801   0.09691332   0.12588046   0.10121725   0.13192519 
 
 ===========================================================================
 *****                  Variance components estimates                  *****
 ===========================================================================
 
 DBM Variance Component Estimates
 (for sample size estimation for future studies)
 Note: These are unbiased ANOVA estimates which can be negative
 
     DBM Component            Estimate    
 -----------------------  ----------------
 Var(R)                         0.00128372
 Var(C)                         0.04242551
 Var(T*R)                      -0.00001092
 Var(T*C)                       0.00423720
 Var(R*C)                       0.01653465
 Var(T*R*C) + Var(Error)        0.09462619
 
 Obuchowski-Rockette variance component and covariance estimates
 (for sample size estimation for future studies)
 Note: These are unbiased ANOVA estimates which can be negative
 
     OR Component             Estimate    
 -----------------------  ----------------
 Reader                         0.00128372
 Treatment*Reader              -0.00001092
 COV1                           0.00029480
 COV2                           0.00023331
 COV3                           0.00021213
 ERROR                          0.00078912
 
 
 ===========================================================================
 *****           Analysis 1: Random Readers and Random Cases           *****
 ===========================================================================
 (Results apply to the population of readers and cases)


    a) Test for H0: Treatments have the same ANALYSIS_METHOD_JAFROC_WEIGHTED
 
 Source        DF    Mean Square      F value  Pr > F 
 ----------  ------  ---------------  -------  -------
 Treatment        4       0.37939129     3.47   0.0305
 Error        16.80       0.10939016
 Error term: MS(TR) + max[MS(TC)-MS(TRC),0]
 
 Conclusion: The treatment ANALYSIS_METHOD_JAFROC_WEIGHTEDs are not equal,
             F(4,16.80) = 3.47, p = 0.0305.


    b) 95% confidence intervals for treatment differences
 
       Treatment         Estimate   StdErr      DF      t     Pr > t          95% CI      
----------   ----------  --------  --------  -------  ------  -------  -------------------
1          - 2           -0.00510   0.01654    16.80   -0.31   0.7616  -0.04002 ,  0.02982
1          - 3            0.03532   0.01654    16.80    2.14   0.0477   0.00040 ,  0.07025
1          - 4           -0.00541   0.01654    16.80   -0.33   0.7475  -0.04033 ,  0.02951
1          - 5            0.03678   0.01654    16.80    2.22   0.0402   0.00185 ,  0.07170
2          - 3            0.04042   0.01654    16.80    2.44   0.0258   0.00550 ,  0.07535
2          - 4           -0.00031   0.01654    16.80   -0.02   0.9851  -0.03523 ,  0.03461
2          - 5            0.04187   0.01654    16.80    2.53   0.0216   0.00695 ,  0.07680
3          - 4           -0.04074   0.01654    16.80   -2.46   0.0249  -0.07566 , -0.00582
3          - 5            0.00145   0.01654    16.80    0.09   0.9312  -0.03347 ,  0.03637
4          - 5            0.04219   0.01654    16.80    2.55   0.0208   0.00727 ,  0.07711
 
 * H0: the 5 treatments are equal.  To control the overall 
 type I error rate at .05, we conclude that treatment differences
 with p < .05 are significant only if the global test in 
 (a) is also significant (i.e, p < .05).
 Error term: MS(TR) + max[MS(TC)-MS(TRC),0]
 

    c) 95% treatment confidence intervals based on reader x case ANOVAs
       for each treatment (each analysis is based only on data for the
       specified treatment
 
  Treatment     Area      Std Error     DF     95% Confidence Interval 
  ----------  ----------  ----------  -------  -------------------------
  1           0.84516250  0.02857825     5.46  (0.77351390 , 0.91681110)
  2           0.85026249  0.01992816    27.72  (0.80942310 , 0.89110188)
  3           0.80983751  0.02664787     7.04  (0.74689263 , 0.87278239)
  4           0.85057499  0.02940869     5.40  (0.77664336 , 0.92450661)
  5           0.80838750  0.02575833     6.78  (0.74706749 , 0.86970751)
 Error term: MS(R) + max[MS(C)-MS(RC),0]
 
 
 
 ===========================================================================
 *****           Analysis 2: Fixed Readers and Random Cases            *****
 ===========================================================================
 
 (Results apply to the population of cases but only for the readers
 used in this study)
 
    a) Test for H0: Treatments have the same ANALYSIS_METHOD_JAFROC_WEIGHTED
 
 Source        DF    Mean Square      F value  Pr > F 
 ----------  ------  ---------------  -------  -------
 Treatment        4       0.37939129     3.40   0.0091
 Error          796       0.11157500
 Error term: MS(TC)
 
 Conclusion: The treatment ANALYSIS_METHOD_JAFROC_WEIGHTEDs are not equal,
             F(4,796.00) = 3.40, p = 0.0091.


    b) 95% confidence intervals for treatment differences
 
       Treatment         Estimate   StdErr      DF      t     Pr > t          95% CI      
----------   ----------  --------  --------  -------  ------  -------  -------------------
1          - 2           -0.00510   0.01670      796   -0.31   0.7602  -0.03788 ,  0.02768
1          - 3            0.03532   0.01670      796    2.12   0.0347   0.00254 ,  0.06811
1          - 4           -0.00541   0.01670      796   -0.32   0.7460  -0.03820 ,  0.02737
1          - 5            0.03678   0.01670      796    2.20   0.0280   0.00399 ,  0.06956
2          - 3            0.04042   0.01670      796    2.42   0.0157   0.00764 ,  0.07321
2          - 4           -0.00031   0.01670      796   -0.02   0.9851  -0.03310 ,  0.03247
2          - 5            0.04187   0.01670      796    2.51   0.0124   0.00909 ,  0.07466
3          - 4           -0.04074   0.01670      796   -2.44   0.0149  -0.07352 , -0.00795
3          - 5            0.00145   0.01670      796    0.09   0.9308  -0.03133 ,  0.03423
4          - 5            0.04219   0.01670      796    2.53   0.0117   0.00940 ,  0.07497
 
 * H0: the 5 treatments are equal.  To control the overall 
 type I error rate at .05, we conclude that treatment differences
 with p < .05 are significant only if the global test in 
 (a) is also significant (i.e, p < .05).
 Error term: MS(TC)
 

    c) 95% treatment confidence intervals based on reader x case ANOVAs
       for each treatment (each analysis is based only on data for the
       specified treatment
 
  Treatment     Area      Std Error     DF     95% Confidence Interval 
  ----------  ----------  ----------  -------  -------------------------
  1           0.84516250  0.01833087      199  (0.80901482 , 0.88131018)
  2           0.85026249  0.01968839      199  (0.81143785 , 0.88908713)
  3           0.80983751  0.02009507      199  (0.77021091 , 0.84946411)
  4           0.85057499  0.01862658      199  (0.81384419 , 0.88730579)
  5           0.80838750  0.01966999      199  (0.76959913 , 0.84717586)
 Error term: MS(C)
 
 
 
 TREATMENT X CASE ANOVAs for each reader
 

                        Sum of Squares
 Source     df   1             3             4             5             
 ------    ---   -----------   -----------   -----------   -----------   
      T      4     1.2404596     0.6354174     0.6285223     0.1224621   
      C    199    61.7127526    87.7938768    96.4320652    67.4180020   
     TC    796    75.4941581    76.0946836   104.2675707    58.9246388   
 

                        Mean Squares
 Source     df   1             3             4             5             
 ------    ---   -----------   -----------   -----------   -----------   
      T      4     0.3101149     0.1588543     0.1571306     0.0306155   
      C    199     0.3101143     0.4411753     0.4845832     0.3387839   
     TC    796     0.0948419     0.0955963     0.1309894     0.0740259   
 
 


    d) Treatment-by-case ANOVA CIs for each reader 
       (each analysis is based only on data for the specified reader)
 
  Reader         Treatment        Estimate  StdErr       DF      t     Pr > t          95% CI      
---------- ---------- ----------  --------  --------  -------  ------  -------  -------------------
1          1         -2            0.04000   0.03080      199    1.30   0.1955  -0.02073 ,  0.10073
1          1         -3            0.09130   0.03080      199    2.96   0.0034   0.03057 ,  0.15203
1          1         -4            0.00190   0.03080      199    0.06   0.9509  -0.05883 ,  0.06263
1          1         -5            0.06285   0.03080      199    2.04   0.0426   0.00212 ,  0.12358
1          2         -3            0.05130   0.03080      199    1.67   0.0973  -0.00943 ,  0.11203
1          2         -4           -0.03810   0.03080      199   -1.24   0.2175  -0.09883 ,  0.02263
1          2         -5            0.02285   0.03080      199    0.74   0.4590  -0.03788 ,  0.08358
1          3         -4           -0.08940   0.03080      199   -2.90   0.0041  -0.15013 , -0.02867
1          3         -5           -0.02845   0.03080      199   -0.92   0.3567  -0.08918 ,  0.03228
1          4         -5            0.06095   0.03080      199    1.98   0.0492   0.00022 ,  0.12168
3          1         -2           -0.04650   0.03092      199   -1.50   0.1342  -0.10747 ,  0.01447
3          1         -3           -0.01815   0.03092      199   -0.59   0.5579  -0.07912 ,  0.04282
3          1         -4           -0.03330   0.03092      199   -1.08   0.2828  -0.09427 ,  0.02767
3          1         -5            0.02520   0.03092      199    0.82   0.4160  -0.03577 ,  0.08617
3          2         -3            0.02835   0.03092      199    0.92   0.3603  -0.03262 ,  0.08932
3          2         -4            0.01320   0.03092      199    0.43   0.6699  -0.04777 ,  0.07417
3          2         -5            0.07170   0.03092      199    2.32   0.0214   0.01073 ,  0.13267
3          3         -4           -0.01515   0.03092      199   -0.49   0.6247  -0.07612 ,  0.04582
3          3         -5            0.04335   0.03092      199    1.40   0.1625  -0.01762 ,  0.10432
3          4         -5            0.05850   0.03092      199    1.89   0.0599  -0.00247 ,  0.11947
4          1         -2           -0.00875   0.03619      199   -0.24   0.8092  -0.08012 ,  0.06262
4          1         -3            0.05900   0.03619      199    1.63   0.1046  -0.01237 ,  0.13037
4          1         -4            0.02310   0.03619      199    0.64   0.5240  -0.04827 ,  0.09447
4          1         -5            0.04060   0.03619      199    1.12   0.2633  -0.03077 ,  0.11197
4          2         -3            0.06775   0.03619      199    1.87   0.0627  -0.00362 ,  0.13912
4          2         -4            0.03185   0.03619      199    0.88   0.3799  -0.03952 ,  0.10322
4          2         -5            0.04935   0.03619      199    1.36   0.1743  -0.02202 ,  0.12072
4          3         -4           -0.03590   0.03619      199   -0.99   0.3224  -0.10727 ,  0.03547
4          3         -5           -0.01840   0.03619      199   -0.51   0.6117  -0.08977 ,  0.05297
4          4         -5            0.01750   0.03619      199    0.48   0.6293  -0.05387 ,  0.08887
5          1         -2           -0.00515   0.02721      199   -0.19   0.8501  -0.05880 ,  0.04850
5          1         -3            0.00915   0.02721      199    0.34   0.7370  -0.04450 ,  0.06280
5          1         -4           -0.01335   0.02721      199   -0.49   0.6242  -0.06700 ,  0.04030
5          1         -5            0.01845   0.02721      199    0.68   0.4985  -0.03520 ,  0.07210
5          2         -3            0.01430   0.02721      199    0.53   0.5998  -0.03935 ,  0.06795
5          2         -4           -0.00820   0.02721      199   -0.30   0.7634  -0.06185 ,  0.04545
5          2         -5            0.02360   0.02721      199    0.87   0.3868  -0.03005 ,  0.07725
5          3         -4           -0.02250   0.02721      199   -0.83   0.4092  -0.07615 ,  0.03115
5          3         -5            0.00930   0.02721      199    0.34   0.7328  -0.04435 ,  0.06295
5          4         -5            0.03180   0.02721      199    1.17   0.2439  -0.02185 ,  0.08545
 
 
 ===========================================================================
 *****           Analysis 3: Random Readers and Fixed Cases            *****
 ===========================================================================
 (Results apply to the population of readers but only for the cases used in this study)


    a) Test for H0: Treatments have the same ANALYSIS_METHOD_JAFROC_WEIGHTED
 
 Source        DF    Mean Square      F value  Pr > F 
 ----------  ------  ---------------  -------  -------
 Treatment        4       0.37939129     4.10   0.0253
 Error        12.00       0.09244135
 Error term: MS(TR)
 
 Conclusion: The treatment ANALYSIS_METHOD_JAFROC_WEIGHTEDs are not equal,
             F(4,12.00) = 4.10, p = 0.0253.


    b) 95% confidence intervals for treatment differences
 
       Treatment         Estimate   StdErr      DF      t     Pr > t          95% CI      
----------   ----------  --------  --------  -------  ------  -------  -------------------
1          - 2           -0.00510   0.01520    12.00   -0.34   0.7431  -0.03822 ,  0.02802
1          - 3            0.03532   0.01520    12.00    2.32   0.0385   0.00220 ,  0.06845
1          - 4           -0.00541   0.01520    12.00   -0.36   0.7280  -0.03853 ,  0.02771
1          - 5            0.03678   0.01520    12.00    2.42   0.0324   0.00365 ,  0.06990
2          - 3            0.04042   0.01520    12.00    2.66   0.0208   0.00730 ,  0.07355
2          - 4           -0.00031   0.01520    12.00   -0.02   0.9839  -0.03343 ,  0.03281
2          - 5            0.04187   0.01520    12.00    2.75   0.0175   0.00875 ,  0.07500
3          - 4           -0.04074   0.01520    12.00   -2.68   0.0200  -0.07386 , -0.00761
3          - 5            0.00145   0.01520    12.00    0.10   0.9256  -0.03167 ,  0.03457
4          - 5            0.04219   0.01520    12.00    2.78   0.0168   0.00906 ,  0.07531
 
 * H0: the 5 treatments are equal.  To control the overall 
 type I error rate at .05, we conclude that treatment differences
 with p < .05 are significant only if the global test in 
 (a) is also significant (i.e, p < .05).
 
 

    c) Reader-by-case ANOVAs for each treatment (each analysis is based
       only on data for the specified treatment)
 

                        Mean Squares
 Source     df   1            2            3            4            5            
 ------    ---   ----------   ----------   ----------   ----------   ----------   
      R      3   0.48442428   0.10451247   0.37091821   0.51555470   0.35319185   
      C    199   0.26881670   0.31010601   0.32304942   0.27755950   0.30952677   
     RC    597   0.09986801   0.09691332   0.12588046   0.10121725   0.13192519   


 Estimates and 95% Confidence Intervals

  Treatment     Area      Std Error     DF     95% Confidence Interval 
  ----------  ----------  ----------  -------  -------------------------
  1           0.84516250  0.02460753        3  (0.76685037 , 0.92347463)
  2           0.85026249  0.01142981        3  (0.81388773 , 0.88663725)
  3           0.80983751  0.02153248        3  (0.74131154 , 0.87836347)
  4           0.85057499  0.02538589        3  (0.76978576 , 0.93136422)
  5           0.80838750  0.02101166        3  (0.74151902 , 0.87525597)
 



                               REFERENCES

      Dorfman, D.D., Berbaum, K.S., & Metz, C.E. (1992). Receiver operating
 characteristic rating analysis: Generalization to the population of 
 readers and patients with the jackknife method. Investigative Radiology,
 27, 723-731.

      Dorfman, D.D., Berbaum, K.S., Lenth, R.V., Chen, Y.F., & Donaghy, B.A. (1998). 
 Monte Carlo validation of a multireader method for receiver operating characteristic 
 discrete rating data: Factorial experimental design. 
 Academic Radiology, 5, 591-602.

      Hillis, S.L., & Berbaum, K.S. (2004). Power estimation for the
 Dorfman-Berbaum-Metz method. Academic Radiology, 11, 1260-1273.

      Hillis, S.L., Obuchowski, N.A., Schartz, K.M., & Berbaum, K.S.
 (2005). A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette
 methods for receiver operating characteristic (ROC) data. 
 Statistics in Medicine, 24, 1579-1607  DOI:10.1002/sim.2024.

      Hillis, S.L. (2005). Monte Carlo validation of the Dorfman-Berbaum-Metz
 method using normalized pseudovalues and less data-based model simplification
 Academic Radiology, 12:1534-1541  DOI:10.1016/j.acra.2005.07.012.

      Hillis, S.L. (2007). A comparison of denominator degrees of freedom for
 multiple observer ROC analysis.  Statistics in Medicine, 26:596-619  DOI:10.1002/sim.2532.

      Hillis, S.L., Berbaum, K.S., & Metz, C.E. (2008). Recent developments in the
 Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Academic Radiology, in press.

================================================================================
