There are several limitations of DIFdetect that need to be considered. The ordinal logistic model makes a proportional odds assumption, which means that it assumes that the slopes on the logit scale between any two adjacent response categories are identical. Subsequent versions of DIFdetect (we hope) will automatically check this assumption.

DIFdetect systematically assesses each item for non-uniform and uniform DIF with respect to one covariate at a time. It is at least theoretically possible for an item to have DIF with respect to an interaction between covariates that was not detectable in either univariate analysis. For example, if an item had DIF that was present for black women but not for women in general or blacks in general, this could not be detected by DIFdetect. The best way to examine this sort of hypothesis is to construct a new dataset limited to women only or to blacks only and run DIFdetect again on the new dataset. Another possibility is to generate a stratification variable that accomplishes the same purpose, and then analyze the data with DIFdetect using the new variable.

At present DIFdetect can only handle one observation for each subject. The dataset thus should have one row for each subject. Subsequent editions will be able to perform cluster analysis to account for within-subject variability.

There has been little theoretical work of which we are aware that deals with changes in DIF over time; at present, time is completely ignored by DIFdetect.

Another topic that we have ignored until now is sample size. In general, the more complicated the question, the larger the sample size needed to answer it. If the sample size is much too small for any particular analysis, ologit in STATA will not converge on a solution. When this happens, it will appear in the output in two ways. The first is that all of the models that do not converge are listed in the output. The second is that the item(s) with no solution due to lack of convergence will not be listed at all in the output. The user is cautioned that items not listed have not been analyzed for DIF at all by DIFdetect.

At present we are still unhappy with how DIFdetect handles categorical variables. In our experience, categories with few members are unstable. Many items are found not to converge in this situation, and of those items that do converge, many of them are in turn found to have especially uniform DIF. We think this may be due to the instability of the beta coefficients with smaller samples, and the 10% change criterion is more likely to be met when there is a broader confidence region around the point estimates for the beta coefficient. We are in the process of designing simulation studies to understand this problem more thoroughly and develop some reasonable solutions.

For further questions about DIFdetect, please refer to the Help file, to the DIFdetect web site (located at http://www.alz.washington.edu/DIFDETECT/welcome.html ), or email us at difdet@u.washington.edu . We hope you enjoy DIFdetect, and are eager to receive comments so we can improve subsequent versions.