Abstract
This paper describes a principled statistical methodof preprocessing incidentally collected electronic medical recordsto facilitate short-term predictions of disease onset withoutexplicit interaction with patients (e.g., medical tests, questionnaires). The model is also applicable to detection of remission. In incidentally collected data, records are possibly left and righttruncated - the first time an event of interest is seen in a patient'sdata may not be the first time in the patient's history that ithappened. It is therefore difficult to know if a disease onsethappens in a given history. If we are unable to determine ifand when the onset occurs, supervised learning and regressionapproaches cannot be applied.Our method determines if an onset occurs in a set of sparseand incomplete patient records, calculates the time of this onsetand provides a principled measure of confidence. It combinesindividual patient history with expectations computed from areference population. We compare the proposed method againststandard change detection algorithms on generated data withrealistic event sparsity and show that it can reliably detect onsetswhere traditional methods fail. We then go on to apply thealgorithm to a large corpus of U.S. Medicare data and show thatthe algorithm scales to large datasets efficiently. The algorithmis currently in trials at a large medical informatics company.