Report Number: CS-TR-95-1547
Institution: Stanford University, Department of Computer Science
Title: Sharp, Reliable Predictions using Supervised Mixture Models
Author: Roy, H. Scott
Date: March 1995
Abstract: This dissertation develops a new way to make probabilistic predictions from a database of examples. The method looks for regions in the data where different predictions are appropriate, and it naturally extends clustering algorithms that have been used with great success in exploratory data analysis. In probabilistic terms, the new method looks at the same models as before, but it only evaluates them for the conditional probability they assign to a single feature rather than the joint probability they assign to all features. A good models is therefore forced to classify the data in a way that is useful for a single, desired prediction, rather than just identifying the strongest overall pattern in the data. The results of this dissertation extend the clean, Bayesian approach of the unsupervised AutoClass system to the supervised learning problems common in everyday practice. Highlights include clear probabilistic semantics, prediction and use of discrete, categorical, and continuous data, priors that avoid the overfitting problem, an explicit noise model to identify unreliable predictions, and the ability to handle missing data. A computer implementation, MultiClass, validates the ideas with performance that exceeds neural nets, decision trees, and other current supervised machine learning systems.
http://i.stanford.edu/pub/cstr/reports/cs/tr/95/1547/CS-TR-95-1547.pdf