HOGWILD!

Hogwild! is a new way of parallelizing incremental gradient algorithms. The biggest obstacle to achieving linear speedup is minimizing lock contention. Hogwild's approach is simple: get rid of locking entirely! We prove that as long as the data are sparse, Hogwild achieves linear speedups. We demonstrate our theory on a diverse set of problems including text classification with support vector machines, cut problems from vision applications, and recommendation via matrix factorization.

Download the Hogwild! source code!
Check out the examples!

Hogwild! is currently maintained by Victor Bittorf. Any bug reports or questions can be sent to (lastname) at cs dot wisc dot edu.

Best Ball

The newest version of Hogwild! includes a new auto tuning technique that we call "Best Ball." This is analogous to playing "Best Ball" in golf where players with worse position move their balls to the "best ball" before taking each stroke. We use this greedy algorithm as to automatically set parameters such as step sizes and regularizers. Hogwild! can train multiple models, each with different parameters, over a shared scan of the data. After this 'epoch,' the best model is chosen as the starting point of the next 'epoch.'

Publications

Feng Niu, Benjamin Recht, Christopher Ré, and Stephen J. Wright. HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent. Published on NIPS 2011.