Back to Search

Sampling-free variational inference of Bayesian neural networks by variance backpropagation

Title	Sampling-free variational inference of Bayesian neural networks by variance backpropagation
Author	Haußmann, M., Hamprecht, F. A., Kandemir, Melih
Publication Date:	2019
Publication Place	- Association For Uncertainty in Artificial Intelligence (AUAI)
Type	Document
Language	English
Digital	Yes
Manuscript	No
Library:	Özyeğin University
Library Asset ID	2-s2.0-85084012503
Record ID	3482b6a0-79c3-4955-9dbc-a336086f2de8
Library Location	Computer Science
Date	2019
Sample Text	We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve this tractability by (i) decomposing ReLU nonlinearities into the product of an identity and a Heaviside step function, (ii) introducing a separate path that decomposes the neural net expectation from its variance. We demonstrate formally that introducing separate latent binary variables to the activations allows representing the neural network likelihood as a chain of linear operations. Performing variational inference on this construction enables a sampling-free computation of the evidence lower bound which is a more effective approximation than the widely applied Monte Carlo sampling and CLT related techniques. We evaluate the model on a range of regression and classification tasks against BNN inference alternatives, showing competitive or improved performance over the current state-of-the-art.

View in source Özyeğin University Özyeğin University - Historical works, archives, and periodicals search engine

Özyeğin University

Sampling-free variational inference of Bayesian neural networks by variance backpropagation

Author Haußmann, M., Hamprecht, F. A., Kandemir, Melih

Publication Date 2019

Publication Place - Association For Uncertainty in Artificial Intelligence (AUAI)

Type Document

Language English

Digital Yes

Manuscript No

Library Özyeğin University

Library Asset ID 2-s2.0-85084012503

Record ID 3482b6a0-79c3-4955-9dbc-a336086f2de8

Library Location Computer Science

Date 2019

Sample Text We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve this tractability by (i) decomposing ReLU nonlinearities into the product of an identity and a Heaviside step function, (ii) introducing a separate path that decomposes the neural net expectation from its variance. We demonstrate formally that introducing separate latent binary variables to the activations allows representing the neural network likelihood as a chain of linear operations. Performing variational inference on this construction enables a sampling-free computation of the evidence lower bound which is a more effective approximation than the widely applied Monte Carlo sampling and CLT related techniques. We evaluate the model on a range of regression and classification tasks against BNN inference alternatives, showing competitive or improved performance over the current state-of-the-art.