Back to Search

Dropout regularization in hierarchical mixture of experts

Title	Dropout regularization in hierarchical mixture of experts
Author	Alpaydın, Ahmet İbrahim Ethem
Publication Date:	2021-01-02
Publication Place	- Elsevier
Subject	Dropout, Hierarchical models, Mixture of experts, Regularization
Type	Periodical
Language	English
Digital	Yes
Manuscript	No
Library:	Özyeğin University
Library Asset ID	0925-2312
Record ID	ac1faa23-e384-4c11-94da-01238a514c35
Library Location	Computer Science
Date	2021-01-02
Sample Text	Dropout is a very effective method in preventing overfitting and has become the go-to regularizer for multi-layer neural networks in recent years. Hierarchical mixture of experts is a hierarchically gated model that defines a soft decision tree where leaves correspond to experts and decision nodes correspond to gating models that softly choose between its children, and as such, the model defines a soft hierarchical partitioning of the input space. In this work, we propose a variant of dropout for hierarchical mixture of experts that is faithful to the tree hierarchy defined by the model, as opposed to having a flat, unitwise independent application of dropout as one has with multi-layer perceptrons. We show that on a synthetic regression data and on MNIST, CIFAR-10, and SSTB datasets, our proposed dropout mechanism prevents overfitting on trees with many levels improving generalization and providing smoother fits.
DOI	10.1016/j.neucom.2020.08.052
Cilt	419

View in source Özyeğin University Özyeğin University - Historical works, archives, and periodicals search engine

Özyeğin University

Dropout regularization in hierarchical mixture of experts

Author Alpaydın, Ahmet İbrahim Ethem

Publication Date 2021-01-02

Publication Place - Elsevier

Subject Dropout, Hierarchical models, Mixture of experts, Regularization

Type Periodical

Language English

Digital Yes

Manuscript No

Library Özyeğin University

Library Asset ID 0925-2312

Record ID ac1faa23-e384-4c11-94da-01238a514c35

Library Location Computer Science

Date 2021-01-02

Sample Text Dropout is a very effective method in preventing overfitting and has become the go-to regularizer for multi-layer neural networks in recent years. Hierarchical mixture of experts is a hierarchically gated model that defines a soft decision tree where leaves correspond to experts and decision nodes correspond to gating models that softly choose between its children, and as such, the model defines a soft hierarchical partitioning of the input space. In this work, we propose a variant of dropout for hierarchical mixture of experts that is faithful to the tree hierarchy defined by the model, as opposed to having a flat, unitwise independent application of dropout as one has with multi-layer perceptrons. We show that on a synthetic regression data and on MNIST, CIFAR-10, and SSTB datasets, our proposed dropout mechanism prevents overfitting on trees with many levels improving generalization and providing smoother fits.

DOI 10.1016/j.neucom.2020.08.052

Cilt 419