Learning long-term dependencies is not as difficult with NARX networks
Journal
Advances in Neural Information Processing Systems
Journal Volume
8
Start Page
577
End Page
583
ISBN (of the container)
978-026220107-0
0262201070
Date Issued
1995
Author(s)
Abstract
It has recently been shown that gradient descent learning algorithms for recurrent neural networks can perform poorly on tasks that involve long-term dependencies. In this paper we explore this problem for a class of architectures called NARX networks, which have powerful representational capabilities. Previous work reported that gradient descent learning is more effective in NARX networks than in recurrent networks with "hidden states". We show that although NARX networks do not circumvent the problem of long-term dependencies, they can greatly improve performance on such problems. We present some experimental results that show that NARX networks can often retain information for two to three times as long as conventional recurrent networks.
Event(s)
8th Advances in Neural Information Processing Systems, NIPS 1995
Publisher
Neural information processing systems foundation
Type
conference paper