skip to content
View online Training Recurrent Neural Networks Preview this item
ClosePreview this item
Checking...

Training Recurrent Neural Networks

Author: Hinton, Geoffrey; Sutskever, Ilya
Publisher: 2013-06 2013-08-13T15:23:21Z NO_RESTRICTION 2013-08-13T15:23:21Z 2013-08-13
Dissertation: Thesis / Dissertation ETD
Edition/Format:   Thesis/dissertation : Thesis/dissertation : eBook
Summary:
Recurrent Neural Networks (RNNs) are powerful sequence models that were believed to be difficult to train, and as a result they were rarely used in machine learning applications. This thesis presents methods that overcome the difficulty of training RNNs, and applications of RNNs to challenging problems. We first describe a new probabilistic sequence model that combines Restricted Boltzmann Machines and RNNs. The new  Read more...
Rating:

(not yet rated) 0 with reviews - Be the first.

Subjects
More like this

Find a copy online

Links to this item

Find a copy in the library

&AllPage.SpinnerRetrieving; Finding libraries that hold this item...

Details

Genre/Form: Thesis
Material Type: Thesis/dissertation, Internet resource
Document Type: Internet Resource
All Authors / Contributors: Hinton, Geoffrey; Sutskever, Ilya
OCLC Number: 889910425
Language Note: en_ca

Abstract:

Recurrent Neural Networks (RNNs) are powerful sequence models that were believed to be difficult to train, and as a result they were rarely used in machine learning applications. This thesis presents methods that overcome the difficulty of training RNNs, and applications of RNNs to challenging problems. We first describe a new probabilistic sequence model that combines Restricted Boltzmann Machines and RNNs. The new model is more powerful than similar models while being less difficult to train. Next, we present a new variant of the Hessian-free (HF) optimizer and show that it can train RNNs on tasks that have extreme long-range temporal dependencies, which were previously considered to be impossibly hard. We then apply HF to character-level language modelling and get excellent results. We also apply HF to optimal control and obtain RNN control laws that can successfully operate under conditions of delayed feedback and unknown disturbances. Finally, we describe a random parameter initialization scheme that allows gradient descent with momentum to train RNNs on problems with long-term dependencies. This directly contradicts widespread beliefs about the inability of first-order methods to do so, and suggests that previous attempts at training RNNs failed partly due to flaws in the random initialization.

Reviews

User-contributed reviews
Retrieving GoodReads reviews...
Retrieving DOGObooks reviews...

Tags

Be the first.

Similar Items

Confirm this request

You may have already requested this item. Please select Ok if you would like to proceed with this request anyway.

Linked Data


Primary Entity

<http://www.worldcat.org/oclc/889910425> # Training Recurrent Neural Networks
    a bgn:Thesis, schema:Book, schema:CreativeWork, pto:Web_document ;
    bgn:inSupportOf "Thesis / Dissertation ETD" ;
    library:oclcnum "889910425" ;
    schema:about <http://experiment.worldcat.org/entity/work/data/2067663534#Thing/0984> ; # 0984
    schema:about <http://experiment.worldcat.org/entity/work/data/2067663534#Thing/0800> ; # 0800
    schema:about <http://experiment.worldcat.org/entity/work/data/2067663534#Thing/recurrent_neural_networks> ; # Recurrent Neural Networks
    schema:about <http://experiment.worldcat.org/entity/work/data/2067663534#Thing/0463> ; # 0463
    schema:about <http://experiment.worldcat.org/entity/work/data/2067663534#Thing/optimization> ; # Optimization
    schema:contributor <http://experiment.worldcat.org/entity/work/data/2067663534#Agent/hinton_geoffrey> ; # Hinton, Geoffrey
    schema:creator <http://experiment.worldcat.org/entity/work/data/2067663534#Agent/sutskever_ilya> ; # Sutskever, Ilya
    schema:datePublished "2013/08/13T15:23:21Z" ;
    schema:datePublished "NO_RESTRICTION" ;
    schema:datePublished "2013" ;
    schema:datePublished "2013/08/13" ;
    schema:datePublished "2013/06" ;
    schema:description "Recurrent Neural Networks (RNNs) are powerful sequence models that were believed to be difficult to train, and as a result they were rarely used in machine learning applications. This thesis presents methods that overcome the difficulty of training RNNs, and applications of RNNs to challenging problems. We first describe a new probabilistic sequence model that combines Restricted Boltzmann Machines and RNNs. The new model is more powerful than similar models while being less difficult to train. Next, we present a new variant of the Hessian-free (HF) optimizer and show that it can train RNNs on tasks that have extreme long-range temporal dependencies, which were previously considered to be impossibly hard. We then apply HF to character-level language modelling and get excellent results. We also apply HF to optimal control and obtain RNN control laws that can successfully operate under conditions of delayed feedback and unknown disturbances. Finally, we describe a random parameter initialization scheme that allows gradient descent with momentum to train RNNs on problems with long-term dependencies. This directly contradicts widespread beliefs about the inability of first-order methods to do so, and suggests that previous attempts at training RNNs failed partly due to flaws in the random initialization." ;
    schema:exampleOfWork <http://worldcat.org/entity/work/id/2067663534> ;
    schema:genre "Thesis" ;
    schema:name "Training Recurrent Neural Networks" ;
    schema:productID "889910425" ;
    schema:publication <http://www.worldcat.org/title/-/oclc/889910425#PublicationEvent/2013_06_2013_08_13t15_23_21z_no_restriction_2013_08_13t15_23_21z_2013_08_13> ;
    schema:url <http://hdl.handle.net/1807/36012> ;
    wdrs:describedby <http://www.worldcat.org/title/-/oclc/889910425> ;
    .


Related Entities

<http://experiment.worldcat.org/entity/work/data/2067663534#Agent/hinton_geoffrey> # Hinton, Geoffrey
    a bgn:Agent ;
    schema:name "Hinton, Geoffrey" ;
    .

<http://experiment.worldcat.org/entity/work/data/2067663534#Agent/sutskever_ilya> # Sutskever, Ilya
    a bgn:Agent ;
    schema:name "Sutskever, Ilya" ;
    .

<http://experiment.worldcat.org/entity/work/data/2067663534#Thing/recurrent_neural_networks> # Recurrent Neural Networks
    a schema:Thing ;
    schema:name "Recurrent Neural Networks" ;
    .


Content-negotiable representations

Close Window

Please sign in to WorldCat 

Don't have an account? You can easily create a free account.