Update from original

.gitignore

@@ -1,9 +1,11 @@
 *.DS_Store
 *.pyc
+*.swp
 temp/*
 dist/*
 build/*
 keras/datasets/data/*
 keras/datasets/temp/*
 docs/site/*
-docs/theme/*
+docs/theme/*
+tags

.travis.yml

@@ -0,0 +1,18 @@
+sudo: false
+language: python
+# Setup anaconda
+before_install:
+  - wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
+  - chmod +x miniconda.sh
+  - ./miniconda.sh -b
+  - export PATH=/home/travis/miniconda/bin:$PATH
+  - conda update --yes conda
+python:
+  - "3.4"
+# command to install dependencies
+install:
+  - conda install --yes python=$TRAVIS_PYTHON_VERSION numpy scipy matplotlib pandas pytest h5py
+  # Coverage packages are on my binstar channel
+  - python setup.py install
+# command to run tests
+script: py.test tests/

README.md

@@ -6,22 +6,23 @@ Keras is a minimalist, highly modular neural network library in the spirit of To
 
 Use Keras if you need a deep learning library that:
 - allows for easy and fast prototyping (through total modularity, minimalism, and extensibility).
-- supports both convolutional networks (for vision) and recurrent networks (for sequence data). As well as combinations of the two. 
-- runs seamlessly on the CPU and the GPU.
+- supports both convolutional networks and recurrent networks, as well as combinations of the two.
+- supports arbitrary connectivity schemes (including multi-input and multi-output training).
+- runs seamlessly on CPU and GPU.
 
 Read the documentation at [Keras.io](http://keras.io).
 
 Keras is compatible with __Python 2.7-3.4__.
 
 ## Guiding principles
 
-- __Modularity.__ A model is understood as a sequence of standalone, fully-configurable modules that can be plugged together with as little restrictions as possible. In particular, neural layers, cost functions, optimizers, initialization schemes, activation functions and dropout are all standalone modules that you can combine to create new models. 
+- __Modularity.__ A model is understood as a sequence or a graph of standalone, fully-configurable modules that can be plugged together with as little restrictions as possible. In particular, neural layers, cost functions, optimizers, initialization schemes, activation functions, regularization schemes are all standalone modules that you can combine to create new models.
 
-- __Minimalism.__ Each module should be kept short and simple (<100 lines of code). Every piece of code should be transparent upon first reading. No black magic: it hurts iteration speed and ability to innovate. 
+- __Minimalism.__ Each module should be kept short and simple (<100 lines of code). Every piece of code should be transparent upon first reading. No black magic: it hurts iteration speed and ability to innovate.
 
-- __Easy extensibility.__ New features (a new module, per the above definition, or a new way to combine modules together) are dead simple to add (as new classes/functions), and existing modules provide ample examples.
+- __Easy extensibility.__ New modules are dead simple to add (as new classes/functions), and existing modules provide ample examples. To be able to easily create new modules allows for total expressiveness, making Keras suitable for advanced research.
 
-- __Work with Python__. No separate models configuration files in a declarative format (like in Caffe or PyLearn2). Models are described in Python code, which is compact, easier to debug, benefits from syntax highlighting, and most of all, allows for ease of extensibility. See for yourself with the examples below.
+- __Work with Python__. No separate models configuration files in a declarative format (like in Caffe or PyLearn2). Models are described in Python code, which is compact, easier to debug, and allows for ease of extensibility.
 
 ## Examples
 
@@ -171,8 +172,11 @@ model.fit(images, captions, batch_size=16, nb_epoch=100)
 In the examples folder, you will find example models for real datasets:
 - CIFAR10 small images classification: Convnet with realtime data augmentation
 - IMDB movie review sentiment classification: LSTM over sequences of words
-- Reuters newswires topic classification: Multilayer Perceptron
-- MNIST handwritten digits classification: Multilayer Perceptron
+- Reuters newswires topic classification: Multilayer Perceptron (MLP)
+- MNIST handwritten digits classification: MLP & CNN
+- Character-level text generation with LSTM
+
+...and more.
 
 
 ## Current capabilities
@@ -186,24 +190,27 @@ A few highlights: convnets, LSTM, GRU, word2vec-style embeddings, PReLU, batch n
 Keras uses the following dependencies:
 
 - numpy, scipy
-
+- pyyaml
 - Theano
     - See installation instructions: http://deeplearning.net/software/theano/install.html#install
-
 - HDF5 and h5py (optional, required if you use model saving/loading functions)
-
 - Optional but recommended if you use CNNs: cuDNN.
 
 Once you have the dependencies installed, cd to the Keras folder and run the install command:
 ```
 sudo python setup.py install
 ```
 
+You can also install Keras from PyPI:
+```
+sudo pip install keras
+```
+
 ## Why this name, Keras?
 
 Keras (κέρας) means _horn_ in Greek. It is a reference to a literary image from ancient Greek and Latin literature, first found in the _Odyssey_, where dream spirits (_Oneiroi_, singular _Oneiros_) are divided between those who deceive men with false visions, who arrive to Earth through a gate of ivory, and those who announce a future that will come to pass, who arrive through a gate of horn. It's a play on the words κέρας (horn) / κραίνω (fulfill), and ἐλέφας (ivory) / ἐλεφαίρομαι (deceive).
 
 Keras was developed as part of the research effort of project ONEIROS (Open-ended Neuro-Electronic Intelligent Robot Operating System).
 
-_"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them."_ Homer, Odyssey 19. 562 ff (Shewring translation).
+>_"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them."_ Homer, Odyssey 19. 562 ff (Shewring translation).
 

docs/mkdocs.yml

@@ -2,11 +2,9 @@ site_name: Keras Documentation
 theme: readthedocs
 docs_dir: sources
 repo_url: http://github.com/fchollet/keras
-site_url: /
+site_url: http://keras.io/
 #theme_dir: theme
 site_description: Documentation for fast and lightweight Keras Deep Learning library.
-include_404: true
-include_search: true
 
 dev_addr: '0.0.0.0:8000'
 google_analytics: ['UA-61785484-1', 'keras.io']
@@ -32,11 +30,10 @@ pages:
   - Advanced Activations Layers: layers/advanced_activations.md
   - Normalization Layers: layers/normalization.md
   - Embedding Layers: layers/embeddings.md
+  - Noise layers: layers/noise.md
   - Containers: layers/containers.md
 - Preprocessing:
   - Sequence Preprocessing: preprocessing/sequence.md
   - Text Preprocessing: preprocessing/text.md
   - Image Preprocessing: preprocessing/image.md
-- Utils:
-  - Visualization Utilities: utils/visualization.md
 

docs/sources/callbacks.md

@@ -1,6 +1,6 @@
 ## Usage of callbacks
 
-A callback is a set of functions to be applied at given stages of the training procedure. You can use callbacks to get a view on internal states and statistics of the model during training. You can pass a list of callback (as the keyword argument `callbacks`) to the `.fit()` method of the `Sequential` model. The relevant methods of the callbacks will then be called at each stage of the training. 
+A callback is a set of functions to be applied at given stages of the training procedure. You can use callbacks to get a view on internal states and statistics of the model during training. You can pass a list of callbacks (as the keyword argument `callbacks`) to the `.fit()` method of the `Sequential` model. The relevant methods of the callbacks will then be called at each stage of the training. 
 
 ---
 
@@ -37,10 +37,10 @@ Save the model after every epoch. If `save_best_only=True`, the latest best mode
 
 
 ```python
-keras.callbacks.EarlyStopping(patience=0, verbose=0)
+keras.callbacks.EarlyStopping(monitor='val_loss', patience=0, verbose=0)
 ```
 
-Stop training after no improvement of the validation loss is seen for `patience` epochs.
+Stop training after no improvement of the metric `monitor` is seen for `patience` epochs.
 
 ---
 
@@ -52,7 +52,7 @@ You can create a custom callback by extending the base class `keras.callbacks.Ca
 Here's a simple example saving a list of losses over each batch during training:
 ```python
 class LossHistory(keras.callbacks.Callback):
-    def on_train_begin(self):
+    def on_train_begin(self, logs={}):
         self.losses = []
 
     def on_batch_end(self, batch, logs={}):
@@ -61,7 +61,7 @@ class LossHistory(keras.callbacks.Callback):
 
 ---
 
-### Example to record the loss history
+### Example: recording loss history
 
 ```python
 class LossHistory(keras.callbacks.Callback):
@@ -88,7 +88,7 @@ print history.losses
 
 ---
 
-### Example to checkpoint models
+### Example: model checkpoints
 
 ```python
 from keras.callbacks import ModelCheckpoint

docs/sources/constraints.md

@@ -2,9 +2,12 @@
 
 Functions from the `constraints` module allow setting constraints (eg. non-negativity) on network parameters during optimization.
 
-The keyword arguments used for passing constraints to parameters in a layer will depend on the layer. 
+The penalties are applied on a per-layer basis. The exact API will depend on the layer, but the layers `Dense`, `TimeDistributedDense`, `MaxoutDense`, `Convolution1D` and `Convolution2D` have a unified API.
 
-In the `Dense` layer it is simply `W_constraint` for the main weights matrix, and `b_constraint` for the bias.
+These layers expose 2 keyword arguments:
+
+- `W_constraint` for the main weights matrix
+- `b_constraint` for the bias.
 
 
 ```python

docs/sources/documentation.md

@@ -15,6 +15,9 @@
 - [Models](models.md)
 - [Activations](activations.md)
 - [Initializations](initializations.md)
+- [Regularizers](regularizers.md)
+- [Constraints](constraints.md)
+- [Callbacks](callbacks.md)
 - [Datasets](datasets.md)
 
 ---

docs/sources/examples.md

@@ -35,7 +35,7 @@ model.add(Dense(20, 64, init='uniform', activation='tanh'))
 model.add(Dropout(0.5))
 model.add(Dense(64, 64, init='uniform', activation='tanh'))
 model.add(Dropout(0.5))
-model.add(Dense(64, 2, init='uniform', activation='softmax')
+model.add(Dense(64, 2, init='uniform', activation='softmax'))
 
 sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
 model.compile(loss='mean_squared_error', optimizer=sgd)
@@ -92,6 +92,7 @@ from keras.layers.embeddings import Embedding
 from keras.layers.recurrent import LSTM
 
 model = Sequential()
+# Add a mask_zero=True to the Embedding connstructor if 0 is a left-padding value in your data
 model.add(Embedding(max_features, 256))
 model.add(LSTM(256, 128, activation='sigmoid', inner_activation='hard_sigmoid'))
 model.add(Dropout(0.5))
@@ -106,8 +107,9 @@ score = model.evaluate(X_test, Y_test, batch_size=16)
 
 ---
 
-### Architecture for learning image captions with a convnet and a Gated Recurrent Unit
-(word-level embedding, caption of maximum length 16 words).
+### Image captioning
+
+Architecture for learning image captions with a convnet and a Gated Recurrent Unit (word-level embedding, caption of maximum length 16 words).
 
 Note that getting this to actually "work" will require using a bigger convnet, initialized with pre-trained weights.
 Displaying readable results will also require an embedding decoder.

docs/sources/index.md

@@ -2,23 +2,24 @@
 
 ## Overview
 
-Keras is a minimalist, highly modular neural network library in the spirit of Torch, written in Python, that uses [Theano](http://deeplearning.net/software/theano/) under the hood for fast tensor manipulation on GPU and CPU. It was developed with a focus on enabling fast experimentation. 
+Keras is a minimalist, highly modular neural network library in the spirit of Torch, written in Python, that uses [Theano](http://deeplearning.net/software/theano/) under the hood for optimized tensor manipulation on GPU and CPU. It was developed with a focus on enabling fast experimentation. 
 
 Use Keras if you need a deep learning library that:
 
 - allows for easy and fast prototyping (through total modularity, minimalism, and extensibility).
-- supports both __convolutional networks__ and __recurrent networks__ (LSTM, GRU, etc). As well as combinations of the two. 
-- runs seamlessly on the CPU and the GPU.
+- supports both convolutional networks and recurrent networks, as well as combinations of the two.
+- supports arbitrary connectivity schemes (including multi-input and multi-output training).
+- runs seamlessly on CPU and GPU.
 
 ## Guiding principles
 
-- __Modularity.__ A model is understood as a sequence of standalone, fully-configurable modules that can be plugged together with as little restrictions as possible. In particular, neural layers, cost functions, optimizers, initialization schemes, activation functions and dropout are all standalone modules that you can combine to create new models. 
+- __Modularity.__ A model is understood as a sequence or a graph of standalone, fully-configurable modules that can be plugged together with as little restrictions as possible. In particular, neural layers, cost functions, optimizers, initialization schemes, activation functions, regularization schemes are all standalone modules that you can combine to create new models.
 
-- __Minimalism.__ Each module should be kept short and simple (<100 lines of code). Every piece of code should be transparent upon first reading. No black magic: it hurts iteration speed and ability to innovate. 
+- __Minimalism.__ Each module should be kept short and simple (<100 lines of code). Every piece of code should be transparent upon first reading. No black magic: it hurts iteration speed and ability to innovate.
 
-- __Easy extensibility.__ A new feature (a new module, per the above definition, or a new way to combine modules together) are dead simple to add (as new classes/functions), and existing modules provide ample examples.
+- __Easy extensibility.__ New modules are dead simple to add (as new classes/functions), and existing modules provide ample examples. To be able to easily create new modules allows for total expressiveness, making Keras suitable for advanced research.
 
-- __Work with Python__. No separate models configuration files in a declarative format (like in Caffe or PyLearn2). Models are described in Python code, which is compact, easier to debug, benefits from syntax highlighting, and most of all, allows for ease of extensibility.
+- __Work with Python__. No separate models configuration files in a declarative format (like in Caffe or PyLearn2). Models are described in Python code, which is compact, easier to debug, and allows for ease of extensibility.
 
 ## Code
 
@@ -30,7 +31,9 @@ Keras is licensed under the [MIT license](http://opensource.org/licenses/MIT).
 
 ## Getting started: 30 seconds to Keras
 
-The core datastructure of Keras is a __model__, a way to organize layers. Here's a sequential model (a linear pile of layers).
+The core datastructure of Keras is a __model__, a way to organize layers. There are two types of models: [`Sequential`](/models/#sequential) and [`Graph`](/models/#graph).
+
+Here's the `Sequential` model (a linear pile of layers):
 
 ```python
 from keras.models import Sequential
@@ -43,9 +46,9 @@ Stacking layers is as easy as `.add()`:
 ```python
 from keras.layers.core import Dense, Activation
 
-model.add(Dense(input_dim=100, output_dim=64, init="uniform"))
+model.add(Dense(input_dim=100, output_dim=64, init="glorot_uniform"))
 model.add(Activation("relu"))
-model.add(Dense(input_dim=64, output_dim=10, init="uniform"))
+model.add(Dense(input_dim=64, output_dim=10, init="glorot_uniform"))
 model.add(Activation("softmax"))
 ```
 
@@ -67,7 +70,7 @@ model.fit(X_train, Y_train, nb_epoch=5, batch_size=32)
 
 Alternatively, you can feed batches to your model manually:
 ```python
-model.train(X_batch, Y_batch)
+model.train_on_batch(X_batch, Y_batch)
 ```
 
 Evaluate your performance in one line:
@@ -81,19 +84,20 @@ classes = model.predict_classes(X_test, batch_size=32)
 proba = model.predict_proba(X_test, batch_size=32)
 ```
 
-Building a network of LSTMs, a deep CNN, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?
+Building a network of LSTMs, a deep CNN, a Neural Turing Machine, a word2vec embedder or any other model is just as fast. The ideas behind deep learning are simple, so why should their implementation be painful?
 
 Have a look at the [examples](examples.md).
 
 ## Installation
 
 Keras uses the following dependencies:
 
-- numpy, scipy
-- Theano
+- __numpy__, __scipy__
+- __pyyaml__
+- __Theano__
     - See [installation instructions](http://deeplearning.net/software/theano/install.html#install).
-- HDF5 and h5py (optional, required if you use model saving/loading functions)
-- Optional but recommended if you use CNNs: cuDNN.
+- __HDF5__ and __h5py__ (optional, required if you use model saving/loading functions)
+- Optional but recommended if you use CNNs: __cuDNN__.
 
 Once you have the dependencies installed, clone the repo:
 ```bash
@@ -104,6 +108,10 @@ Go to the Keras folder and run the install command:
 cd keras
 sudo python setup.py install
 ```
+You can also install Keras from PyPI:
+```
+sudo pip install keras
+```
 
 ## Support
 
@@ -116,15 +124,16 @@ Keras welcomes all contributions from the community.
 - Keep a pragmatic mindset and avoid bloat. Only add to the source if that is the only path forward.
 - New features should be documented. Make sure you update the documentation along with your Pull Request.
 - The documentation for every new feature should include a usage example in the form of a code snippet. 
-- All changes should be tested. A formal test process will be introduced very soon.
+- All changes should be tested. Make sure any new feature you add has a corresponding unit test.
+- Please no Pull Requests about coding style.
 - Even if you don't contribute to the Keras source code, if you have an application of Keras that is concise and powerful, please consider adding it to our collection of [examples](https://github.com/fchollet/keras/tree/master/examples).
 
 
 ## Why this name, Keras?
 
 Keras (κέρας) means _horn_ in Greek. It is a reference to a literary image from ancient Greek and Latin literature, first found in the _Odyssey_, where dream spirits (_Oneiroi_, singular _Oneiros_) are divided between those who deceive men with false visions, who arrive to Earth through a gate of ivory, and those who announce a future that will come to pass, who arrive through a gate of horn. It's a play on the words κέρας (horn) / κραίνω (fulfill), and ἐλέφας (ivory) / ἐλεφαίρομαι (deceive).
 
-Keras was developed as part of the research effort of project ONEIROS (Open-ended Neuro-Electronic Intelligent Robot Operating System).
+Keras was developed as part of the research effort of project __ONEIROS__ (*Open-ended Neuro-Electronic Intelligent Robot Operating System*).
 
 > _"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them."_ 
 

docs/sources/initializations.md

@@ -1,4 +1,3 @@
-# Initializations
 
 ## Usage of initializations