Reported by : Razvan

Observations : 

    1.  First thing, working with their dataset model is a pain ! Either I had 
        not figure it out, or it allows you to add only one datapoint at a time 
        in the dataset. This seems to me highly unoptimal ...

    2.  You do not get batches for sgd ! The only thing you can do is compare with 
        batch size of 1.

    3.  Their early stopping is different from ours. Differences : 
            - You can not set how often you do a pass on the validation set 
              (i.e. ``patience`` in our case).  You always do one epoch of training 
              and then you go through the validation set.
            - You do not have an improvement thereshold, any improvement in
              validation score leads to storing the new best parameters, and
              increasing the time you will still look for better parameters
            - The increase is not by multiplication but summation. So if at
              epoch x you do better on the validation step, you will go on for 
              x+y epochs to look for something better ( we do x*y )

    4.  The errors return by pyBrain are divided by the number of
        classes. So if you do classification, you take the number of
        errors and divide it by the number of test examples times the
        number of classes. For MNIST this yields 10 times smaller
        errors. Is this something standard .. should we do it ? It
        definetelly makes error look smaller.

    5.  There is no straight forward way of adding L1/L2 regularization (from
        what I've seen), unless you go into their code and change it. That is not
        ard to do .. but for now I do not want to meangle with the library

    6.  The code for RBM is not ready (they say that it is work in progress). It seems to me that the 
        code is wrong .. They have 3 loops, which to me would mean that the inner most is for CD-k (
        second is for one epoch / third for training). But they update the weights after each Gibbs 
        step in CD-k .. which results in a strage form of CD-1 that sees same example several time before
        moving to the next one. I could (?) potentially fix the code but it is outside the scope of 
        benchmarking. 

    7.  There are question marks of how easy it would be to implement a SdA ( autoassociators might be 
        easy to do though).


    RESULTS : 
    logistic_sgd on maggie46

Total error: 0.015611011103
Total error: 0.00966772673335
Total error: 0.00860664508883
Time spend per epoch: 43.32
Final error is : 10.44
Time spend per epoch: 43.32
Final error is : 10.44

    Arac : 

Total error: 0.0366924968888
Total error: 0.0366576944937
Total error: 0.0367442383338
Time spend per epoch: 24.71
Final error is : 69.28
Time spend per epoch: 24.71
Final error is : 69.28


    ** Our thing with batchsize =1 **

test error of best model  8.45
time : 12.99
12.01




    Results : 
    mlp on maggie46


    pybrain :: 

Total error: 0.0124744609817
Total error: 0.00722484141084
Total error: 0.00599591269763
Time spend per epoch : 1226.69
Final error is : 8.68
Time spend per epoch: 1226.69
Final error is : 8.68

20.4448 min

    arac::

Total error: 0.0318599056504
Total error: 0.0316029246672
Total error: 0.0315542295953
Time spend per epoch: 860.336666667 (s)
Final error is : 58.59

    our thing::

test error of best model  3.88
time: 381.92