Implemented R-max learning rule and SRM0 neurons

[Huizerd]

2 additions to BindsNET:

• R-max: Reward-modulated learning rule derived from reward-maximisation principles (see, e.g., Vasilaki et al., 2009, Fremaux et al., 2010, Fremaux et al., 2016)
• SRM0 neurons: Simplified version of the spike response model. Compared to LIF it is more general and it features a stochastic firing threshold (also known as escape noise) that simulates nearby, non-connected neurons that could cause the neuron to spike. This stochastic firing is crucial to the derivation of R-max and other learning rules based on reward-maximisation principles (e.g., Pfister et al., 2006, Fremaux et al., 2013)

I also added the corresponding tests, and changed something about the way spike traces are kept track of:

https://github.com/Hananel-Hazan/bindsnet/blob/4021360cc0f715947bf6dc3f7effab003141cb4c/bindsnet/network/nodes.py#L69

In literature (e.g. Paredes-Valles et al., 2018, Scholarpedia), and in all eligibility traces implemented in the code, spikes/values are added to the trace instead of the trace being reset to 1. It also seems more natural this way.

[djsaunde]

Looks good to me!

In literature (e.g. Paredes-Valles et al., 2018, Scholarpedia), and in all eligibility traces implemented in the code, spikes/values are added to the trace instead of the trace being reset to 1. It also seems more natural this way.

I'm not so sure about this. This should be an option; I like resetting to one as a default for simplicity. This also makes certain learning rule updates easier to interpret.

Out of curiosity, do you have an example showing how this rule works?

[Hananel-Hazan]

@Huizerd Great Job!
If you can make a simple example for each rule that would be great. It will give other people that want to use BindsNET an easy start.
We are compiling a set of examples to use and a Readme file to each one of the examples. It will be great to add your code with your name as the writer to those examples.

[djsaunde]

Maybe a single README for the examples/ folder, with a list entry for each example? I can get this started @Hananel-Hazan.

[Hananel-Hazan]

@djsaunde this is better discussed via email...

[Huizerd]

Do you mean an example of how the different learning rules change weights? Something like this:

This is only for MSTDP and MSTDPET, but I could add a row for R-max

[Hananel-Hazan]

Yes, that is useful.
If you can think about other examples that show how the model works, like implementing the example from the paper, or something simpler.

[djsaunde]

I'm not so sure about this. This should be an option; I like resetting to one as a default for simplicity. This also makes certain learning rule updates easier to interpret.

Bumping this. @Huizerd could you implement this change? Namely, make resetting to 1.0 the default, and adding an option for additive spike traces.

[Huizerd]

@djsaunde Sure! Seems like a good way to allow both.

[djsaunde]

@Hananel-Hazan Looks good to me. Good to merge?

[Huizerd]

I am currently working on a simple simulator in which learning rules like these could be demonstrated/experimented with very easily. However, as it's part of my thesis, I cannot add it just yet. I will certainly do so when it's finished, so any examples that might be lacking now will be made up for later!

[Hananel-Hazan]

Thanks @Huizerd, that is great! We will be glad to see your thesis examples.
Good luck!