The Credit Assignment Problem

Conditional reflexes. It is what an animal learns by repeated exposure. If a bell rings, then food is served, the dog will (over time) start salivating when the bell rings. Even before the food is served. What is strange is that such conditional learning does not need a brain. It even happens in creatures without brains. Brain or no brain, such learning is called acquired learning.

 

Say, you now start ringing the bell but don’t serve the food afterwards. For just a few days. As expected, over time, the animal will stop salivating when the bell rings. The association has been removed, a process called extinction. After those few days, you re-establish the bell-food sequence. The animal will start salivating again, a process called spontaneous recovery. Turns out the association was supressed, not deleted.

 

Next, you break the bell-food for a long time. Then after that long period, you restart the bell-food sequence. The animal will start salivating again. The surprise was that the “association will be relearned far more rapidly than the first time”. This is called reacquisition:

“Old extinguished associations are reacquired faster than entirely new associations.”

 

Max Bennett puts all this graphically A Brief History of Intelligence:

This graph makes sense – (1) learn an association only if it keeps recurring and (2) don’t throw it out just because it stopped working a few times.

 

But now let us get into real-world scenarios where:

“There is a never a single predictive cue beforehand but rather a swath of cues.”

Which cue should be learnt? This is called the credit assignment problem.

 

Animals evolved a few simple rules to address this challenge: (1) Eligibility trace: A cue is also considered as worth learning if it happens within a short, specified period of time before the event. (2) Overshadowing: If multiple cuses overlap, pick the strongest ones only. (3) Latent inhibition: Cues experienced regularly are ignored. It is a smart way of asking, “What was different this time?” The rest is just background. (4) Blocking: Once some cue has been picked as the association, all other cues that overlap are blocked from associating with that matter.

 

These 4 methods are ubiquitous across all bilaterally symmetrical species. But clearly, they are nowhere near perfect. Over time, more sophisticated strategies emerged, but these 4 basic rules continue “in our most ancient brain circuits”.

 

Ok, all of the above was the logic. But how does it work physically in the brain? Remember those synapses from a few blogs back? No? Quick recap then. Synapses are tiny gaps between neurons which emit chemicals to strengthen or weaken connections between connected neurons. Well, scientists found that learning happens when synapses change their strength or when new synapses are formed or old synapses are removed.