Quinn Spencer - Neural Networks: Deep Learning and Machine Learning Outlined

Neural networks are till organized in lr. Lr r made u f a number f interconnected 'nodes' which contain an 'tivtin funtin'. Patterns are rntd t the network vi th 'input lr', whih mmunit t n or more 'hiddn lr' where th tul processing is dn vi a system f wightd 'nntin'.

Mt ANNs ntin some form f 'lrning rule' whih modifies the wight of th nntin according t th inut ttrn that it i presented with. In a sense, ANN lrn by xml as d thir biological counterparts; a hild learns t rgniz dg from xml f dg.

Althugh thr are mn diffrnt kind f lrning rul used by neural networks, this dmntrtin is concerned nl with n; th dlt rul. Th dlt rul is ftn utilizd by th mt common class f ANN called 'bkrgtinl nurl ntwrk' (BPNNs). Bkrgtin is n bbrvitin fr th bkwrd rgtin f rrr.

With th dlt rule, with thr t of backpropagation, 'lrning' i a supervised r tht ur with each l or 'h' (i.. each time th ntwrk i rntd with a nw input ttrn) thrugh a frwrd activation flw of outputs, and th bkwrd rrr propagation f weight djutmnt. More simply, whn a nurl network i initially rntd with a ttrn it mk a random 'guess' t wht it might be. It thn hw fr it answer was from th actual one and mk an rrit djutmnt to its connection wight.

Bkrgtin rfrm a grdint descent within th lutin' vector twrd a 'glbl minimum' lng th steepest vector of th rrr urf. Th global minimum is tht thrtil solution with th lwt possible rrr. Th rrr urf itlf i a hrrblid but i ldm 'mth' as i depicted in th grhi below. Indeed, in most problems, th solution space is quite irrgulr with numru 'pits' nd 'hills' whih may cause th ntwrk t ttl down in a 'local minum' whih is not th bt vrll lutin. Hw the dlt rule find th rrt nwr

Since th ntur f th rrr cannot b known a prioi, nurl network nli ftn r uir a large numbr f individual runs t dtrmin th best solution. Most lrning rul hv built-in mathematical terms t it in thi r whih control the 'speed' (Bt-ffiint) nd the 'mmntum' f the lrning. The d of lrning i actually th rate of convergence btwn the urrnt lutin nd the global minimum. Momentum helps th network t overcome btl (ll minim) in th rrr urf nd ttl down at r nr th global minimum.

Once a nurl ntwrk i 'trind' t a satisfactory lvl it m b used as n nltil tl on thr dt. To d this, th ur n lngr ifi n training run and intd llw the network t wrk in frwrd rgtin md only. New inut r rntd t the inut ttrn whr th filtr int nd are rd by th middl layers thugh training wr taking place, hwvr, t thi int the utut i rtind and no backpropagation ur. The output f a frwrd propagation run i th predicted mdl fr th dt whih n thn b ud for further analysis nd intrrttin.

It is l ibl to vr-trin a nurl ntwrk, whih mn that th network h bn trind xtl t rnd to only n type f inut; which is muh lik rote mmriztin. If thi should hn thn lrning can n lngr ur and th network i rfrrd t having bn "grandmothered" in neural network jrgn. In rl-wrld litin this itutin is not vr useful since one would nd a rt grandmothered ntwrk fr h nw kind f input.