Machine Learning for Data Science (CS4786) Lecture 8. Kernel PCA & Isomap + TSNE

Transkrypt

1 Machie Learig for Data Sciece (CS4786) Lecture 8 Kerel PCA & Isomap + TSNE

2 LINEAR PROJECTIONS X d W = Y K d K Works whe data lies i a low dimesioal liear sub-space

3 KERNEL TRICK ( ) ( ) We have have ice methods for liear dimesioality reductio Ca we use this beyod the liear realm?

4 KERNEL TRICK ( ) ( ) Lift to higher dimesios (itroduces o-liearity) Perform liear dimesioality reductio i this high dimesioal space

5 EXAMPLE Origial Data i 2D (x,y) Data Lifted to 3D (x, y, x^2 + y^2)

6 AFIRST CUT Give x t R d, the feature space vector is give by mappig (x t ) = (x t [1],...,x t [d], x t [1] x t [1], x t [1] x t [2],...,x t [d] x t [d],...) Eumeratig products up to order K (ie. products of at most K coordiates) we ca get degree K polyomials. However dimesio blows up as d K Is there a way to do this without eumeratig?

7 KERNEL TRICK Essece of Kerel trick: If we ca write dow a algorithm oly i terms of (x t ) (x s ) for data poits x t ad x s The we do t eed to explicitly eumerate (x t ) s but istead, compute k(x t, x s ) = (x t ) (x s ) (eve if maps to ifiite dimesioal space) Example: RBF kerel k(x t, x s ) = exp( x t x s 2 2 ), polyomial kerel k(x t, x s ) = x t y t p Kerel fuctio measures similarity betwee poits.

8 <latexit sha1_base64="bmiwz9ejxhvgllsfqmjz/cgzm4=">aaadp3iclvjnj9mwfhtc11k+udkylgbwrgqkhusxcqt4mjxkejusuboy7tuviy7+gjal8my78bw5cuxaaia7ccnkayjcd+6qkk3zzhlbszbcqhb89pxll69cvbzzvxfj5q3bd/q7d4+skgzju6akmicxtvyke9bgoq2caxzifx5sxtf/4ltdwqpw1ljrpm7rmrsoybudfu950sdikqzittusifgsuxr+y0jgjhcapjpjyiouqtrtix3gt7bs7kykc274meij1jsugcvspzbjwj029ukhexqtlrrsvstj2g9spdwnilzeo7l9jfaoo91tujvp7oi5ckmix1jonixin2mnmlqeyowkxi5aafuhphflcvflmvx2pg/uewdtrc50hygqhq6jdqfycjykxgc2cswjslaw3zihoqtpk6rwrlnwubuuqzb3oactuv2v2v8uphjdhvxl054jb9c6kimbvlfjtls7p2bk8h/9wbfza+m1ci1wxwg2d0kjiulg5tdgrhjoqpqougeg+fbmvnzsbo3i9twjh2v8+d472x2ewdl89grw875zjb91hd9aqhegpokav0sgaiua98z55x7yv/v/s//n/76v+l43cw/9vf6p8zgb38=</latexit> <latexit sha1_base64="bmiwz9ejxhvgllsfqmjz/cgzm4=">aaadp3iclvjnj9mwfhtc11k+udkylgbwrgqkhusxcqt4mjxkejusuboy7tuviy7+gjal8my78bw5cuxaaia7ccnkayjcd+6qkk3zzhlbszbcqhb89pxll69cvbzzvxfj5q3bd/q7d4+skgzju6akmicxtvyke9bgoq2caxzifx5sxtf/4ltdwqpw1ljrpm7rmrsoybudfu950sdikqzittusifgsuxr+y0jgjhcapjpjyiouqtrtix3gt7bs7kykc274meij1jsugcvspzbjwj029ukhexqtlrrsvstj2g9spdwnilzeo7l9jfaoo91tujvp7oi5ckmix1jonixin2mnmlqeyowkxi5aafuhphflcvflmvx2pg/uewdtrc50hygqhq6jdqfycjykxgc2cswjslaw3zihoqtpk6rwrlnwubuuqzb3oactuv2v2v8uphjdhvxl054jb9c6kimbvlfjtls7p2bk8h/9wbfza+m1ci1wxwg2d0kjiulg5tdgrhjoqpqougeg+fbmvnzsbo3i9twjh2v8+d472x2ewdl89grw875zjb91hd9aqhegpokav0sgaiua98z55x7yv/v/s//n/76v+l43cw/9vf6p8zgb38=</latexit> <latexit sha1_base64="bmiwz9ejxhvgllsfqmjz/cgzm4=">aaadp3iclvjnj9mwfhtc11k+udkylgbwrgqkhusxcqt4mjxkejusuboy7tuviy7+gjal8my78bw5cuxaaia7ccnkayjcd+6qkk3zzhlbszbcqhb89pxll69cvbzzvxfj5q3bd/q7d4+skgzju6akmicxtvyke9bgoq2caxzifx5sxtf/4ltdwqpw1ljrpm7rmrsoybudfu950sdikqzittusifgsuxr+y0jgjhcapjpjyiouqtrtix3gt7bs7kykc274meij1jsugcvspzbjwj029ukhexqtlrrsvstj2g9spdwnilzeo7l9jfaoo91tujvp7oi5ckmix1jonixin2mnmlqeyowkxi5aafuhphflcvflmvx2pg/uewdtrc50hygqhq6jdqfycjykxgc2cswjslaw3zihoqtpk6rwrlnwubuuqzb3oactuv2v2v8uphjdhvxl054jb9c6kimbvlfjtls7p2bk8h/9wbfza+m1ci1wxwg2d0kjiulg5tdgrhjoqpqougeg+fbmvnzsbo3i9twjh2v8+d472x2ewdl89grw875zjb91hd9aqhegpokav0sgaiua98z55x7yv/v/s//n/76v+l43cw/9vf6p8zgb38=</latexit> <latexit sha1_base64="bmiwz9ejxhvgllsfqmjz/cgzm4=">aaadp3iclvjnj9mwfhtc11k+udkylgbwrgqkhusxcqt4mjxkejusuboy7tuviy7+gjal8my78bw5cuxaaia7ccnkayjcd+6qkk3zzhlbszbcqhb89pxll69cvbzzvxfj5q3bd/q7d4+skgzju6akmicxtvyke9bgoq2caxzifx5sxtf/4ltdwqpw1ljrpm7rmrsoybudfu950sdikqzittusifgsuxr+y0jgjhcapjpjyiouqtrtix3gt7bs7kykc274meij1jsugcvspzbjwj029ukhexqtlrrsvstj2g9spdwnilzeo7l9jfaoo91tujvp7oi5ckmix1jonixin2mnmlqeyowkxi5aafuhphflcvflmvx2pg/uewdtrc50hygqhq6jdqfycjykxgc2cswjslaw3zihoqtpk6rwrlnwubuuqzb3oactuv2v2v8uphjdhvxl054jb9c6kimbvlfjtls7p2bk8h/9wbfza+m1ci1wxwg2d0kjiulg5tdgrhjoqpqougeg+fbmvnzsbo3i9twjh2v8+d472x2ewdl89grw875zjb91hd9aqhegpokav0sgaiua98z55x7yv/v/s//n/76v+l43cw/9vf6p8zgb38=</latexit> <latexit sha1_base64="pt6maugdexvb+ij9aqpoyuzdf8=">aaacfhicbvfnj9mwehxc11k+chw5yfgqwm2pkgobl5vwcofyjlq7ut1ajum03jixssdoqyi/g/gjz/cbef0c4bdrrl1/n6mx36tgcudrnhpilxx89btowd3b/fup3j4apj4yytteviybxs9ixhtihzisviuolmwmhkritppjy6affhhvsv19gz8s6zntkzpiz8bqdfielxi5jysbpsuaiwwiaiopmfeigzfrqqy3xcr9tdaythiutro4opg0opnprxc0btexvqe0ot+k202klyisztebpqdq+vp9hizwboyum/g+ndfmvd7r3xh1dc2lr2oolm0d3wdxd0yot4wdpidpjrxpaiak+bcko4mrbtmqxil2ggptcmf2wrvh5wrbru3ezna/erz6q409avcvce/buiyavzuzlxmz1l7qrwkf/tvjvk79enrewnoukxjbjaydc4mwropruc1m4dxq30b8u8z5zx8pmaebpiq1++dk7msziaxz/fji4/9hycogfobrqjgl1dx+gtwqal4uhx8dwyb5pgd/gypaxfx6agqv/zfp0t4ds/oo++2q==</latexit> <latexit sha1_base64="pt6maugdexvb+ij9aqpoyuzdf8=">aaacfhicbvfnj9mwehxc11k+chw5yfgqwm2pkgobl5vwcofyjlq7ut1ajum03jixssdoqyi/g/gjz/cbef0c4bdrrl1/n6mx36tgcudrnhpilxx89btowd3b/fup3j4apj4yytteviybxs9ixhtihzisviuolmwmhkritppjy6affhhvsv19gz8s6zntkzpiz8bqdfielxi5jysbpsuaiwwiaiopmfeigzfrqqy3xcr9tdaythiutro4opg0opnprxc0btexvqe0ot+k202klyisztebpqdq+vp9hizwboyum/g+ndfmvd7r3xh1dc2lr2oolm0d3wdxd0yot4wdpidpjrxpaiak+bcko4mrbtmqxil2ggptcmf2wrvh5wrbru3ezna/erz6q409avcvce/buiyavzuzlxmz1l7qrwkf/tvjvk79enrewnoukxjbjaydc4mwropruc1m4dxq30b8u8z5zx8pmaebpiq1++dk7msziaxz/fji4/9hycogfobrqjgl1dx+gtwqal4uhx8dwyb5pgd/gypaxfx6agqv/zfp0t4ds/oo++2q==</latexit> <latexit sha1_base64="pt6maugdexvb+ij9aqpoyuzdf8=">aaacfhicbvfnj9mwehxc11k+chw5yfgqwm2pkgobl5vwcofyjlq7ut1ajum03jixssdoqyi/g/gjz/cbef0c4bdrrl1/n6mx36tgcudrnhpilxx89btowd3b/fup3j4apj4yytteviybxs9ixhtihzisviuolmwmhkritppjy6affhhvsv19gz8s6zntkzpiz8bqdfielxi5jysbpsuaiwwiaiopmfeigzfrqqy3xcr9tdaythiutro4opg0opnprxc0btexvqe0ot+k202klyisztebpqdq+vp9hizwboyum/g+ndfmvd7r3xh1dc2lr2oolm0d3wdxd0yot4wdpidpjrxpaiak+bcko4mrbtmqxil2ggptcmf2wrvh5wrbru3ezna/erz6q409avcvce/buiyavzuzlxmz1l7qrwkf/tvjvk79enrewnoukxjbjaydc4mwropruc1m4dxq30b8u8z5zx8pmaebpiq1++dk7msziaxz/fji4/9hycogfobrqjgl1dx+gtwqal4uhx8dwyb5pgd/gypaxfx6agqv/zfp0t4ds/oo++2q==</latexit> <latexit sha1_base64="pt6maugdexvb+ij9aqpoyuzdf8=">aaacfhicbvfnj9mwehxc11k+chw5yfgqwm2pkgobl5vwcofyjlq7ut1ajum03jixssdoqyi/g/gjz/cbef0c4bdrrl1/n6mx36tgcudrnhpilxx89btowd3b/fup3j4apj4yytteviybxs9ixhtihzisviuolmwmhkritppjy6affhhvsv19gz8s6zntkzpiz8bqdfielxi5jysbpsuaiwwiaiopmfeigzfrqqy3xcr9tdaythiutro4opg0opnprxc0btexvqe0ot+k202klyisztebpqdq+vp9hizwboyum/g+ndfmvd7r3xh1dc2lr2oolm0d3wdxd0yot4wdpidpjrxpaiak+bcko4mrbtmqxil2ggptcmf2wrvh5wrbru3ezna/erz6q409avcvce/buiyavzuzlxmz1l7qrwkf/tvjvk79enrewnoukxjbjaydc4mwropruc1m4dxq30b8u8z5zx8pmaebpiq1++dk7msziaxz/fji4/9hycogfobrqjgl1dx+gtwqal4uhx8dwyb5pgd/gypaxfx6agqv/zfp0t4ds/oo++2q==</latexit> KERNEL TRICK (x > t y t ) p = = X k 1 +k k d =p X k 1 +k k d =p dy c (x t [j]y t [j]) k j k 1,k 2,...,k d j=1 0s 1 p x t [j] k ta k 1,k 2,...,k d j=1 ( ) ( ) 0s 1 p y t [j] k j A k 1,k 2,...,k d j=1 (x) > = s dy p k 1,k 2,...,k d j=1 1 x t [j] k t,... A k 1 +k k d =p

9 Key Idea: If a algorithm oly depeds o ier products, we ca simply replace ier product i x space by ier product i ɸ(x) space

10 Ca we write PCA so it oly depeds o ier products?

11 <latexit sha1_base64="bsc8xui1gy48u0jcvya5uj5kofy=">aaachicdzdlssnafiyxmu9vv26gsycq5lu2upckhxjsok1qhldzdrrozmlmydicxksn76kgxekck70bzxofrx9yedo+cw5/x+irgc03w1jiapmdms3pl+yxfpexkyuqjilnjwy/gipaplfm8ij1gingp4lkjpqf6/vdg6lev2rs8tg6hlhc3jccrzzglibgxmxhaxyfmsw6keacoygbcz/iripzhyie9z3hj/i480t4euv6matb3dzr3rxgbnnftw3spmvdxybmblk0jvnfbxqzw7g5imiyuacqkubzkjubmrwklgedljfusihzjzzmsbkzapn3s/l8ebmgxweev9isdv9pterkklrqgvo4s11e9aaf+q2skeu27goyqfftgpj4juyihxkruecmkoije2hequd8x0gkhcqsda1if8xor/nyf1mmxwrkngtd0zx1fc62gdbseltvabhaiu6igkrtetukcpxo1xzzwatx+te8z4zg39kphybq+bo3m=</latexit> <latexit sha1_base64="bsc8xui1gy48u0jcvya5uj5kofy=">aaachicdzdlssnafiyxmu9vv26gsycq5lu2upckhxjsok1qhldzdrrozmlmydicxksn76kgxekck70bzxofrx9yedo+cw5/x+irgc03w1jiapmdms3pl+yxfpexkyuqjilnjwy/gipaplfm8ij1gingp4lkjpqf6/vdg6lev2rs8tg6hlhc3jccrzzglibgxmxhaxyfmsw6keacoygbcz/iripzhyie9z3hj/i480t4euv6matb3dzr3rxgbnnftw3spmvdxybmblk0jvnfbxqzw7g5imiyuacqkubzkjubmrwklgedljfusihzjzzmsbkzapn3s/l8ebmgxweev9isdv9pterkklrqgvo4s11e9aaf+q2skeu27goyqfftgpj4juyihxkruecmkoije2hequd8x0gkhcqsda1if8xor/nyf1mmxwrkngtd0zx1fc62gdbseltvabhaiu6igkrtetukcpxo1xzzwatx+te8z4zg39kphybq+bo3m=</latexit> <latexit sha1_base64="bsc8xui1gy48u0jcvya5uj5kofy=">aaachicdzdlssnafiyxmu9vv26gsycq5lu2upckhxjsok1qhldzdrrozmlmydicxksn76kgxekck70bzxofrx9yedo+cw5/x+irgc03w1jiapmdms3pl+yxfpexkyuqjilnjwy/gipaplfm8ij1gingp4lkjpqf6/vdg6lev2rs8tg6hlhc3jccrzzglibgxmxhaxyfmsw6keacoygbcz/iripzhyie9z3hj/i480t4euv6matb3dzr3rxgbnnftw3spmvdxybmblk0jvnfbxqzw7g5imiyuacqkubzkjubmrwklgedljfusihzjzzmsbkzapn3s/l8ebmgxweev9isdv9pterkklrqgvo4s11e9aaf+q2skeu27goyqfftgpj4juyihxkruecmkoije2hequd8x0gkhcqsda1if8xor/nyf1mmxwrkngtd0zx1fc62gdbseltvabhaiu6igkrtetukcpxo1xzzwatx+te8z4zg39kphybq+bo3m=</latexit> <latexit sha1_base64="bsc8xui1gy48u0jcvya5uj5kofy=">aaachicdzdlssnafiyxmu9vv26gsycq5lu2upckhxjsok1qhldzdrrozmlmydicxksn76kgxekck70bzxofrx9yedo+cw5/x+irgc03w1jiapmdms3pl+yxfpexkyuqjilnjwy/gipaplfm8ij1gingp4lkjpqf6/vdg6lev2rs8tg6hlhc3jccrzzglibgxmxhaxyfmsw6keacoygbcz/iripzhyie9z3hj/i480t4euv6matb3dzr3rxgbnnftw3spmvdxybmblk0jvnfbxqzw7g5imiyuacqkubzkjubmrwklgedljfusihzjzzmsbkzapn3s/l8ebmgxweev9isdv9pterkklrqgvo4s11e9aaf+q2skeu27goyqfftgpj4juyihxkruecmkoije2hequd8x0gkhcqsda1if8xor/nyf1mmxwrkngtd0zx1fc62gdbseltvabhaiu6igkrtetukcpxo1xzzwatx+te8z4zg39kphybq+bo3m=</latexit> <latexit sha1_base64="0cw8fbgqg5grgg2m4qf50g2emu=">aaacpicdvbnsxxbeo3rfjjn18ycc2mybhjauixe9sbivhg04lrczmto6arzt6hu6a4nlml8slv8fbjrl4ieiuhu1zvzbicgoe71vrr15wa+wqsv/ryuqjx0+erj3rpx/x8txr/pv1q1c1vsjyvrqyr5lwojwbmsrucfrbegwmyzlndjt98h2su5u5whknssmojsqufbiotd+odrjortqjk9dbnc6skj633rsxa8ru4zzvvxkai5zi4p63klc+lgweziwftyloz0llb4tqkt9arsyxjjtan88zmlygtrtmfhlhdsqafz17ap4vzsjylgjraodflrmbec4tkamh7ceogfimjmeaobelumqv7lt0q2bywlq2teg6yo9uefe6ny+zmnl5dpe1jximzzyjbkvtn0gghlzqgg0xyp2wdjcwzco5weiavxwsuwjcplhslwxqrj9lp4fhg4morvyr58go1+wcayrd+q9+ug42sq7zi/skzgr5af5tf6qi+hdb5dr9vrlei5c5b8k9fv9c4rbgm</latexit> <latexit sha1_base64="0cw8fbgqg5grgg2m4qf50g2emu=">aaacpicdvbnsxxbeo3rfjjn18ycc2mybhjauixe9sbivhg04lrczmto6arzt6hu6a4nlml8slv8fbjrl4ieiuhu1zvzbicgoe71vrr15wa+wqsv/ryuqjx0+erj3rpx/x8txr/pv1q1c1vsjyvrqyr5lwojwbmsrucfrbegwmyzlndjt98h2su5u5whknssmojsqufbiotd+odrjortqjk9dbnc6skj633rsxa8ru4zzvvxkai5zi4p63klc+lgweziwftyloz0llb4tqkt9arsyxjjtan88zmlygtrtmfhlhdsqafz17ap4vzsjylgjraodflrmbec4tkamh7ceogfimjmeaobelumqv7lt0q2bywlq2teg6yo9uefe6ny+zmnl5dpe1jximzzyjbkvtn0gghlzqgg0xyp2wdjcwzco5weiavxwsuwjcplhslwxqrj9lp4fhg4morvyr58go1+wcayrd+q9+ug42sq7zi/skzgr5af5tf6qi+hdb5dr9vrlei5c5b8k9fv9c4rbgm</latexit> <latexit sha1_base64="0cw8fbgqg5grgg2m4qf50g2emu=">aaacpicdvbnsxxbeo3rfjjn18ycc2mybhjauixe9sbivhg04lrczmto6arzt6hu6a4nlml8slv8fbjrl4ieiuhu1zvzbicgoe71vrr15wa+wqsv/ryuqjx0+erj3rpx/x8txr/pv1q1c1vsjyvrqyr5lwojwbmsrucfrbegwmyzlndjt98h2su5u5whknssmojsqufbiotd+odrjortqjk9dbnc6skj633rsxa8ru4zzvvxkai5zi4p63klc+lgweziwftyloz0llb4tqkt9arsyxjjtan88zmlygtrtmfhlhdsqafz17ap4vzsjylgjraodflrmbec4tkamh7ceogfimjmeaobelumqv7lt0q2bywlq2teg6yo9uefe6ny+zmnl5dpe1jximzzyjbkvtn0gghlzqgg0xyp2wdjcwzco5weiavxwsuwjcplhslwxqrj9lp4fhg4morvyr58go1+wcayrd+q9+ug42sq7zi/skzgr5af5tf6qi+hdb5dr9vrlei5c5b8k9fv9c4rbgm</latexit> <latexit sha1_base64="0cw8fbgqg5grgg2m4qf50g2emu=">aaacpicdvbnsxxbeo3rfjjn18ycc2mybhjauixe9sbivhg04lrczmto6arzt6hu6a4nlml8slv8fbjrl4ieiuhu1zvzbicgoe71vrr15wa+wqsv/ryuqjx0+erj3rpx/x8txr/pv1q1c1vsjyvrqyr5lwojwbmsrucfrbegwmyzlndjt98h2su5u5whknssmojsqufbiotd+odrjortqjk9dbnc6skj633rsxa8ru4zzvvxkai5zi4p63klc+lgweziwftyloz0llb4tqkt9arsyxjjtan88zmlygtrtmfhlhdsqafz17ap4vzsjylgjraodflrmbec4tkamh7ceogfimjmeaobelumqv7lt0q2bywlq2teg6yo9uefe6ny+zmnl5dpe1jximzzyjbkvtn0gghlzqgg0xyp2wdjcwzco5weiavxwsuwjcplhslwxqrj9lp4fhg4morvyr58go1+wcayrd+q9+ug42sq7zi/skzgr5af5tf6qi+hdb5dr9vrlei5c5b8k9fv9c4rbgm</latexit> LETS REWRITE PCA Lets start with the assumptio that Data is cetered! (i.e. Sum of xt s is 0) k th colum of W is eigevector of covariace matrix That is, kw k = W k. Rewritig, for cetered X kw k = 1 t=1 x t x t W k = 1 x t W kx t t=1 W s ca be writte as liear combiatio of x s, as But x > t W k = y t [k] = [ ] X kw k = 1 t=1 y t [k]x t

12 <latexit sha1_base64="llr4kx6xpssm0qrocivs2uafm0=">aaacwxicdvbna9taffypteq4x05zzovru+ihgg0ijxmihpzs6cwfog5yrlitulfq5xyfso2q+yh0dox+mhk8ebtlrzgmbe56svvo6i6dyihz3e2x3s2+s/ffb8xcvb/qtlv9zw4kswurrxqxcolcejkdj4vvkurapxmi4/dv70o1qsvov1hxoc3ftvk6kic8lgyomxjetmukcluibk9iznp/bphhvqiuziatbizrvvm1c32iqq4ekg/t+ua5ogbbea1cdl7fqbxlpqaihpzudtjbok8ewgkvrxdmhjvct95ep6fjiz4g3lkeq7bfrtl4ewelras/rwrh3ixhfc0byuljjw0/rh1wqi7fnc48najan282ybtwxivz5oq8naqb9wfhiwr1kxqk7u3n9ej/7lm9wuj+enmlvnaotdorzwqcv0muomlersa0+etmrfcihrja+ctf3idx/cv8l0cjho34l+ph+ydthd12yf6zt4yze3boprelnmgs3bbfwu6wg/wmg7ax9u9kw2dbc8d+qhjwg5pitpy=</latexit> <latexit sha1_base64="llr4kx6xpssm0qrocivs2uafm0=">aaacwxicdvbna9taffypteq4x05zzovru+ihgg0ijxmihpzs6cwfog5yrlitulfq5xyfso2q+yh0dox+mhk8ebtlrzgmbe56svvo6i6dyihz3e2x3s2+s/ffb8xcvb/qtlv9zw4kswurrxqxcolcejkdj4vvkurapxmi4/dv70o1qsvov1hxoc3ftvk6kic8lgyomxjetmukcluibk9iznp/bphhvqiuziatbizrvvm1c32iqq4ekg/t+ua5ogbbea1cdl7fqbxlpqaihpzudtjbok8ewgkvrxdmhjvct95ep6fjiz4g3lkeq7bfrtl4ewelras/rwrh3ixhfc0byuljjw0/rh1wqi7fnc48najan282ybtwxivz5oq8naqb9wfhiwr1kxqk7u3n9ej/7lm9wuj+enmlvnaotdorzwqcv0muomlersa0+etmrfcihrja+ctf3idx/cv8l0cjho34l+ph+ydthd12yf6zt4yze3boprelnmgs3bbfwu6wg/wmg7ax9u9kw2dbc8d+qhjwg5pitpy=</latexit> <latexit sha1_base64="llr4kx6xpssm0qrocivs2uafm0=">aaacwxicdvbna9taffypteq4x05zzovru+ihgg0ijxmihpzs6cwfog5yrlitulfq5xyfso2q+yh0dox+mhk8ebtlrzgmbe56svvo6i6dyihz3e2x3s2+s/ffb8xcvb/qtlv9zw4kswurrxqxcolcejkdj4vvkurapxmi4/dv70o1qsvov1hxoc3ftvk6kic8lgyomxjetmukcluibk9iznp/bphhvqiuziatbizrvvm1c32iqq4ekg/t+ua5ogbbea1cdl7fqbxlpqaihpzudtjbok8ewgkvrxdmhjvct95ep6fjiz4g3lkeq7bfrtl4ewelras/rwrh3ixhfc0byuljjw0/rh1wqi7fnc48najan282ybtwxivz5oq8naqb9wfhiwr1kxqk7u3n9ej/7lm9wuj+enmlvnaotdorzwqcv0muomlersa0+etmrfcihrja+ctf3idx/cv8l0cjho34l+ph+ydthd12yf6zt4yze3boprelnmgs3bbfwu6wg/wmg7ax9u9kw2dbc8d+qhjwg5pitpy=</latexit> <latexit sha1_base64="llr4kx6xpssm0qrocivs2uafm0=">aaacwxicdvbna9taffypteq4x05zzovru+ihgg0ijxmihpzs6cwfog5yrlitulfq5xyfso2q+yh0dox+mhk8ebtlrzgmbe56svvo6i6dyihz3e2x3s2+s/ffb8xcvb/qtlv9zw4kswurrxqxcolcejkdj4vvkurapxmi4/dv70o1qsvov1hxoc3ftvk6kic8lgyomxjetmukcluibk9iznp/bphhvqiuziatbizrvvm1c32iqq4ekg/t+ua5ogbbea1cdl7fqbxlpqaihpzudtjbok8ewgkvrxdmhjvct95ep6fjiz4g3lkeq7bfrtl4ewelras/rwrh3ixhfc0byuljjw0/rh1wqi7fnc48najan282ybtwxivz5oq8naqb9wfhiwr1kxqk7u3n9ej/7lm9wuj+enmlvnaotdorzwqcv0muomlersa0+etmrfcihrja+ctf3idx/cv8l0cjho34l+ph+ydthd12yf6zt4yze3boprelnmgs3bbfwu6wg/wmg7ax9u9kw2dbc8d+qhjwg5pitpy=</latexit> <latexit sha1_base64="nhbhfbioqxlsb95yeu1kk6xrv5e=">aaadrxictvjbixmxge1vkz11tvhx4jll1wko1e7t4uf0rfklgt6tno2qymty0cyh5rrae/dr/gw/+gzpdrlzrenepaodztuac+uhskwkgir60gvda9rs3d261b9+5e+9+z/fbmslrzcwil6ru5wkzqslcjecceuevfixplbgy1enpv4otjfl8q5wlzjmbf7ithigop3w5/2qpl2lmvlppzggmmgbfe2cjru+exhqfxswjtebewtrrapm7smeiyezkxtbzth2+ii48qwl7szdex/udjmbgoay2rqz77/527rdcjiqrwcg/katlfafp/idef/h7gb/o+79bifuq7bsxw/mmxnzprr0oigghuahk+exkwclj/4j0mwkkp120mwhc+uztkte5kiarzsyerbvmldmgurkutwsjksaxbc4mhhysf2zq140c3vnmirns+1maxrpbnyzljvlixc2vc3pwkp+spvukpwvhzvdalgl0fzrtcuuhm6ojvacfardxjx0fffmh8mse/8lzfwtwf4t+ds6meixrk7fpu6cvnobqi/qyhsccjtepeo2gair48dqybu+dd+fhoappolu0bq3nyfohwxwhiadmx</latexit> <latexit sha1_base64="nhbhfbioqxlsb95yeu1kk6xrv5e=">aaadrxictvjbixmxge1vkz11tvhx4jll1wko1e7t4uf0rfklgt6tno2qymty0cyh5rrae/dr/gw/+gzpdrlzrenepaodztuac+uhskwkgir60gvda9rs3d261b9+5e+9+z/fbmslrzcwil6ru5wkzqslcjecceuevfixplbgy1enpv4otjfl8q5wlzjmbf7ithigop3w5/2qpl2lmvlppzggmmgbfe2cjru+exhqfxswjtebewtrrapm7smeiyezkxtbzth2+ii48qwl7szdex/udjmbgoay2rqz77/527rdcjiqrwcg/katlfafp/idef/h7gb/o+79bifuq7bsxw/mmxnzprr0oigghuahk+exkwclj/4j0mwkkp120mwhc+uztkte5kiarzsyerbvmldmgurkutwsjksaxbc4mhhysf2zq140c3vnmirns+1maxrpbnyzljvlixc2vc3pwkp+spvukpwvhzvdalgl0fzrtcuuhm6ojvacfardxjx0fffmh8mse/8lzfwtwf4t+ds6meixrk7fpu6cvnobqi/qyhsccjtepeo2gair48dqybu+dd+fhoappolu0bq3nyfohwxwhiadmx</latexit> <latexit sha1_base64="nhbhfbioqxlsb95yeu1kk6xrv5e=">aaadrxictvjbixmxge1vkz11tvhx4jll1wko1e7t4uf0rfklgt6tno2qymty0cyh5rrae/dr/gw/+gzpdrlzrenepaodztuac+uhskwkgir60gvda9rs3d261b9+5e+9+z/fbmslrzcwil6ru5wkzqslcjecceuevfixplbgy1enpv4otjfl8q5wlzjmbf7ithigop3w5/2qpl2lmvlppzggmmgbfe2cjru+exhqfxswjtebewtrrapm7smeiyezkxtbzth2+ii48qwl7szdex/udjmbgoay2rqz77/527rdcjiqrwcg/katlfafp/idef/h7gb/o+79bifuq7bsxw/mmxnzprr0oigghuahk+exkwclj/4j0mwkkp120mwhc+uztkte5kiarzsyerbvmldmgurkutwsjksaxbc4mhhysf2zq140c3vnmirns+1maxrpbnyzljvlixc2vc3pwkp+spvukpwvhzvdalgl0fzrtcuuhm6ojvacfardxjx0fffmh8mse/8lzfwtwf4t+ds6meixrk7fpu6cvnobqi/qyhsccjtepeo2gair48dqybu+dd+fhoappolu0bq3nyfohwxwhiadmx</latexit> <latexit sha1_base64="nhbhfbioqxlsb95yeu1kk6xrv5e=">aaadrxictvjbixmxge1vkz11tvhx4jll1wko1e7t4uf0rfklgt6tno2qymty0cyh5rrae/dr/gw/+gzpdrlzrenepaodztuac+uhskwkgir60gvda9rs3d261b9+5e+9+z/fbmslrzcwil6ru5wkzqslcjecceuevfixplbgy1enpv4otjfl8q5wlzjmbf7ithigop3w5/2qpl2lmvlppzggmmgbfe2cjru+exhqfxswjtebewtrrapm7smeiyezkxtbzth2+ii48qwl7szdex/udjmbgoay2rqz77/527rdcjiqrwcg/katlfafp/idef/h7gb/o+79bifuq7bsxw/mmxnzprr0oigghuahk+exkwclj/4j0mwkkp120mwhc+uztkte5kiarzsyerbvmldmgurkutwsjksaxbc4mhhysf2zq140c3vnmirns+1maxrpbnyzljvlixc2vc3pwkp+spvukpwvhzvdalgl0fzrtcuuhm6ojvacfardxjx0fffmh8mse/8lzfwtwf4t+ds6meixrk7fpu6cvnobqi/qyhsccjtepeo2gair48dqybu+dd+fhoappolu0bq3nyfohwxwhiadmx</latexit> LETS REWRITE PCA y s [k] =W > k x s = = 1 k = 1 k = 1 k 1 X t=1 X t=1 X t=1 y t [k]x t! > x s y t [k]x > t x s y t [k] K s,t Where K s,t = x > t x s is the kerel matrix for cetered data

13 <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixv1l86vq1rn8eiucozbqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zkivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlt1eldvesarh6kc0vo5wtdmajnchizrxwyasblykjn1yqa8z3hysbg29d77odbu3wmya6mmfxeein3ahd9cblghi4bxevy352suqp569lo4i+8zx84xio4</latexit> <latexit sha1_base64="u4pe0v6npfpzyml7ofa1h+xhtu=">aaacbicbzfnb9qweiad8nuuhs5c4tbqfvucloqlxcpv4lkjyyj120qbkhkcya4v2w2zgev5t/0t3hjj3dhgvcdawmz06t3xjpjz/jgsudr9dmi791/8pdr3v7g8cgtp4fdzwexrm6twimovw2vc+5qsymtkqtwurhida7wkq8+rfjxc7ro1uaclg2mms+mlkxg5k1sejmqfieru3m0aqefzxwcqqi5zfoyw/btk2xuk+wogoj80llu9/h0f0ge3ghkgzogcagt1cx1cuhxyfd4p/czpvqmdbvethwxjjgvt253ctu9sebsnoxaxrfvxrhbxjgb/kikwrqadqfzvguunpxy1jobafjk3dhoukz3dqpeeaxdqtyfvw7j0csr92wructbv7oupauaxofeukibudw5/y01bkj+mtrnsx7szldzkqaavmeaqlqpsy294mjkvyuiobdckd/wweoib3/5rrh8p4qjufwlyvsbxvntljmprazds7gbmie+xw8di6dn8hv8fv4ssevbltuz9k/eb79a5hfwhs=</latexit> <latexit sha1_base64="ipto5fwmk6whph0/jct2hs+hik=">aaacbicdvhlbtnafb2hvwkfurawufavvszctbc0xvrcylojtzbiwym2rph4ohl5zmxmximr5x/g29jxi2zymg5aksc4q6nz7vpcrfbseup/rim7d+/df7dzcpho9/gtp6o93xnxnvbgtfsqspczd6ikwzmxxuflbzhrtoffv7o9ysvwicr89mva0w0xxhzsmf9onlrt9jjv291e4zgihtttirtidx3y6xo1928tn7g7ov1ryfyhb45t8sobqtaouaguc7qrfd4ykjvqbd7qxjsp24lxkm/3otbiqsmjij4wbuo7u5hq3c62qdjsiljdhrajt/rae5ojkdsaqyxquytm5imo+9xxolgo/fccefmjny+ab1uijshhjsoai5aucb2i4rpe01851cbcyhiqwdlezd9fsdkxltxnryxm3hl3t9at/9lmjs8msstn3fhg7gzq0sjwffrvgfzayjdab8cflwfxeetuufdhwcngwu2l8h9wfjrmdmw+ubjdpnx5jawckzekzmyjtmiym/orxqqvy5+dv4odjd2daib356rp2lw5jfcg8cx</latexit> <latexit sha1_base64="olr4a9hk5lk6m6wtezmyarkau9o=">aaacexicdvhlbtnafb2bvwmvfjawubavvszsmsk16ajsbztkbije2kqxzy018i47gzgqciy//at7hjr9iwydykukbwv0fpuym2mlphwu/gjcw7fv3l23c7/34ogjx0/6u08vbfkbgrnrqtjcpdyikhotjqfv5vbxqqkl9p8fadfltfyweppblvhxpc5lpku3hkq6x+lhh51pmjouquedpzkcaprwd0izzpvo83jg8h+nq7reck/d8atviwblpawuaauc9rlfk40ugbqrk6qgtyftoxfpfusfbdkqvarijrzovdjtmfrdtif0cglldeghwdhr9sdk5prirsb6yrfa7kpcdl/hs1kureovdc2imjlysbbpwucttevfusumj5hkceal6gjzvr5fry88wmm92vmoh1+x2r8mla1df6p1djpzvrsp/pu1rl43iruqqdj7y9aksvubk6l4bztl4untkay6m9lecwhddhfof1fmh3lwu/g8udoemdtlhojh7t4ljhzwr8g+yesyjfzmiytisjp4ewwf7wofouvw/3wzdoabpuez+spct/+bjqfwzi=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixv1l86vq1rn8eiucozbqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zkivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlt1eldvesarh6kc0vo5wtdmajnchizrxwyasblykjn1yqa8z3hysbg29d77odbu3wmya6mmfxeein3ahd9cblghi4bxevy352suqp569lo4i+8zx84xio4</latexit> <latexit sha1_base64="u4pe0v6npfpzyml7ofa1h+xhtu=">aaacbicbzfnb9qweiad8nuuhs5c4tbqfvucloqlxcpv4lkjyyj120qbkhkcya4v2w2zgev5t/0t3hjj3dhgvcdawmz06t3xjpjz/jgsudr9dmi791/8pdr3v7g8cgtp4fdzwexrm6twimovw2vc+5qsymtkqtwurhida7wkq8+rfjxc7ro1uaclg2mms+mlkxg5k1sejmqfieru3m0aqefzxwcqqi5zfoyw/btk2xuk+wogoj80llu9/h0f0ge3ghkgzogcagt1cx1cuhxyfd4p/czpvqmdbvethwxjjgvt253ctu9sebsnoxaxrfvxrhbxjgb/kikwrqadqfzvguunpxy1jobafjk3dhoukz3dqpeeaxdqtyfvw7j0csr92wructbv7oupauaxofeukibudw5/y01bkj+mtrnsx7szldzkqaavmeaqlqpsy294mjkvyuiobdckd/wweoib3/5rrh8p4qjufwlyvsbxvntljmprazds7gbmie+xw8di6dn8hv8fv4ssevbltuz9k/eb79a5hfwhs=</latexit> <latexit sha1_base64="ipto5fwmk6whph0/jct2hs+hik=">aaacbicdvhlbtnafb2hvwkfurawufavvszctbc0xvrcylojtzbiwym2rph4ohl5zmxmximr5x/g29jxi2zymg5aksc4q6nz7vpcrfbseup/rim7d+/df7dzcpho9/gtp6o93xnxnvbgtfsqspczd6ikwzmxxuflbzhrtoffv7o9ysvwicr89mva0w0xxhzsmf9onlrt9jjv291e4zgihtttirtidx3y6xo1928tn7g7ov1ryfyhb45t8sobqtaouaguc7qrfd4ykjvqbd7qxjsp24lxkm/3otbiqsmjij4wbuo7u5hq3c62qdjsiljdhrajt/rae5ojkdsaqyxquytm5imo+9xxolgo/fccefmjny+ab1uijshhjsoai5aucb2i4rpe01851cbcyhiqwdlezd9fsdkxltxnryxm3hl3t9at/9lmjs8msstn3fhg7gzq0sjwffrvgfzayjdab8cflwfxeetuufdhwcngwu2l8h9wfjrmdmw+ubjdpnx5jawckzekzmyjtmiym/orxqqvy5+dv4odjd2daib356rp2lw5jfcg8cx</latexit> <latexit sha1_base64="olr4a9hk5lk6m6wtezmyarkau9o=">aaacexicdvhlbtnafb2bvwmvfjawubavvszsmsk16ajsbztkbije2kqxzy018i47gzgqciy//at7hjr9iwydykukbwv0fpuym2mlphwu/gjcw7fv3l23c7/34ogjx0/6u08vbfkbgrnrqtjcpdyikhotjqfv5vbxqqkl9p8fadfltfyweppblvhxpc5lpku3hkq6x+lhh51pmjouquedpzkcaprwd0izzpvo83jg8h+nq7reck/d8atviwblpawuaauc9rlfk40ugbqrk6qgtyftoxfpfusfbdkqvarijrzovdjtmfrdtif0cglldeghwdhr9sdk5prirsb6yrfa7kpcdl/hs1kureovdc2imjlysbbpwucttevfusumj5hkceal6gjzvr5fry88wmm92vmoh1+x2r8mla1df6p1djpzvrsp/pu1rl43iruqqdj7y9aksvubk6l4bztl4untkay6m9lecwhddhfof1fmh3lwu/g8udoemdtlhojh7t4ljhzwr8g+yesyjfzmiytisjp4ewwf7wofouvw/3wzdoabpuez+spct/+bjqfwzi=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="qeecuxpqvxgd3jf+6rf1gdtflkk=">aaacexicdvhlbtnafb2bvwmvaetyxiikkhbbuyqalipvskejkikrrrb1hyyw8m3zxohi8j/wbez4etzsgdepfbdc1de55z7mtfzjyv0u/qjcgzdv3b6zc7d37/6dh4/6j5+c2bi2hce8lkw5yjhfktrohaslyqdtgusz7piq6efl9fyuerpblvhothci1xw5jyv9r/fdr86o5pt1byhhacfheosmftkebnqp0xyjraxxjuayuzlhathddbcic04bijqj3jugyhza2ak5w+bjlvglciu1d5zm14vxhomlb93dfuz0/4ggkzrrcmfdtddd5ehr0ejftoc2km+bmrt47t/pz6vvfaohzfm2imnkpc0zdjbjba9ulzymv6wou491eyhtzqr5fry9cwmc92xmohv+x2r8outsuvewcxif1b68h/adpa5aokebqqq92vsivjbgsum+amta+lryghej/k3af8ww7vx9xwi1y+f/4oz/sgnhvttwedk/saohfkmvcr7hjjdckjoyzhmccc/g+fbbvaq+bw+cpfc12trggx6pi/kz7gzvfwzy=</latexit> <latexit sha1_base64="al25javjn3mfuqxiacqe/m4dpay=">aaadzicjvjnaxsxejxx/ujcnessg+9ddunkrszmqvxdog0vrr6safoapa9agwtlaz9qjpny9b7j3vruzf+jgodn8tu54dg6c28madbyaavrd//wvpqd+7eu7+x2xjw8nhwdn98ymurgyj1kdmouqw6lviuoumulzegeh1qeh9n3vf78uhqr0uqtzji5ipk4uzeshb0v7ntmdouvmrh4o20kyvm+kv8cswf/wng8/ngyubddjgcgs4kwhzmu/yjhkwdqw7jtatizwazvm4kpcylsmewmxxekbfraywpx3c3k/crk1uclicm0wxxbx8r2twyxysya+383telmf7svpd0yc2z9lf8efvo/ai9kj2txyy1ftaqalb/t+z6lfcpad1/7dhwddtu0c7rkuwirzzwgzs9slio8lgkkza3tuz/dqcenkqfl2wc5lrkxuz6wfqcthks7kbbfpitjhlblbp3eoqfe1tr8njawry6ysqpxc9v5o9y/ryj7qbqszajtmt1ocjxgclufw5gykibeuyaf0y5ryam3k0p3c9sucx8fc8gzx12trv04+vwidvl+vyie/jm3jakdkkj+q9osu9imrfve1v19vzvte36o/rt65lvdpss0dwog4/adobf6i=</latexit> <latexit sha1_base64="al25javjn3mfuqxiacqe/m4dpay=">aaadzicjvjnaxsxejxx/ujcnessg+9ddunkrszmqvxdog0vrr6safoapa9agwtlaz9qjpny9b7j3vruzf+jgodn8tu54dg6c28madbyaavrd//wvpqd+7eu7+x2xjw8nhwdn98ymurgyj1kdmouqw6lviuoumulzegeh1qeh9n3vf78uhqr0uqtzji5ipk4uzeshb0v7ntmdouvmrh4o20kyvm+kv8cswf/wng8/ngyubddjgcgs4kwhzmu/yjhkwdqw7jtatizwazvm4kpcylsmewmxxekbfraywpx3c3k/crk1uclicm0wxxbx8r2twyxysya+383telmf7svpd0yc2z9lf8efvo/ai9kj2txyy1ftaqalb/t+z6lfcpad1/7dhwddtu0c7rkuwirzzwgzs9slio8lgkkza3tuz/dqcenkqfl2wc5lrkxuz6wfqcthks7kbbfpitjhlblbp3eoqfe1tr8njawry6ysqpxc9v5o9y/ryj7qbqszajtmt1ocjxgclufw5gykibeuyaf0y5ryam3k0p3c9sucx8fc8gzx12trv04+vwidvl+vyie/jm3jakdkkj+q9osu9imrfve1v19vzvte36o/rt65lvdpss0dwog4/adobf6i=</latexit> <latexit sha1_base64="al25javjn3mfuqxiacqe/m4dpay=">aaadzicjvjnaxsxejxx/ujcnessg+9ddunkrszmqvxdog0vrr6safoapa9agwtlaz9qjpny9b7j3vruzf+jgodn8tu54dg6c28madbyaavrd//wvpqd+7eu7+x2xjw8nhwdn98ymurgyj1kdmouqw6lviuoumulzegeh1qeh9n3vf78uhqr0uqtzji5ipk4uzeshb0v7ntmdouvmrh4o20kyvm+kv8cswf/wng8/ngyubddjgcgs4kwhzmu/yjhkwdqw7jtatizwazvm4kpcylsmewmxxekbfraywpx3c3k/crk1uclicm0wxxbx8r2twyxysya+383telmf7svpd0yc2z9lf8efvo/ai9kj2txyy1ftaqalb/t+z6lfcpad1/7dhwddtu0c7rkuwirzzwgzs9slio8lgkkza3tuz/dqcenkqfl2wc5lrkxuz6wfqcthks7kbbfpitjhlblbp3eoqfe1tr8njawry6ysqpxc9v5o9y/ryj7qbqszajtmt1ocjxgclufw5gykibeuyaf0y5ryam3k0p3c9sucx8fc8gzx12trv04+vwidvl+vyie/jm3jakdkkj+q9osu9imrfve1v19vzvte36o/rt65lvdpss0dwog4/adobf6i=</latexit> <latexit sha1_base64="al25javjn3mfuqxiacqe/m4dpay=">aaadzicjvjnaxsxejxx/ujcnessg+9ddunkrszmqvxdog0vrr6safoapa9agwtlaz9qjpny9b7j3vruzf+jgodn8tu54dg6c28madbyaavrd//wvpqd+7eu7+x2xjw8nhwdn98ymurgyj1kdmouqw6lviuoumulzegeh1qeh9n3vf78uhqr0uqtzji5ipk4uzeshb0v7ntmdouvmrh4o20kyvm+kv8cswf/wng8/ngyubddjgcgs4kwhzmu/yjhkwdqw7jtatizwazvm4kpcylsmewmxxekbfraywpx3c3k/crk1uclicm0wxxbx8r2twyxysya+383telmf7svpd0yc2z9lf8efvo/ai9kj2txyy1ftaqalb/t+z6lfcpad1/7dhwddtu0c7rkuwirzzwgzs9slio8lgkkza3tuz/dqcenkqfl2wc5lrkxuz6wfqcthks7kbbfpitjhlblbp3eoqfe1tr8njawry6ysqpxc9v5o9y/ryj7qbqszajtmt1ocjxgclufw5gykibeuyaf0y5ryam3k0p3c9sucx8fc8gzx12trv04+vwidvl+vyie/jm3jakdkkj+q9osu9imrfve1v19vzvte36o/rt65lvdpss0dwog4/adobf6i=</latexit> <latexit sha1_base64="wwkwlkoxaiojpo+4jxu9hfswj7m=">aaacjicdvdlsgmxfm34rpu16tjnsaiusiji60iouhhcvlct0blkjpopozmkgsemszxupfx3lioiljzu8xobst6iha4596be0+qcq4nqm/o3pzc4tjyzaw6ura+selubxd1kikojqribojigacs9yx3ah2kypg4kcwxja6l/3ehvoaj/lajfpmx2qoecqpmvyaukdetmxteoxjoj/y4s0ikvojotcqk/yosezjao4w+uzlkkwxxydt4bqccgmmswjbhwjs05omoe4cxfpwdtafo2bo/hchgyxk4ykoufo9t4ovggu8gkqpdplhi6ikpwt1ssmgk//zyzgptwcwgukpukgz/qz46cxfqp48bwluvq314p/ux1mxm1/zzlndnm0q+pokxak8aymxhyxagry0sivdzucuktstkzm2zvhvb9kfyfda/rgnxx1vgtdtanowj2wr44abg0qatcgdboaaruwsoyggfwxlyxpzxr9i5z9qza2bgvh8a1m+mow==</latexit> <latexit sha1_base64="wwkwlkoxaiojpo+4jxu9hfswj7m=">aaacjicdvdlsgmxfm34rpu16tjnsaiusiji60iouhhcvlct0blkjpopozmkgsemszxupfx3lioiljzu8xobst6iha4596be0+qcq4nqm/o3pzc4tjyzaw6ura+selubxd1kikojqribojigacs9yx3ah2kypg4kcwxja6l/3ehvoaj/lajfpmx2qoecqpmvyaukdetmxteoxjoj/y4s0ikvojotcqk/yosezjao4w+uzlkkwxxydt4bqccgmmswjbhwjs05omoe4cxfpwdtafo2bo/hchgyxk4ykoufo9t4ovggu8gkqpdplhi6ikpwt1ssmgk//zyzgptwcwgukpukgz/qz46cxfqp48bwluvq314p/ux1mxm1/zzlndnm0q+pokxak8aymxhyxagry0sivdzucuktstkzm2zvhvb9kfyfda/rgnxx1vgtdtanowj2wr44abg0qatcgdboaaruwsoyggfwxlyxpzxr9i5z9qza2bgvh8a1m+mow==</latexit> <latexit sha1_base64="wwkwlkoxaiojpo+4jxu9hfswj7m=">aaacjicdvdlsgmxfm34rpu16tjnsaiusiji60iouhhcvlct0blkjpopozmkgsemszxupfx3lioiljzu8xobst6iha4596be0+qcq4nqm/o3pzc4tjyzaw6ura+selubxd1kikojqribojigacs9yx3ah2kypg4kcwxja6l/3ehvoaj/lajfpmx2qoecqpmvyaukdetmxteoxjoj/y4s0ikvojotcqk/yosezjao4w+uzlkkwxxydt4bqccgmmswjbhwjs05omoe4cxfpwdtafo2bo/hchgyxk4ykoufo9t4ovggu8gkqpdplhi6ikpwt1ssmgk//zyzgptwcwgukpukgz/qz46cxfqp48bwluvq314p/ux1mxm1/zzlndnm0q+pokxak8aymxhyxagry0sivdzucuktstkzm2zvhvb9kfyfda/rgnxx1vgtdtanowj2wr44abg0qatcgdboaaruwsoyggfwxlyxpzxr9i5z9qza2bgvh8a1m+mow==</latexit> <latexit sha1_base64="wwkwlkoxaiojpo+4jxu9hfswj7m=">aaacjicdvdlsgmxfm34rpu16tjnsaiusiji60iouhhcvlct0blkjpopozmkgsemszxupfx3lioiljzu8xobst6iha4596be0+qcq4nqm/o3pzc4tjyzaw6ura+selubxd1kikojqribojigacs9yx3ah2kypg4kcwxja6l/3ehvoaj/lajfpmx2qoecqpmvyaukdetmxteoxjoj/y4s0ikvojotcqk/yosezjao4w+uzlkkwxxydt4bqccgmmswjbhwjs05omoe4cxfpwdtafo2bo/hchgyxk4ykoufo9t4ovggu8gkqpdplhi6ikpwt1ssmgk//zyzgptwcwgukpukgz/qz46cxfqp48bwluvq314p/ux1mxm1/zzlndnm0q+pokxak8aymxhyxagry0sivdzucuktstkzm2zvhvb9kfyfda/rgnxx1vgtdtanowj2wr44abg0qatcgdboaaruwsoyggfwxlyxpzxr9i5z9qza2bgvh8a1m+mow==</latexit> LETS REWRITE PCA Hece, the k th colum o Y matrix is such that =! > Also we have, 1 = kw k k 2 = 1 X 2 y t [k]x t k 2 t=1 = 1 X X 2 y s [k]x > k 2 s x t y t [k] = 1 2 k y[k] = 1 t=1 s=1 2 y[k] Ky[k] > k y[k] K = 1 k ky[k]k 2 <latexit sha1_base64="ixrlrdpat+k71kovixh3sjimojs=">aaacfhicdvdnsgmxgmz6w+tf1aoxybeeosrfbd0irs8ek1gvdtclm2y1njtdkqxq10il76kfw+kepxgzbcxayuo6ebgmpmgfn+eqedaiptutexotc/mvuaq8wuls8u1ldvtwsksh5nrkloq6kz4jl1ddecaekktgu7cwchjb+2tvtmifyxaxt5sfkuvkiu2ksfns296exkujzxoqsesim+yqyfnc78wjirsiohxbuwpdulppbry4accgmmswjbu0is/b22k3chri0lopgjg5qe/p6cc1ijg0vrgsxo9t4ovggu8gkqpdplhi6ijfmtvssmgk//zyqgjtw6cmoufzjaz/v74mcxfop49bolovq314p/uw5myafs5lmhkm6eijkbpqjlbscpa5ytsioswekm53hfsk2i6m7bfqs/i6fp5ptpsnjbr4ekfeorjxuqhryansaqxaoaooqbf0aaw34b48gifzlwp2x0eiem86sgr9wxj8ad/ee3a==</latexit> <latexit sha1_base64="ixrlrdpat+k71kovixh3sjimojs=">aaacfhicdvdnsgmxgmz6w+tf1aoxybeeosrfbd0irs8ek1gvdtclm2y1njtdkqxq10il76kfw+kepxgzbcxayuo6ebgmpmgfn+eqedaiptutexotc/mvuaq8wuls8u1ldvtwsksh5nrkloq6kz4jl1ddecaekktgu7cwchjb+2tvtmifyxaxt5sfkuvkiu2ksfns296exkujzxoqsesim+yqyfnc78wjirsiohxbuwpdulppbry4accgmmswjbu0is/b22k3chri0lopgjg5qe/p6cc1ijg0vrgsxo9t4ovggu8gkqpdplhi6ijfmtvssmgk//zyqgjtw6cmoufzjaz/v74mcxfop49bolovq314p/uw5myafs5lmhkm6eijkbpqjlbscpa5ytsioswekm53hfsk2i6m7bfqs/i6fp5ptpsnjbr4ekfeorjxuqhryansaqxaoaooqbf0aaw34b48gifzlwp2x0eiem86sgr9wxj8ad/ee3a==</latexit> <latexit sha1_base64="ixrlrdpat+k71kovixh3sjimojs=">aaacfhicdvdnsgmxgmz6w+tf1aoxybeeosrfbd0irs8ek1gvdtclm2y1njtdkqxq10il76kfw+kepxgzbcxayuo6ebgmpmgfn+eqedaiptutexotc/mvuaq8wuls8u1ldvtwsksh5nrkloq6kz4jl1ddecaekktgu7cwchjb+2tvtmifyxaxt5sfkuvkiu2ksfns296exkujzxoqsesim+yqyfnc78wjirsiohxbuwpdulppbry4accgmmswjbu0is/b22k3chri0lopgjg5qe/p6cc1ijg0vrgsxo9t4ovggu8gkqpdplhi6ijfmtvssmgk//zyqgjtw6cmoufzjaz/v74mcxfop49bolovq314p/uw5myafs5lmhkm6eijkbpqjlbscpa5ytsioswekm53hfsk2i6m7bfqs/i6fp5ptpsnjbr4ekfeorjxuqhryansaqxaoaooqbf0aaw34b48gifzlwp2x0eiem86sgr9wxj8ad/ee3a==</latexit> <latexit sha1_base64="ixrlrdpat+k71kovixh3sjimojs=">aaacfhicdvdnsgmxgmz6w+tf1aoxybeeosrfbd0irs8ek1gvdtclm2y1njtdkqxq10il76kfw+kepxgzbcxayuo6ebgmpmgfn+eqedaiptutexotc/mvuaq8wuls8u1ldvtwsksh5nrkloq6kz4jl1ddecaekktgu7cwchjb+2tvtmifyxaxt5sfkuvkiu2ksfns296exkujzxoqsesim+yqyfnc78wjirsiohxbuwpdulppbry4accgmmswjbu0is/b22k3chri0lopgjg5qe/p6cc1ijg0vrgsxo9t4ovggu8gkqpdplhi6ijfmtvssmgk//zyqgjtw6cmoufzjaz/v74mcxfop49bolovq314p/uw5myafs5lmhkm6eijkbpqjlbscpa5ytsioswekm53hfsk2i6m7bfqs/i6fp5ptpsnjbr4ekfeorjxuqhryansaqxaoaooqbf0aaw34b48gifzlwp2x0eiem86sgr9wxj8ad/ee3a==</latexit> X s=1 y s [k]x s! Hece P k = y[k]/ p k is a eigevector of K with eige value k = k

14 REWRITTING PCA We assumed cetered data, what if its ot, K s,t = x t 1 = x t x s = x t x s 1 x u ) u=1 u=1 x u u=1 x u x s 1 x s 1 x v v=1 u=1 x ux s 1 x u u=1 x u u=1 x ux t + 1 u=1 2 x t u=1 x ux v v=1

15 REWRITING PCA Equivaletly, if Ker is the matrix (Ker t,s = x t x s), K = Ker (1 Ker) (Ker 1 ) + (1 Ker 1 ) 2

16 KERNEL PCA If we wat to2 port PCA to kerel PCA, we eed to be able to write 3 K i terms of kerel fuctios We assumed cetered data, so Ker K s,t = = (x 6 t ) 1 K = (x t ) (x s ) u=1 = k(x t, x s ) 1 k(x 1,x 1 ) k(x 1,x 2 )... k(x 1,x ) k(x 2,x 1 ) k(x 2,x 2 )... k(x 2,x ) (x u )) (x s ) 1 (x u )) u=1 u=1 k(x 1,x 1 ) k(x 1,x 2 )... k(x 1,x ) =Ker (x u )) (x s ) 1 u=1 u=1 k(x,x 1 ) k(x,x 2 )... k(x,x ) (x u )) v=1 u=1 k(x u, x s ) 1 1 (x v )) k(x u, x t ) + 1 u=1 2 (x u )) (x t ) u=1 k(x u, x v )) v=1 (1 Ker + Ker 1) Ker 1 Kowig kerel fuctio, we ca perform Kerel PCA eve whe maps to ifiite dimesioal space! 7 5

17 <latexit sha1_base64="xeettkp7+vcog1/ac4gzko3/zq=">aaab/hicbvdlsgmxfm3uv62v0s7dbivgqkxe0gxrjcsk9ggdycikmty0yyxjrhig+ituxcji1g9x59+ytrpq1gmxdufcy733rcl2jet1nzw9/y3kpu13z29/yp3mojrk4yrwihjdxr/qhrypmkhcmmp/1uuswitvr5gbm9x6p0iyr9yzpasdwslkyewysflr1doigrx+ukfwrfgkhabq6da/pzqfxcspja5roh+6xp0xijqg0hgotb8hltvbgzrjhdfrzm01ttcz4raewsiyodor58vn4apuhjbnlsxo4v39pffhoyvidgpsxrzm4/eypmxfdbwwsagsrjylgccwgsoescdpmixpdcekwus7dcmsyke2pzqtkq0pllq6r73kree91dnfrxzrxvcaxowbla4bk0wc1ogw4giafp4bw8ou/oi/pufcxak045uwd/4hz+ac7zlhy=</latexit> <latexit sha1_base64="xeettkp7+vcog1/ac4gzko3/zq=">aaab/hicbvdlsgmxfm3uv62v0s7dbivgqkxe0gxrjcsk9ggdycikmty0yyxjrhig+ituxcji1g9x59+ytrpq1gmxdufcy733rcl2jet1nzw9/y3kpu13z29/yp3mojrk4yrwihjdxr/qhrypmkhcmmp/1uuswitvr5gbm9x6p0iyr9yzpasdwslkyewysflr1doigrx+ukfwrfgkhabq6da/pzqfxcspja5roh+6xp0xijqg0hgotb8hltvbgzrjhdfrzm01ttcz4raewsiyodor58vn4apuhjbnlsxo4v39pffhoyvidgpsxrzm4/eypmxfdbwwsagsrjylgccwgsoescdpmixpdcekwus7dcmsyke2pzqtkq0pllq6r73kree91dnfrxzrxvcaxowbla4bk0wc1ogw4giafp4bw8ou/oi/pufcxak045uwd/4hz+ac7zlhy=</latexit> <latexit sha1_base64="xeettkp7+vcog1/ac4gzko3/zq=">aaab/hicbvdlsgmxfm3uv62v0s7dbivgqkxe0gxrjcsk9ggdycikmty0yyxjrhig+ituxcji1g9x59+ytrpq1gmxdufcy733rcl2jet1nzw9/y3kpu13z29/yp3mojrk4yrwihjdxr/qhrypmkhcmmp/1uuswitvr5gbm9x6p0iyr9yzpasdwslkyewysflr1doigrx+ukfwrfgkhabq6da/pzqfxcspja5roh+6xp0xijqg0hgotb8hltvbgzrjhdfrzm01ttcz4raewsiyodor58vn4apuhjbnlsxo4v39pffhoyvidgpsxrzm4/eypmxfdbwwsagsrjylgccwgsoescdpmixpdcekwus7dcmsyke2pzqtkq0pllq6r73kree91dnfrxzrxvcaxowbla4bk0wc1ogw4giafp4bw8ou/oi/pufcxak045uwd/4hz+ac7zlhy=</latexit> <latexit sha1_base64="xeettkp7+vcog1/ac4gzko3/zq=">aaab/hicbvdlsgmxfm3uv62v0s7dbivgqkxe0gxrjcsk9ggdycikmty0yyxjrhig+ituxcji1g9x59+ytrpq1gmxdufcy733rcl2jet1nzw9/y3kpu13z29/yp3mojrk4yrwihjdxr/qhrypmkhcmmp/1uuswitvr5gbm9x6p0iyr9yzpasdwslkyewysflr1doigrx+ukfwrfgkhabq6da/pzqfxcspja5roh+6xp0xijqg0hgotb8hltvbgzrjhdfrzm01ttcz4raewsiyodor58vn4apuhjbnlsxo4v39pffhoyvidgpsxrzm4/eypmxfdbwwsagsrjylgccwgsoescdpmixpdcekwus7dcmsyke2pzqtkq0pllq6r73kree91dnfrxzrxvcaxowbla4bk0wc1ogw4giafp4bw8ou/oi/pufcxak045uwd/4hz+ac7zlhy=</latexit> <latexit sha1_base64="h5wcmylajlk1kf oj2k=">aaab/hicbvbns8naen3ur1q/oj16wsycp5kiomeif6gxcvyd2ham2027ddejuxshhppxvhhqxks/xjv/xm2bg7y+ghi8n8pmvcdhtgh+bzka+sbm1vl7cro7t7+gx141ffxkgltk5jhsheaopxftk2z5rsxsaoi4lqbtg5mfverssxi6f5cfuejciwmglasl5dbflnpfapuuedeqgbfpq2zw7sybv4lbkboq0pltr8ewjqmgksycloq7tqk9hkrmhnnpzzaqmgczwij2dy1auoxl8+o+nqoqxzg0lsk8vz9pzgducotgekuomdq2zuj/39vidxxs6ijnu0iotfycqxjvesctxkkhlnm0oasgzuxwqmeog2evvmco7yy6ukc153bp7d1frxbdxlnexokfyewxqifuuqu1euezekav6m16sl6sd+tj0vqyipkq+gpr8wd/fzsq</latexit> <latexit sha1_base64="h5wcmylajlk1kf oj2k=">aaab/hicbvbns8naen3ur1q/oj16wsycp5kiomeif6gxcvyd2ham2027ddejuxshhppxvhhqxks/xjv/xm2bg7y+ghi8n8pmvcdhtgh+bzka+sbm1vl7cro7t7+gx141ffxkgltk5jhsheaopxftk2z5rsxsaoi4lqbtg5mfverssxi6f5cfuejciwmglasl5dbflnpfapuuedeqgbfpq2zw7sybv4lbkboq0pltr8ewjqmgksycloq7tqk9hkrmhnnpzzaqmgczwij2dy1auoxl8+o+nqoqxzg0lsk8vz9pzgducotgekuomdq2zuj/39vidxxs6ijnu0iotfycqxjvesctxkkhlnm0oasgzuxwqmeog2evvmco7yy6ukc153bp7d1frxbdxlnexokfyewxqifuuqu1euezekav6m16sl6sd+tj0vqyipkq+gpr8wd/fzsq</latexit> <latexit sha1_base64="h5wcmylajlk1kf oj2k=">aaab/hicbvbns8naen3ur1q/oj16wsycp5kiomeif6gxcvyd2ham2027ddejuxshhppxvhhqxks/xjv/xm2bg7y+ghi8n8pmvcdhtgh+bzka+sbm1vl7cro7t7+gx141ffxkgltk5jhsheaopxftk2z5rsxsaoi4lqbtg5mfverssxi6f5cfuejciwmglasl5dbflnpfapuuedeqgbfpq2zw7sybv4lbkboq0pltr8ewjqmgksycloq7tqk9hkrmhnnpzzaqmgczwij2dy1auoxl8+o+nqoqxzg0lsk8vz9pzgducotgekuomdq2zuj/39vidxxs6ijnu0iotfycqxjvesctxkkhlnm0oasgzuxwqmeog2evvmco7yy6ukc153bp7d1frxbdxlnexokfyewxqifuuqu1euezekav6m16sl6sd+tj0vqyipkq+gpr8wd/fzsq</latexit> <latexit sha1_base64="h5wcmylajlk1kf oj2k=">aaab/hicbvbns8naen3ur1q/oj16wsycp5kiomeif6gxcvyd2ham2027ddejuxshhppxvhhqxks/xjv/xm2bg7y+ghi8n8pmvcdhtgh+bzka+sbm1vl7cro7t7+gx141ffxkgltk5jhsheaopxftk2z5rsxsaoi4lqbtg5mfverssxi6f5cfuejciwmglasl5dbflnpfapuuedeqgbfpq2zw7sybv4lbkboq0pltr8ewjqmgksycloq7tqk9hkrmhnnpzzaqmgczwij2dy1auoxl8+o+nqoqxzg0lsk8vz9pzgducotgekuomdq2zuj/39vidxxs6ijnu0iotfycqxjvesctxkkhlnm0oasgzuxwqmeog2evvmco7yy6ukc153bp7d1frxbdxlnexokfyewxqifuuqu1euezekav6m16sl6sd+tj0vqyipkq+gpr8wd/fzsq</latexit> 3. " # P KERNEL PCA (,K) = eigs, K K 4. Y = P 1 p 1 P K p K K K

18 Demo

19 Kerel Methods: A ote We ca kerelize CCA ad ay other liear dimesioality reductio method. For ay liear method, solutio lies withi liear spa of data Hece y s ca be computed oly based o ier products.

20 MANIFOLD BASED DIMENSIONALITY REDUCTION Key Assumptio: Poits live o a low dimesioal maifold Maifold: subspace that looks locally Euclidea Give data, ca we ucover this maifold? Ca we ufold this?

21 METHOD I: ISOMAP 1 For every poit, fid its (k-) Nearest Neighbors 2 Form the Nearest Neighbor graph 3 For every pair of poits A ad B, distace betwee poit A to B is shortest distace betwee A ad B o graph 4 Fid poits i low dimesioal space such that distaces betwee poits i this space is equal to distace o graph.

22 METHOD I: ISOMAP 1 For every poit, fid its (k-) Nearest Neighbors 2 Form the Nearest Neighbor graph 3 For every pair of poits A ad B, distace betwee poit A to B is shortest distace betwee A ad B o graph 4 Fid poits i low dimesioal space such that distaces betwee poits i this space is equal to distace o graph.

23 METHOD I: ISOMAP 1 For every poit, fid its (k-) Nearest Neighbors 2 Form the Nearest Neighbor graph 3 For every pair of poits A ad B, distace betwee poit A to B is shortest distace betwee A ad B o graph 4 Fid poits i low dimesioal space such that distaces betwee poits i this space is equal to distace o graph. Pair-wise Distace Matrix

24 METHOD I: ISOMAP 1 For every poit, fid its (k-) Nearest Neighbors 2 Form the Nearest Neighbor graph 3 For every pair of poits A ad B, distace betwee poit A to B is shortest distace betwee A ad B o graph 4 Fid poits i low dimesioal space such that distaces betwee poits i this space is equal to distace o graph. Pair-wise Distace Matrix

25 ISOMAP: PITFALLS 1 If we do t take eough earest eighbors, the graph may ot be coected 2 If we coect poits too far away, poits that should ot be coected ca get coected 3 There may ot be a right umber of earest eighbors we should cosider!

26 STOCHASTIC NEIGHBORHOOD EMBEDDING Use a probabilistic otio of which poits are eighbors. Close by poits are eighbors with high probability,... Eg: For poit x t, poit x s is picked as eighbor with probability p t s = exp( x s x t ) u t exp( x u x t ) Probability that poits s ad t are coected P s,t = P t,s = p t s+p s t 2 Goal: Fid y 1,...,y with stochastic eighborhood distributio Q such that P ad Q are similar i.e. miimize: KL(PQ) = s,t P s,t log P s,t = P s,t log (P s,t ) P s,t log (Q s,t ) Q s,t s,t s,t

27 STOCHASTIC NEIGHBORHOOD EMBEDDING Use a probabilistic otio of which poits are eighbors. Close by poits are eighbors with high probability,... Eg: For poit x t, poit x s is picked as eighbor with probability p t s = exp( x s x t ) u t exp( x u x t ) Probability that poits s ad t are coected P s,t = P t,s = p t s+p s t 2 Goal: Fid y 1,...,y with stochastic eighborhood distributio Q such that P ad Q are similar i.e. miimize: KL(PQ) = s,t P s,t log P s,t = P s,t log (P s,t ) P s,t log (Q s,t ) Q s,t s,t s,t

28 STOCHASTIC NEIGHBORHOOD EMBEDDING Use a probabilistic otio of which poits are eighbors. Close by poits are eighbors with high probability,... Eg: For poit x t, poit x s is picked as eighbor with probability p t s = exp( x s x t ) u t exp( x u x t ) Probability that poits s ad t are coected P s,t = P t,s = p t s+p s t 2 Goal: Fid y 1,...,y with stochastic eighborhood distributio Q such that P ad Q are similar i.e. miimize: KL(PQ) = s,t P s,t log P s,t = P s,t log (P s,t ) P s,t log (Q s,t ) Q s,t s,t s,t

29 STOCHASTIC NEIGHBORHOOD EMBEDDING Use a probabilistic otio of which poits are eighbors. Close by poits are eighbors with high probability,... Eg: For poit x t, poit x s is picked as eighbor with probability p t s = exp( x s x t ) u t exp( x u x t ) Probability that poits s ad t are coected P s,t = P t,s = p t s+p s t 2 Goal: Fid y 1,...,y with stochastic eighborhood distributio Q such that P ad Q are similar i.e. miimize: KL(PQ) = s,t P s,t log P s,t = P s,t log (P s,t ) P s,t log (Q s,t ) Q s,t s,t s,t

30 CHOICE FOR Q Just like we defied P, we ca defie Q for a give y 1,...,y by q t s = exp( y s y t ) u t exp( y u y t ) ad the set Q s,t = q t s+q s t 2 However we are faced with the crowdig problem: I high dimesio we have a lot of space, Eg. i d dimesio we have d + 1 equidistat poit For d dimesioal gaussias, most poits are foud at distace d from mea! If we use gaussias i both high ad low dimesioal space, all the poits are squished i to a small space Too may poits crowd the ceter!

31 CHOICE FOR Q Just like we defied P, we ca defie Q for a give y 1,...,y by q t s = exp( y s y t ) u t exp( y u y t ) ad the set Q s,t = q t s+q s t 2 However we are faced with the crowdig problem: I high dimesio we have a lot of space, Eg. i d dimesio we have d + 1 equidistat poit For d dimesioal gaussias, most poits are foud at distace d from mea! If we use gaussias i both high ad low dimesioal space, all the poits are squished i to a small space Too may poits crowd the ceter!

32 CHOICE FOR Q Just like we defied P, we ca defie Q for a give y 1,...,y by q t s = exp( y s y t ) u t exp( y u y t ) ad the set Q s,t = q t s+q s t 2 However we are faced with the crowdig problem: I high dimesio we have a lot of space, Eg. i d dimesio we have d + 1 equidistat poit For d dimesioal gaussias, most poits are foud at distace d from mea! If we use gaussias i both high ad low dimesioal space, all the poits are squished i to a small space Too may poits crowd the ceter!

33 CHOICE FOR Q Just like we defied P, we ca defie Q for a give y 1,...,y by q t s = exp( y s y t ) u t exp( y u y t ) ad the set Q s,t = q t s+q s t 2 However we are faced with the crowdig problem: I high dimesio we have a lot of space, Eg. i d dimesio we have d + 1 equidistat poit For d dimesioal gaussias, most poits are foud at distace d from mea! If we use gaussias i both high ad low dimesioal space, all the poits are squished i to a small space Too may poits crowd the ceter!

34 CHOICE FOR Q Just like we defied P, we ca defie Q for a give y 1,...,y by q t s = exp( y s y t ) u t exp( y u y t ) ad the set Q s,t = q t s+q s t 2 However we are faced with the crowdig problem: I high dimesio we have a lot of space, Eg. i d dimesio we have d + 1 equidistat poit For d dimesioal gaussias, most poits are foud at distace d from mea! If we use gaussias i both high ad low dimesioal space, all the poits are squished i to a small space Too may poits crowd the ceter!