Machine Learning for Data Science (CS4786) Lecture11 5 Random Projections & Canonical Correlation Analysis
The Tall, THE FAT AND THE UGLY n X d
The Tall, THE FAT AND THE UGLY d X > n X d n = n d d
The Tall, THE FAT AND THE UGLY d X > n X d n = n d d,k Eigs( d W = ) K
THE TALL, the Fat AND THE UGLY n X d
THE TALL, the Fat AND THE UGLY n X d SVD(X) d d V > n U n
THE TALL, the Fat AND THE UGLY n X d SVD(X) d K d V > n U n
THE TALL, THE FAT AND the Ugly X d and n so large we can t even store in memory Only have time to be linear in size(x) = n d I there any hope?
PICK A RANDOM W
PICK A RANDOM W 2 3 +1... 1 Y = X 6 4 1... +1 +1... 1 d 7 5 p K +1... 1 K
WHY SHOULD RANDOM PROJECTIONS EVEN WORK?!
RANDOM PROJECTION What does it works even mean? Distances between all pairs of data-points in low dim. projection is roughly the same as their distances in the high dim. space. That is, when K is large enough, with high probability, for all pairs of data points i, j {1,...,n}, (1 ) y i y j 2 x i x j 2 (1 + ) y i y j 2
RANDOM PROJECTION What does it works even mean? Distances between all pairs of data-points in low dim. projection is roughly the same as their distances in the high dim. space. That is, when K is large enough, with high probability, for all pairs of data points i, j {1,...,n}, (1 ) y i y j 2 x i x j 2 (1 + ) y i y j 2
RANDOM PROJECTION What does it works even mean? Distances between all pairs of data-points in low dim. projection is roughly the same as their distances in the high dim. space. That is, when K is large enough, with high probability, for all pairs of data points i, j {1,...,n}, (1 ) y i y j 2 x i x j 2 (1 + ) y i y j 2
<latexit sha1_base64="dooojhwiqbghtwtykiinimx1knw=">aaacfhicdzdlsgmxfiyzxmu9vv26crzlrtokrwy7atgnywr2am0tmttthmyujbmhlpoobnwvny4ucevcnw9jpq2gogcsfr7/hjlzo6hgsip0ys0sli2vrcbwkusbm1vbqz3dugoisvmnbikqtycojrjpapprwzqhzmrzbgs4w/pyb9wwqxjgx+lrydoe6fvc5zrog7qp4wy2b03buvfdnlkvllfbyux+rrmg5ww8us7dcjevrjzccgmmy4elp8iiuqmyx0wiy8tugsyr2k29t3sbjtzmayqiui2mqt0ze6k5fwysbeekhyqosz+1jpsjx1rnpf1qag8n6ue3kob4gk7p94kx8zqaey7p9igeqn9edp/ywpf2i50x98nim5/ohnijaxua44rgj0tgtrgzqajk5q+qdogkvjsckyaer03h/6ketzgy8evjuni2jymb9sebyaimcqaclkav1aafd+abpifn6956tf6s11nrgjwf2qm/ynr7bhygm/m=</latexit> <latexit sha1_base64="dooojhwiqbghtwtykiinimx1knw=">aaacfhicdzdlsgmxfiyzxmu9vv26crzlrtokrwy7atgnywr2am0tmttthmyujbmhlpoobnwvny4ucevcnw9jpq2gogcsfr7/hjlzo6hgsip0ys0sli2vrcbwkusbm1vbqz3dugoisvmnbikqtycojrjpapprwzqhzmrzbgs4w/pyb9wwqxjgx+lrydoe6fvc5zrog7qp4wy2b03buvfdnlkvllfbyux+rrmg5ww8us7dcjevrjzccgmmy4elp8iiuqmyx0wiy8tugsyr2k29t3sbjtzmayqiui2mqt0ze6k5fwysbeekhyqosz+1jpsjx1rnpf1qag8n6ue3kob4gk7p94kx8zqaey7p9igeqn9edp/ywpf2i50x98nim5/ohnijaxua44rgj0tgtrgzqajk5q+qdogkvjsckyaer03h/6ketzgy8evjuni2jymb9sebyaimcqaclkav1aafd+abpifn6956tf6s11nrgjwf2qm/ynr7bhygm/m=</latexit> <latexit sha1_base64="dooojhwiqbghtwtykiinimx1knw=">aaacfhicdzdlsgmxfiyzxmu9vv26crzlrtokrwy7atgnywr2am0tmttthmyujbmhlpoobnwvny4ucevcnw9jpq2gogcsfr7/hjlzo6hgsip0ys0sli2vrcbwkusbm1vbqz3dugoisvmnbikqtycojrjpapprwzqhzmrzbgs4w/pyb9wwqxjgx+lrydoe6fvc5zrog7qp4wy2b03buvfdnlkvllfbyux+rrmg5ww8us7dcjevrjzccgmmy4elp8iiuqmyx0wiy8tugsyr2k29t3sbjtzmayqiui2mqt0ze6k5fwysbeekhyqosz+1jpsjx1rnpf1qag8n6ue3kob4gk7p94kx8zqaey7p9igeqn9edp/ywpf2i50x98nim5/ohnijaxua44rgj0tgtrgzqajk5q+qdogkvjsckyaer03h/6ketzgy8evjuni2jymb9sebyaimcqaclkav1aafd+abpifn6956tf6s11nrgjwf2qm/ynr7bhygm/m=</latexit> <latexit sha1_base64="dooojhwiqbghtwtykiinimx1knw=">aaacfhicdzdlsgmxfiyzxmu9vv26crzlrtokrwy7atgnywr2am0tmttthmyujbmhlpoobnwvny4ucevcnw9jpq2gogcsfr7/hjlzo6hgsip0ys0sli2vrcbwkusbm1vbqz3dugoisvmnbikqtycojrjpapprwzqhzmrzbgs4w/pyb9wwqxjgx+lrydoe6fvc5zrog7qp4wy2b03buvfdnlkvllfbyux+rrmg5ww8us7dcjevrjzccgmmy4elp8iiuqmyx0wiy8tugsyr2k29t3sbjtzmayqiui2mqt0ze6k5fwysbeekhyqosz+1jpsjx1rnpf1qag8n6ue3kob4gk7p94kx8zqaey7p9igeqn9edp/ywpf2i50x98nim5/ohnijaxua44rgj0tgtrgzqajk5q+qdogkvjsckyaer03h/6ketzgy8evjuni2jymb9sebyaimcqaclkav1aafd+abpifn6956tf6s11nrgjwf2qm/ynr7bhygm/m=</latexit> <latexit sha1_base64="ir6boziv9gol8fef+ofcpr4okwo=">aaacsnicdvbnbxmxfpsgam34suhi5ykiivnkv6hnd5uqupryjnjwyi4rr/o2sbr2ruy30gi1+xtcopxgj+dcayr6wdugt8filkyz8+znyapce+l8y9s7sxbz1u31jf6du/fupxhspjzyze0utlrzlo4kkx4lbxfcmgo8qrxkkxv4nj297pzjt+i8lu1rwlsyghlqda6vpccla7licfygnplmwd6ctym9iamsfip1c8vlemlcc3kmetdhh4bszah9fznqpltklqen7aw0lcsvazeohnzeordcqefezjypzhd3vcxgidorymhwoewhf/gsvlvbs6qq3k8fryhppcotcmz7ce2xkupmnui0ucsn+qs5qqkfp0gzqv66cczblfr7rcon9wuthws3vf/b68r/edoa8nhsafvvhfzdp5txbvajxa8w0w4vfytaphi67apqlp1ufnrvhxj+/bt+t462rokpxkvnw/0xqzrw2wp2hd1jgu2wfxbadtmekfaefwjf2nfoq/q5+hzdxkd70wrmefsdvbxvdzy0qa==</latexit> <latexit sha1_base64="ir6boziv9gol8fef+ofcpr4okwo=">aaacsnicdvbnbxmxfpsgam34suhi5ykiivnkv6hnd5uqupryjnjwyi4rr/o2sbr2ruy30gi1+xtcopxgj+dcayr6wdugt8filkyz8+znyapce+l8y9s7sxbz1u31jf6du/fupxhspjzyze0utlrzlo4kkx4lbxfcmgo8qrxkkxv4nj297pzjt+i8lu1rwlsyghlqda6vpccla7licfygnplmwd6ctym9iamsfip1c8vlemlcc3kmetdhh4bszah9fznqpltklqen7aw0lcsvazeohnzeordcqefezjypzhd3vcxgidorymhwoewhf/gsvlvbs6qq3k8fryhppcotcmz7ce2xkupmnui0ucsn+qs5qqkfp0gzqv66cczblfr7rcon9wuthws3vf/b68r/edoa8nhsafvvhfzdp5txbvajxa8w0w4vfytaphi67apqlp1ufnrvhxj+/bt+t462rokpxkvnw/0xqzrw2wp2hd1jgu2wfxbadtmekfaefwjf2nfoq/q5+hzdxkd70wrmefsdvbxvdzy0qa==</latexit> <latexit sha1_base64="ir6boziv9gol8fef+ofcpr4okwo=">aaacsnicdvbnbxmxfpsgam34suhi5ykiivnkv6hnd5uqupryjnjwyi4rr/o2sbr2ruy30gi1+xtcopxgj+dcayr6wdugt8filkyz8+znyapce+l8y9s7sxbz1u31jf6du/fupxhspjzyze0utlrzlo4kkx4lbxfcmgo8qrxkkxv4nj297pzjt+i8lu1rwlsyghlqda6vpccla7licfygnplmwd6ctym9iamsfip1c8vlemlcc3kmetdhh4bszah9fznqpltklqen7aw0lcsvazeohnzeordcqefezjypzhd3vcxgidorymhwoewhf/gsvlvbs6qq3k8fryhppcotcmz7ce2xkupmnui0ucsn+qs5qqkfp0gzqv66cczblfr7rcon9wuthws3vf/b68r/edoa8nhsafvvhfzdp5txbvajxa8w0w4vfytaphi67apqlp1ufnrvhxj+/bt+t462rokpxkvnw/0xqzrw2wp2hd1jgu2wfxbadtmekfaefwjf2nfoq/q5+hzdxkd70wrmefsdvbxvdzy0qa==</latexit> <latexit sha1_base64="ir6boziv9gol8fef+ofcpr4okwo=">aaacsnicdvbnbxmxfpsgam34suhi5ykiivnkv6hnd5uqupryjnjwyi4rr/o2sbr2ruy30gi1+xtcopxgj+dcayr6wdugt8filkyz8+znyapce+l8y9s7sxbz1u31jf6du/fupxhspjzyze0utlrzlo4kkx4lbxfcmgo8qrxkkxv4nj297pzjt+i8lu1rwlsyghlqda6vpccla7licfygnplmwd6ctym9iamsfip1c8vlemlcc3kmetdhh4bszah9fznqpltklqen7aw0lcsvazeohnzeordcqefezjypzhd3vcxgidorymhwoewhf/gsvlvbs6qq3k8fryhppcotcmz7ce2xkupmnui0ucsn+qs5qqkfp0gzqv66cczblfr7rcon9wuthws3vf/b68r/edoa8nhsafvvhfzdp5txbvajxa8w0w4vfytaphi67apqlp1ufnrvhxj+/bt+t462rokpxkvnw/0xqzrw2wp2hd1jgu2wfxbadtmekfaefwjf2nfoq/q5+hzdxkd70wrmefsdvbxvdzy0qa==</latexit> WHY SHOULD RANDOM PROJECTIONS EVEN WORK?! Lets = start with a one dimensional projection = (K = 1) y t = x > t u where each u[i] = random ± 1 = [ ] [ ] What is the expected value of: 1. y t y s? 2. (y t y s ) 2?
= WHY SHOULD RANDOM PROJECTIONS EVEN WORK?! = = = Hence= for any s, t {1,...,n}, = = [ ] [ ] Lets try this in Matlab... Ey s y t 2 = x s x t 2 2
= WHY SHOULD RANDOM PROJECTIONS EVEN WORK?! = = = Hence= for any s, t {1,...,n}, = = [ ] [ ] Lets try this in Matlab... Ey s y t 2 = x s x t 2 2 Law of large numbers says that average over multiple draws is close to expectation
PICK A RANDOM W
PICK A RANDOM W 2 3 +1... 1 Y = X 6 4 1... +1 +1... 1 d 7 5 p K +1... 1 K
<latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> WHY SHOULD RANDOM PROJECTIONS EVEN WORK?! Like = repeating the experiment K times and = averaging y t [k] =x > t u k / p K where each u k [i] = random ± 1 = [ ] [ ]
<latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="y6qd7ctg3gwq+5slaxnajaose5o=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveyjelhozrxc3q0i6tad/baww45vgkhweyarndpfnmsrmgexfpluswwct1bvu7grk1m4o78omqsdyoo18vtvnzsvowt6xsyzsskggtj0p8z+3ghouk1tww3sj/v2oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h7xcbw0=</latexit> <latexit sha1_base64="y6qd7ctg3gwq+5slaxnajaose5o=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveyjelhozrxc3q0i6tad/baww45vgkhweyarndpfnmsrmgexfpluswwct1bvu7grk1m4o78omqsdyoo18vtvnzsvowt6xsyzsskggtj0p8z+3ghouk1tww3sj/v2oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h7xcbw0=</latexit> <latexit sha1_base64="y6qd7ctg3gwq+5slaxnajaose5o=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveyjelhozrxc3q0i6tad/baww45vgkhweyarndpfnmsrmgexfpluswwct1bvu7grk1m4o78omqsdyoo18vtvnzsvowt6xsyzsskggtj0p8z+3ghouk1tww3sj/v2oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h7xcbw0=</latexit> <latexit sha1_base64="y6qd7ctg3gwq+5slaxnajaose5o=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveyjelhozrxc3q0i6tad/baww45vgkhweyarndpfnmsrmgexfpluswwct1bvu7grk1m4o78omqsdyoo18vtvnzsvowt6xsyzsskggtj0p8z+3ghouk1tww3sj/v2oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h7xcbw0=</latexit> WHY SHOULD RANDOM PROJECTIONS EVEN WORK?! Like = repeating the experiment K times and = averaging y t [k] =x > t u k / p K where each u k [i] = random ± 1 = [ ] [ ] (y s [k] y t [k]) 2 = x > t u k x > s u k 2 /K
<latexit sha1_base64="3yp9ejgszib0dby1vx1w49yzwpo=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveygexprzmuvwt6tcokwa/wwsmoobxpichmgkzw6x5zektbnlxas7rmfnldqb7u4eznzudu/dpklewo1+vtfnysvowtyxsyzsskggtj0p8z93ghouk1tww3sj/v1oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h6bfbw0=</latexit> <latexit sha1_base64="3yp9ejgszib0dby1vx1w49yzwpo=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveygexprzmuvwt6tcokwa/wwsmoobxpichmgkzw6x5zektbnlxas7rmfnldqb7u4eznzudu/dpklewo1+vtfnysvowtyxsyzsskggtj0p8z93ghouk1tww3sj/v1oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h6bfbw0=</latexit> <latexit sha1_base64="3yp9ejgszib0dby1vx1w49yzwpo=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveygexprzmuvwt6tcokwa/wwsmoobxpichmgkzw6x5zektbnlxas7rmfnldqb7u4eznzudu/dpklewo1+vtfnysvowtyxsyzsskggtj0p8z93ghouk1tww3sj/v1oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h6bfbw0=</latexit> <latexit sha1_base64="3yp9ejgszib0dby1vx1w49yzwpo=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveygexprzmuvwt6tcokwa/wwsmoobxpichmgkzw6x5zektbnlxas7rmfnldqb7u4eznzudu/dpklewo1+vtfnysvowtyxsyzsskggtj0p8z93ghouk1tww3sj/v1oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h6bfbw0=</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="y6qd7ctg3gwq+5slaxnajaose5o=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveyjelhozrxc3q0i6tad/baww45vgkhweyarndpfnmsrmgexfpluswwct1bvu7grk1m4o78omqsdyoo18vtvnzsvowt6xsyzsskggtj0p8z+3ghouk1tww3sj/v2oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h7xcbw0=</latexit> <latexit sha1_base64="y6qd7ctg3gwq+5slaxnajaose5o=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveyjelhozrxc3q0i6tad/baww45vgkhweyarndpfnmsrmgexfpluswwct1bvu7grk1m4o78omqsdyoo18vtvnzsvowt6xsyzsskggtj0p8z+3ghouk1tww3sj/v2oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h7xcbw0=</latexit> <latexit sha1_base64="y6qd7ctg3gwq+5slaxnajaose5o=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveyjelhozrxc3q0i6tad/baww45vgkhweyarndpfnmsrmgexfpluswwct1bvu7grk1m4o78omqsdyoo18vtvnzsvowt6xsyzsskggtj0p8z+3ghouk1tww3sj/v2oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h7xcbw0=</latexit> <latexit sha1_base64="y6qd7ctg3gwq+5slaxnajaose5o=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveyjelhozrxc3q0i6tad/baww45vgkhweyarndpfnmsrmgexfpluswwct1bvu7grk1m4o78omqsdyoo18vtvnzsvowt6xsyzsskggtj0p8z+3ghouk1tww3sj/v2oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h7xcbw0=</latexit> WHY SHOULD RANDOM PROJECTIONS EVEN WORK?! Like = repeating the experiment K times and = averaging y t [k] =x > t u k / p K where each u k [i] = random ± 1 = [ ] [ ] (y s [k] y t [k]) 2 = x > t u k x > s u k 2 /K ky t y s k 2 2 = KX (y s [k] y t [k]) 2 = 1 K KX x > t u k x > s u k 2 k=1 k=1
<latexit sha1_base64="3yp9ejgszib0dby1vx1w49yzwpo=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveygexprzmuvwt6tcokwa/wwsmoobxpichmgkzw6x5zektbnlxas7rmfnldqb7u4eznzudu/dpklewo1+vtfnysvowtyxsyzsskggtj0p8z93ghouk1tww3sj/v1oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h6bfbw0=</latexit> <latexit sha1_base64="3yp9ejgszib0dby1vx1w49yzwpo=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveygexprzmuvwt6tcokwa/wwsmoobxpichmgkzw6x5zektbnlxas7rmfnldqb7u4eznzudu/dpklewo1+vtfnysvowtyxsyzsskggtj0p8z93ghouk1tww3sj/v1oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h6bfbw0=</latexit> <latexit sha1_base64="3yp9ejgszib0dby1vx1w49yzwpo=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveygexprzmuvwt6tcokwa/wwsmoobxpichmgkzw6x5zektbnlxas7rmfnldqb7u4eznzudu/dpklewo1+vtfnysvowtyxsyzsskggtj0p8z93ghouk1tww3sj/v1oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h6bfbw0=</latexit> <latexit sha1_base64="3yp9ejgszib0dby1vx1w49yzwpo=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveygexprzmuvwt6tcokwa/wwsmoobxpichmgkzw6x5zektbnlxas7rmfnldqb7u4eznzudu/dpklewo1+vtfnysvowtyxsyzsskggtj0p8z93ghouk1tww3sj/v1oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h6bfbw0=</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="piwkdkn7kx4tj0ajldd+otvtvhq=">aaacy3icdvfba9rafj5erxwrnq3erhi4cj7wtcl2exckxgqvfdy2kmqwmx3pdptj0pkx7rqyp9kbny/+dyfbrvtrbwmf3/e9n2++yepcwqrd755/6/adjbub9wzb9x883a52dk9s1rije1kvltnlhmvcltghrqwe1qafzgo8zezve/30mxqrqvijlwpmtdgvva6kieelwddyc5plebvouormcbyga+bsmz9iqurftnolc3gz2wtd7ftuuvxcthhjrrdfzmgqumgzddfdkvpnxlvaihja6q7iwgnpg2e4csoqcw494aevqgcod8d7fay8l1wn2bqo0+bbpk1ko7ekwqhrix7wlltckjifdoo4svglorfngdlyco02avczdfdcmvpik+nosbbib3a0qlu70jlz9vvbv7we/jcwnzspk1avdunyyqul8qyaqqaphkbkokri4ycqrrldqc6eezlctwxccncvhf+dk70rd0f8w/7w6m06jk32hd1jlxhnb+yivwphbmik++ftente4p30t/xd//gv1ffwpy/yh+u//qvm1bm9</latexit> <latexit sha1_base64="y6qd7ctg3gwq+5slaxnajaose5o=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveyjelhozrxc3q0i6tad/baww45vgkhweyarndpfnmsrmgexfpluswwct1bvu7grk1m4o78omqsdyoo18vtvnzsvowt6xsyzsskggtj0p8z+3ghouk1tww3sj/v2oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h7xcbw0=</latexit> <latexit sha1_base64="y6qd7ctg3gwq+5slaxnajaose5o=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveyjelhozrxc3q0i6tad/baww45vgkhweyarndpfnmsrmgexfpluswwct1bvu7grk1m4o78omqsdyoo18vtvnzsvowt6xsyzsskggtj0p8z+3ghouk1tww3sj/v2oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h7xcbw0=</latexit> <latexit sha1_base64="y6qd7ctg3gwq+5slaxnajaose5o=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveyjelhozrxc3q0i6tad/baww45vgkhweyarndpfnmsrmgexfpluswwct1bvu7grk1m4o78omqsdyoo18vtvnzsvowt6xsyzsskggtj0p8z+3ghouk1tww3sj/v2oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h7xcbw0=</latexit> <latexit sha1_base64="y6qd7ctg3gwq+5slaxnajaose5o=">aaadj3icrvllbhmxfpumrxjekszzwesvyojgr4imi0gvbjcykrjpk8wtkcfxjnz4hrlvvi2m8zds+bu2siaqlpktpe2oeh4sc65k6eicc30fdlrozygqb55/5eq16ze2brzu3b5z9157+/6rzusj5ejkojcnebdsq0yoqigwj4wrpi20pi6sl41+fcqnvxn2bhafdfi+y1ssbadhhdveyjelhozrxc3q0i6tad/baww45vgkhweyarndpfnmsrmgexfpluswwct1bvu7grk1m4o78omqsdyoo18vtvnzsvowt6xsyzsskggtj0p8z+3ghouk1tww3sj/v2oe7q7pekiopbgbdo85cwb/v9+jfuwbyuuhreiwbh9k01yuqcxaag7tmjicgoobuellusvkkwsuej6tywcznkobvbfvxomdx0xxnbt3msax7hpgxvnrf2nkne2z9letif+kjuui+0glsqiemyllobjughlcfbo8vuyk0ashuddk9yrfnlsfg/talbeen5piv4ojxpesln39rhpwyrwolfqqpuk7iki9dibeoum0qsj76733pnmf/xf+b/+l/3vp9b1vzgo0ef73h7xcbw0=</latexit> WHY SHOULD RANDOM PROJECTIONS EVEN WORK?! Like = repeating the experiment K times and = averaging y t [k] =x > t u k / p K where each u k [i] = random ± 1 = [ ] [ ] (y s [k] y t [k]) 2 = x > t u k x > s u k 2 /K ky t y s k 2 2 = KX (y s [k] y t [k]) 2 = 1 K KX x > t u k x > s u k 2 k=1 k=1 This is an average over K trials
WHY SHOULD RANDOM PROJECTIONS EVEN WORK?! For large K, not only true in expectation but also with high probability For any > 0, if K log (n ) 2, with probability 1 W, for all pairs of data points i, j {1,...,n}, over draw of 2 2 (1 ) y i y j 2 x i x j 2 (1 + ) y i y j 2
WHY SHOULD RANDOM PROJECTIONS EVEN WORK?! For large K, not only true in expectation but also with high probability For any > 0, if K log (n ) 2, with probability 1 W, for all pairs of data points i, j {1,...,n}, over draw of 2 2 (1 ) y i y j 2 x i x j 2 (1 + ) y i y j 2 Lets try on Matlab...
WHY SHOULD RANDOM PROJECTIONS EVEN WORK?! For large K, not only true in expectation but also with high probability For any > 0, if K log (n ) 2, with probability 1 W, for all pairs of data points i, j {1,...,n}, over draw of 2 2 (1 ) y i y j 2 x i x j 2 (1 + ) y i y j 2 Lets try on Matlab... This is called the Johnson-Lindenstrauss lemma or JL lemma for short.
WHY IS THIS SO RIDICULOUSLY MAGICAL? n= 1000 d = 1000
WHY IS THIS SO RIDICULOUSLY MAGICAL? n= 1000 d = 1000 If we take K = 69.1/ 2, with probability 0.99 distances are preserved to accuracy
WHY IS THIS SO RIDICULOUSLY MAGICAL? n= 1000 d = 10000 If we take K = 69.1/ 2, with probability 0.99 distances are preserved to accuracy
WHY IS THIS SO RIDICULOUSLY MAGICAL? n= 1000 d = 1000000 If we take K = 69.1/ 2, with probability 0.99 distances are preserved to accuracy
TWO VIEW DIMENSIONALITY REDUCTION Data comes in pairs (x 1, x 1 ),...,(x n, x n) where x t s are d dimensional and x t s are d dimensional Goal: Compress say view one into y 1,...,y n, that are K dimensional vectors Retain information redundant between the two views Eliminate noise specific to only one of the views
Canonical Correlation Analysis z y x
Canonical Correlation Analysis Age + Gender z y Candies per week x
EXAMPLE I: SPEECH RECOGNITION + Audio might have background sounds uncorrelated with video Video might have lighting changes uncorrelated with audio Redundant information between two views: the speech
EXAMPLE II: COMBINING FEATURE EXTRACTIONS Method A and Method B are both equally good feature extraction techniques Concatenating the two features blindly yields large dimensional feature vector with redundancy Applying techniques like CCA extracts the key information between the two methods Removes extra unwanted information
How do we get the right direction? (say K = 1)
How do we get the right direction? (say K = 1) Age + Gender Candies per week
WHICH DIRECTION TO PICK? View I View II
WHICH DIRECTION TO PICK? View I View II
WHICH DIRECTION TO PICK? View I View II
WHICH DIRECTION TO PICK? View I View II
WHICH DIRECTION TO PICK? PCA direction 0 0
WHICH DIRECTION TO PICK? Direction has large covariance
How do we pick the right direction to project to?