TTIC 31210: Advanced Natural Language Processing. Kevin Gimpel Spring Lecture 7: Structured Prediction 1

Transkrypt

1 TTIC 31210: Advanced Natural Language Processing Kevin Gimpel Spring 2019 Lecture 7: Structured Prediction 1 1

2 intro (1 lecture) Roadmap deep learning for NLP (5 lectures) structured prediction (4 lectures) introducing/formalizing structured prediction, categories of structures inference: dynamic programming, greedy algorithms, beam search inference with non-local features learning in structured prediction generative models, latent variables, unsupervised learning, variational autoencoders (2 lectures) Bayesian methods in NLP (2 lectures) Bayesian nonparametrics in NLP (2 lectures) review & other topics (1 lecture) 2

3 Assignments we will briefly go over Assignment 1 today Assignment 2 was posted last week, due May 1 st reminder: for those graduating this quarter, Assignment 5 is optional 3

4 What is Structured Prediction? 4

5 <latexit sha1_base64="u0xwf3yzhfhquldgyizfugbiywg=">aaa1bxicnvtbc922ezbtw6reknb61e6hrasjhuuqjhonmaseiwkrdiekqtpybqksaukchki8gqdpxqz7w/pp+tbx9qv/obsazyejgjiczdgigg8xi8xuynfrucrmyl29f91643vf/8epf/tmj9d/8tof/fwxb739yy9exvkqpg/zjodfbutqhgx0uwqyov8vnji0soixwevdbp9ysrlgexyifwu9s0mcsteliysq87dvpdvkkml04udembah2hjhizakhk9v+smre55wy4b6tj+loazm9bb+8owesllf8r54pufxsubn1al2p/a2fjypaubuhdzy1bityc5ph5e32pb6/na/u8+6zvepctdy/5lqemnrfwvhg59fdgw94901qvwjqwqqosuzmg0oxz9/a2nvd0/9epbhqpnywgt+js/ffu+wh+vhmdjmqkgfjvykevyrllmy0hrdlwutshhjynoknxljqtir1ptx3hburn445/avk56q7vjujbvikqaarb0isw0rxw2nprx/efaxrcglzuld0bhmpjl7aetexdhoivnabwk5a1m9cei4csvy3hqvmwjssris7w2lknquwzwt+ruxj8wehfn+bvomkvf81q8n8ynlypln/epxcnzltpzrkcwp5ursgzmahhglozaaof/bqd0macpe0xe3vruh/jbhrjcdqsfeheypl+weagiqxwuonfgakck+ulmwpvdhodynqu7sfp0ogzglzdhihpeqec5rsx3gkyydoa2rrmbhszzd/ezse8bqa6ptgrelmbucdmg4mb5mbryhrapbdhkkn5mhkvdhp6suk5wvkohbhoyixmuidjipkqkgwctqcj5ruz88jmvfmnbosnyegqpqp6hxyevcqyyhl1o8qd72amjl6ixmuxalq4eqmuktl0smjnxlbzrmwaumt/itekdwraqugglcwzwmla+mjhz1tuftutugcgysap+p9nbvm610lfmissyre1cfrjhmh1hyvq0fglowrf5x4wdvsanu+zdfjn8wnvp0dazbsrkotemvjcxfuy09rrzv1nsyets7nx0/pmu0seksefgaxi72n51zwhboqfl9p5s+z/feeg6zvnlwzht2aokob55cjyoooh4dxi0qvy/pr+mrzmlcvwdhpdtnjy3xkxgjak3hl8lqkuklwa6tculo4zlw3psfvsxh0jsmgl4eh0u7voo8eln7d5/tcbuxslivrueyv9odxebxvozlc6xwqui6evfpp8lzuqka2spnmzmrnhis2qqwwz9byriwf7lhuphzizdbig1+uuvbyoilnvfdtxl3miyusu22o8l23gswxqd5rjm+ha69sayjgfoebvxawgfvidda8ytcxtbp0buchjgugrgikuwtp19ud10i0fwhsmnciprbsqa4fopgojobft84ymos4ycxarcwydhbxkthlgoyuqlvzppuo9rff3jofasg/cnbzwbksliyg04goyysuwc5t0ngk+cqb785z9akzrg4qlbxbwradd/bc+9bb35naiqmlpcl6ja7uw+rljhfpamgssunojeshham41w5rypexti3nuhx5xv7mkpfznaetf+hq3yxeftz3druy0ccegsw1lezealyrr54pf9a8/j2fhtxx25jc6zb+4i4kqvnkpx10b6qhfk1sfph9eccqxzephp42i4dvq3wxvohstueapnlvecadc61psle29wybq6y9shnr4fhs9n+txbmngsiezdjshftecverauldo6efc2kmhanbenbsxd2jhzzywclbobiscjjc4gyczksycvc2mkgwssifkesu1q/ugeaxsytar6n1itt6msydshdwc4jymexzct8cytisqd1axrlnkqdlsynpwboxccifu91ltcgip0/gxy/bqeowcamczkoo+psnzy+iihccsvink3zecdpgoxmdqynuqkte/rtryvkt6u8jmoscygmy5xktnhhqaud0e6iphystluv0f+cckxnwgknlyedvdw0jyhrflh/mw7k0b/zhewlsbymsaaegbbh+7ydw140gqqfq6gxlql4g5fn7bkthuy1x6yal1znkdg32avc6gzlhpkln97wo5fkiwb8pih3q486pwdtdc5pe8cw2bbkxt72gqzmej30o5hyjkijvjgtbxu7ydvur5tymujzcets1l7ln/3urtciazuhes2xguu1lk9ivnjh1klryldaq80kh4kchmelt1edxcr/h2qb+5um6w6tjr2f3hyflcyjqim3frppdw+eo381alzf08u0rv1vn6izmgzduxdowlg9ysncftb6ektmqpxtbmajgpponfssnvdmxl7b/gkq4sjfqyq1ojwqwps8ytgfvglhx1so6wqpf7vssrfyxpveefdms2vdfjmdmxpoqipfjoeyt2hgyj72mqdcjhhrxs5ncuc1u7ozhxbxnqmi0zadlm7sge3xhdqv3q7yag0h/lq2vjzc/10dzqfhgdnkcimdvgkvtyxzcikcruzedhjzeststparubedrrycm0b2xdaypty0sicdrma7i2pgj4n+6aqpu6jr8cfd3uigdzvkolpohcuo2gokbdckj0mk+z+rn8wqg/tn75ud5jwee9bt9wemacsq66egbtkizwf8zs/6cnzhaf+yy8kzhs5xre5kv1splxcr0yqxoswokohic/ezhk7tne78siqqn8a40ybn1swnyaohrk6rb0r7wx5uqzzmscztl2uegacaeywj+5ip3hpepj1bhp4luc8flxnixh9d3iavr0/vnrkjjgl8otq4p7jhnkochutgynlkeupj6lzuj0fckkzlqmdsj2fxszfcw8oew2jgda0jewxsj2ghxgd7ailo9corme8qwn0huqkr01sh+fqnxtayt8tvjsgqwwahgc0orhcidiziaxgjstzt6kq8fzebgaeelyjnqj76gm11dpxkab1nlcvdr9uosv6hjipfvuqsdixjmyrpolebjultse5usvtnl2mrjnikpu2djcxwd0b6ckrjdejderygllybnoqzqj/lfjcgmhwnsjrnteu6a19g9jvjbvij7lljkjbdobc3riw9iu2glwsh1cgkma5zbzukquho6r7aa/9y2guea1vdawybkgzh5jsovdvclzuih2g0diou9a3c1nlg5runklaodbjvadxopbr91eczlgjhc1qjrhce1v9rltwfbox0inxmrprj2vtmvde67urcdmdbxcz5z63fgnmbqyq2hqvmdrshea6e7cgkrsc8paj4awsax3h43og1bmi4bjordcq1itznmhrinhsfmdmmkjnoi/ystiwuc5tolzwrybuzyqhbmppex1+0qeqmlpmu6iloietsmfodngip5pqvkgrjn1c9tulp45x0jvvuyxup6uxu8ptgxafpbn7klebgfq9xg9tbu47gdt1elgrjdun3dhqq3vzmnkexuewkkorluy3wjdmxttbvaa+pm9rneg6uspcadpr9nn8sd8ctdr7q3jqax8n4yejk1bam31mcptlf4lgpiekdvlfwvvbrgkxytjqh8zgo4mffagde2afshh1mfq20c5k+liw7eiyadictfzf2c2mzakjh0uer57knjsggkhw3qmwlcigqantsnb8+6gbfr3bwpcp52okfr7a6hh57sayvwdjfelnebj4rw+sdlicmi0uhcsjctcbwqxubjqgl9mkr5s0cvvv+yb6wg7qkniwk9rlep5koouvgzqnnjhuk9vr2uge2j/rt1sl/rantczst5/atwigpv2stwi+3hbuj3gaxftmhkkltrzrlngxng5lny2aam7hnvsupybhsot0ze5cqyirdphjunal6dgqtq5c2y5von9d7tz07glhkmpgozfegslp77tcq1toxjk8zj8lsjdjr8lqfivxogcaiiikiiqv1pmxx70oxrja28qwzj0orxlnp3p99a/cuzwkgazfqgompzrvu0xygzckvjtzq+2zc4b1qzvfbhs5r1chrt8t80f/wladvftsmc6yxwkyezkoqoq1b11alk40l80r5qs45jnwpzmheqqvmgz7oz8znzzccj27jxfgqumndwaysgbqfwqqaa/sf+ncpzb0xsyrfo/l+6lg0ylnefnnt06xui433e/z9wx8wv+zgbhhndcwre+djswbe583vic6w2c/crhuln0zmui6yuoo0w0cc4bwfbrgn7poxgvv6rszk60z8x5lr0cca7xa1g903klfeqr7my4sqb5y+rfqywkj+ktkf++pjo0r0we79mk09+nnyduyfdiwzdgp1lhctmu7dz9vrxfftt/ukuz0tplxhydc0qkecercwqdit3a9cd65++om38d/hyefwcnq1tnhb863anofbespzqhw4hd7pik7fez6aqsxvtmdup2sbu/v2ttajschvdl+i0yfdb56mnuf2dq9usaedky8dbe38ur5qhvvmxee6hupabiwe5xzsdeq4l1sreqlbpg7igexlobbzcml9xy/ievsfu5f9p48pqk1fmwdjhuaurvnxvvncazchc5hwasx98z2k1krxdnc9lfcymujcbbutjffpwpt3j3i2sum3lzuk8benoyv+dlp4twidk4c9xuhwr3okrfrpemtjdtlivwkq+2ly+yiruiuvk7sbpcvxtkonucrccyi/ww2/fdbcv4tgplymqt825+qusa54hibrboprbu/vfugaxvw9rfaypcb8yggiyxgc1rky hwqhfefztqvldvyfnx6mg6h3nwq9w8lzihvrs6tom8qw67tt24ghoc7b2vuiah67qetce4rvtcztu5r4ek3b+2ce81ru9ulvmdocp2sp117y9wzv/kfnxze+rybgmm9/4rzmj809k7i8v7u2o9nzhf39/45m/n38+8+bab9b+shz7bbt2p7vp1p6sha89xwtv/fpwv2/959z/3//f/v/f/+3932nog7caml+t9x7uv/n/igns0g==</latexit> <latexit sha1_base64="bn/zsaedvk1udtj0wpzbn/egj8m=">aaa1bnicnvtbc922ezz6tdvb0s70pdmpw0lto5zuhsdom0k9e8v27u4uvbxlxczkgpde4yhemwhwxmywf6y/pi99bv/6e7ol8bysacjj0ywtavh2svjslhyxbuxchnzf/9f6d777ve//4idv/wjjxz/56c9+/vy7v/hc5cup6fmwt3l+vuaetvhgn0sme/pvwsljg4r+gvw+wpyvp5qllmcnclhqs5tegruzkeioon9n/wq7s4rrhx95yukeyoofj0ksel6x7adetnhalrvqw/6uhrka1zv6w5ctkkl927vv+ythkzmfv4va/9jb9lmmqiftdvhjvctohdmnhv7eysfr8/uwxeuwrt0kwaaas0jbjo3txw98ftev9rz3x6jyjwspb2/4wz6vaud5xvnbm/t7++rhsz9gzcfmwvnzfp7oe+t+lidlsjopbn062i/kwuw4zgfc6w2/flqg4swj6sl8zisl4qxs819721ateeocw79meqq2s1grvihfggasdsdmnqx0tz2wcvzhwcwyopq0c3vh4zlxzo6hmxkr46cfzaefjoqmzpxcceeklgbyg71ucjaqybk8n5rk0rkm87to18acfbmwzvu1azlixvnzvzbmpyylzz71q8cp8c0z2a+lmd+ue0knzmgyyiym2migdgij9pikkxtn393xvhzww4z73gwenxgtmqvelnscieecl6byrmmmcpvqtsh4wcgd2pweoun7daimxiw2r4st3qviuutf9ognga+etkyagr0lcq7zm0nvgkmpje5nrczavhdqiengejc1wrwqkww5zjdczizk3oedkljocr6gho9wmcmqlymyykzkiojmeumjeeblqfkafh/jwdnbwhnkked9of4hfwmtshy+ahg/+tiros2jm5pnqzqmhapqpk+8kpmy1juf0jhllzrcyk9ianetevaapgkfc5puvpo6wttvpk3rbomimleifkba37tnttkxtceksazuqx0srtifrmfxnrrgzfqyekona7wjjvdnxxyxfsn46tprgqqp5towdvwqlbrhnfa0crxtb3pk7e5+dvz4rnhgiktgxbtyotp4dmyjqzgfyq+ebvmcxrnjomz61qmjo7pxe0s9ehi1kqkievjoalspenx+ty80s+o6entuq33+2bi/itgc1nr8iqyekrsa9daaj4q+ngphlf/zqxigvwuc6cvwubrdpc5hr/b+4xm6rjzhfquykajjarqzgpzkl7f8idvksfxhij7tvhg+v0btw/ksczlcbonvw9uy63nlwrbmqvyykn4+exjhscevtyhigbhujr6jke8mk5eytsl2ndmumwzw+dspanknw7uxisexuj/aob4yk6wqerp4nks4goyjekoqennfefckpaj158vqjgsbmutadk1irdny1hsxfw4oqrlh8cjbhpycb7lmg7aew9n2ijpmgcputxhjnqfe0chbg0oviaxlh9lqbloclcuydalzywsnost5ggjboyc4+okcwzowreakqv0vkgk1fg8tvg+8+e0bkdkywc+rw+x2pusyybarjhgmlzyarriyqjcmc+uudxpnsd/yhf+fv+z+qh6zwrmyfrmo2r3guwv3ulonhxfiembs387icwc3e+dxfahn41vzncvtu40tmg7tc+jefpypcnzf+6pysnuitr3tawmucnit6sfhow74tlp7zyfkvnsitjb1ggm2onf2ihhva3o0nwdajz4/ojxemvybsjpnjbongexwumjpwyzmlsihcyekachfhdshsnpourmzschaplgwaerkek5acjrnyjralhhsfgmvjkb0o5kc0vrlxqp0km4r8g20jmlhj5a1cx7och5bdo5pjuzpwbrlaqgfocizirpomwk4buarbhatekpzuqyr6fzjsptpc3ainmzcyhhyuzeqsfrbdbrwwbb3bjqhahngygdqmdjrtz2y0g8/kig7xeulgzzmcgqcq1r2ijsowu2o9ky0tzcw3aqm/hduydottmh4crno5zy5qynii/2pcsoh/pgnlcqftwbimiumhd06sb0y9qijjpgyco11scubjgxql20tnkr5ytv84wqgxlnjxhvgz1s2zfz49lbeivranoandbwfettpawtnchmaaibndlqu9rhxmdwd6ktfwcqyquqkimw92ngfq6f6w5tlfmpcwgkj9i3/7hdvhuecqanra44nxkp1erlzlqujfl0hu9vabsb5sbqwdfu4jrogvvn/iavy3zrdd2gt6sjktiugzeletby10h16+og5+lejzlu0xuxj1p92gzqlady1byeujnujnixkya0vswqmxdlfajygmencuc/scwvu7bxmlwypllwurfiduwmjtc+tjfpamxb2tewwtpdyvyuvdchptu8mcwu2wzbelzkym4confhmci77fgas7gmvaypmgpfv5diud1wzqzvtarfxosasdtng4syo2htdof3a7yan0hbar2trxcp939vrfhcwku2umdlam1bxxdihj8hpze7kgjetscxmaxmzgztgzcuwawrpbcpry00jezjgzla7oaz+muihtvcwk7ogfzlsjw74semoquvesykqpubkckpwmksy/xl5uwy5qd/8xttbxuk5yt1df2aktgkxhmpgic3ahmzn9qyvwbco8iu5jjxr6bxx5k70u5vycbe6txihwowqgyi8cj9p6nto6csvsgjxy4wbddhuxxkarhra6rq6r7s/7ee2xgmeyzx9u+yjcac4xzkyj5nkpedn2rnt4bka9flxkohm9dffdvjx+vtumzhakmyvq827intmm8pzugsene0mmz6mzuv+gcqkyvqodcz1gby3g8m1rowxlabg0dcyxyd1gbzqdlcvjojer7wc86sc1r2smxaxvxwar9xoby3xrfxdiaisdbyc2ijcdcfcyii0hjgizd2lrsrbcbmjced5gtks6qov0vzhv0+o1theutj1vi2s5bvkmlhcpstpujk8qzkue7vrkjsv5e6vtnx0mrzigkdq2t5jwmj1z6spodpoj9slz2vaygfast6d+lluaqoqwdeis3lgu6r79mvexznwjpxalpsmcdkvcnvrytit0gtaqnzihuka6zp1uemshism7gs88i+fvey5udu0ybaaaxzlvotctcndwylwgezrn+ja3ypsnqdwxtgaauvaifgvxu1eqtfpdttlihy2r5vgdefq/bvwwfpp5kinygfmspmuvcvufa3jrs7tabbmzzj31losmbcypgjbswdushsa4udihkjqbmzbq4q3tqdxhyfhjv5diixdoblrp1xbanmeo4fmuvcam8ogmum0ws9ky+iysilwwbkkvpehhrofm0/89uwbsm7ckk+plukcum4gm9xjkfzgtlkhszb0c9vjk/48zknnwe+vxjeszuqmt2lmhpdk5qdcaq5u8qa3qr2862js0e1eszbkmh5nrwhhbw/mgxqrxu6iqli222jdkh3bbfmv8pq4qx2u50aje6/hqnlh881ydzxh56vuran5piwhhqrcbwv6psvgkl0mhvuymd5adwvdaxutycrhsxkyh+miflysducffsdn0wdw0uc7koldybgdwqhod9jyewm3l5yjibomfr55ltvoigygwnejvlyih6icts3w+evdbvb9aaffo5ynnvhwzktl6lchy/ckjvmemetf4bmxtd7hamk6uhsyimlfbak3uhfygrbglxzz3ilgw++x7au7oavoyajql0f8momsu2tunhduwqfgv2+p+2b7qn9unfzhota2tznxbt+kffh4rtap9xjludnqbrdblxmqmtt1slpgwda8idm6baa4hxvmo5wkfsge0zmzaqmjtdheikddk6rhody5dgmzvil+a71f27mdgusgg4/sfg8iq7vxdqnarxpb+jrxmmxwir0bt/uryowuyyyimcgssldpw3z1pntjamkbxzd7obtplfxm/d81co/snaasfgkfmp7ywug2zafklllryko+byye3ghlhb9k6lxghbj+s8wx/w9mefdmt80wx3qpxutipaqz07b1dvwy0lgyt5sn6pzdwp3oheasumqe4el8ztx0dinw4pbmhquhndyvbmocqpgppqo4sf+ad/1y1huzrfi0ku+phkulnetmmt01xeo+0ng/yd+39qhza27eftdcx7y2dz6wboz13lyd6gyb/szmutd6w2su64ijpu4zdmqzxvvpftvornmhcabfzek2bsr3nbkvdazzdlcz2h0jgeul5mu8tkh6yoldpculloqnkl2wr960o0qf7n0l09qdn23v0byajnggyazeeqjv0334uxqt6376qz5kqmfclcss7jpgnyhdahy2fxu1+1xo7tvpp/e2/ls8+dxetq+wcxtrvlxbm3bltzklqa63wwdzxokfhkdmksub05lbkvng7r59ru0iusjbw69ododqmydjzzg93asb0sfgyumaqro/ri+urz1tv7ioxziwayvhoqc7gxlujdvqxskwj4m4ifmzqmw8qlrf14u43n7b7uxg6enh1zavgybxd3pxoqz4qizhdzjqouyrgjl++fzsatiro7heyiuztcehgo1wkqtnlcr7ezxb2flbli1f+jcg0ei/hyn8w4golcoelisr1zlsvqo0iouku2yxqjsi/ts5fmhi3mkxfdpnnpi4ptftxbkhbmgf4yzf3h+hx0jwuysg1w+bc3xkwfc8dsa1d8b1d/brpisnyq4eiivmpma+sddrzqybagnze2+n+aejliqick5p1fyuq5pm41rq9y7mracref7kqi1p9wltevydd2p78ddnovg7x3tad1z1pwhc8dqphr2qcg+csti/7gce6ooeqf1zo8chbewga97erzo/ykmr5vdvmaygzw/8o5mr+scy9scxd/dg+3ujv7+/+cmfmz+fewvt12u/x7u1nlr709ona0/wjteer4xr/1z/9/p/1v/7/v/u/ereb+79vko/s97q/hkt93pv1v8brphtaq==</latexit> Classifiers a function from inputs x to outputs y one simple type of classifier: for any input x, assign a score to each output y, parameterized by parameters w: classify by choosing highest-scoring output: 5

6 Notation a vector entry i in the vector a matrix entry (i,j) in the matrix a structured object item i in the structured object 6

7 <latexit sha1_base64="u0xwf3yzhfhquldgyizfugbiywg=">aaa1bxicnvtbc922ezbtw6reknb61e6hrasjhuuqjhonmaseiwkrdiekqtpybqksaukchki8gqdpxqz7w/pp+tbx9qv/obsazyejgjiczdgigg8xi8xuynfrucrmyl29f91643vf/8epf/tmj9d/8tof/fwxb739yy9exvkqpg/zjodfbutqhgx0uwqyov8vnji0soixwevdbp9ysrlgexyifwu9s0mcsteliysq87dvpdvkkml04udembah2hjhizakhk9v+smre55wy4b6tj+loazm9bb+8owesllf8r54pufxsubn1al2p/a2fjypaubuhdzy1bityc5ph5e32pb6/na/u8+6zvepctdy/5lqemnrfwvhg59fdgw94901qvwjqwqqosuzmg0oxz9/a2nvd0/9epbhqpnywgt+js/ffu+wh+vhmdjmqkgfjvykevyrllmy0hrdlwutshhjynoknxljqtir1ptx3hburn445/avk56q7vjujbvikqaarb0isw0rxw2nprx/efaxrcglzuld0bhmpjl7aetexdhoivnabwk5a1m9cei4csvy3hqvmwjssris7w2lknquwzwt+ruxj8wehfn+bvomkvf81q8n8ynlypln/epxcnzltpzrkcwp5ursgzmahhglozaaof/bqd0macpe0xe3vruh/jbhrjcdqsfeheypl+weagiqxwuonfgakck+ulmwpvdhodynqu7sfp0ogzglzdhihpeqec5rsx3gkyydoa2rrmbhszzd/ezse8bqa6ptgrelmbucdmg4mb5mbryhrapbdhkkn5mhkvdhp6suk5wvkohbhoyixmuidjipkqkgwctqcj5ruz88jmvfmnbosnyegqpqp6hxyevcqyyhl1o8qd72amjl6ixmuxalq4eqmuktl0smjnxlbzrmwaumt/itekdwraqugglcwzwmla+mjhz1tuftutugcgysap+p9nbvm610lfmissyre1cfrjhmh1hyvq0fglowrf5x4wdvsanu+zdfjn8wnvp0dazbsrkotemvjcxfuy09rrzv1nsyets7nx0/pmu0seksefgaxi72n51zwhboqfl9p5s+z/feeg6zvnlwzht2aokob55cjyoooh4dxi0qvy/pr+mrzmlcvwdhpdtnjy3xkxgjak3hl8lqkuklwa6tculo4zlw3psfvsxh0jsmgl4eh0u7voo8eln7d5/tcbuxslivrueyv9odxebxvozlc6xwqui6evfpp8lzuqka2spnmzmrnhis2qqwwz9byriwf7lhuphzizdbig1+uuvbyoilnvfdtxl3miyusu22o8l23gswxqd5rjm+ha69sayjgfoebvxawgfvidda8ytcxtbp0buchjgugrgikuwtp19ud10i0fwhsmnciprbsqa4fopgojobft84ymos4ycxarcwydhbxkthlgoyuqlvzppuo9rff3jofasg/cnbzwbksliyg04goyysuwc5t0ngk+cqb785z9akzrg4qlbxbwradd/bc+9bb35naiqmlpcl6ja7uw+rljhfpamgssunojeshham41w5rypexti3nuhx5xv7mkpfznaetf+hq3yxeftz3druy0ccegsw1lezealyrr54pf9a8/j2fhtxx25jc6zb+4i4kqvnkpx10b6qhfk1sfph9eccqxzephp42i4dvq3wxvohstueapnlvecadc61psle29wybq6y9shnr4fhs9n+txbmngsiezdjshftecverauldo6efc2kmhanbenbsxd2jhzzywclbobiscjjc4gyczksycvc2mkgwssifkesu1q/ugeaxsytar6n1itt6msydshdwc4jymexzct8cytisqd1axrlnkqdlsynpwboxccifu91ltcgip0/gxy/bqeowcamczkoo+psnzy+iihccsvink3zecdpgoxmdqynuqkte/rtryvkt6u8jmoscygmy5xktnhhqaud0e6iphystluv0f+cckxnwgknlyedvdw0jyhrflh/mw7k0b/zhewlsbymsaaegbbh+7ydw140gqqfq6gxlql4g5fn7bkthuy1x6yal1znkdg32avc6gzlhpkln97wo5fkiwb8pih3q486pwdtdc5pe8cw2bbkxt72gqzmej30o5hyjkijvjgtbxu7ydvur5tymujzcets1l7ln/3urtciazuhes2xguu1lk9ivnjh1klryldaq80kh4kchmelt1edxcr/h2qb+5um6w6tjr2f3hyflcyjqim3frppdw+eo381alzf08u0rv1vn6izmgzduxdowlg9ysncftb6ektmqpxtbmajgpponfssnvdmxl7b/gkq4sjfqyq1ojwqwps8ytgfvglhx1so6wqpf7vssrfyxpveefdms2vdfjmdmxpoqipfjoeyt2hgyj72mqdcjhhrxs5ncuc1u7ozhxbxnqmi0zadlm7sge3xhdqv3q7yag0h/lq2vjzc/10dzqfhgdnkcimdvgkvtyxzcikcruzedhjzeststparubedrrycm0b2xdaypty0sicdrma7i2pgj4n+6aqpu6jr8cfd3uigdzvkolpohcuo2gokbdckj0mk+z+rn8wqg/tn75ud5jwee9bt9wemacsq66egbtkizwf8zs/6cnzhaf+yy8kzhs5xre5kv1splxcr0yqxoswokohic/ezhk7tne78siqqn8a40ybn1swnyaohrk6rb0r7wx5uqzzmscztl2uegacaeywj+5ip3hpepj1bhp4luc8flxnixh9d3iavr0/vnrkjjgl8otq4p7jhnkochutgynlkeupj6lzuj0fckkzlqmdsj2fxszfcw8oew2jgda0jewxsj2ghxgd7ailo9corme8qwn0huqkr01sh+fqnxtayt8tvjsgqwwahgc0orhcidiziaxgjstzt6kq8fzebgaeelyjnqj76gm11dpxkab1nlcvdr9uosv6hjipfvuqsdixjmyrpolebjultse5usvtnl2mrjnikpu2djcxwd0b6ckrjdejderygllybnoqzqj/lfjcgmhwnsjrnteu6a19g9jvjbvij7lljkjbdobc3riw9iu2glwsh1cgkma5zbzukquho6r7aa/9y2guea1vdawybkgzh5jsovdvclzuih2g0diou9a3c1nlg5runklaodbjvadxopbr91eczlgjhc1qjrhce1v9rltwfbox0inxmrprj2vtmvde67urcdmdbxcz5z63fgnmbqyq2hqvmdrshea6e7cgkrsc8paj4awsax3h43og1bmi4bjordcq1itznmhrinhsfmdmmkjnoi/ystiwuc5tolzwrybuzyqhbmppex1+0qeqmlpmu6iloietsmfodngip5pqvkgrjn1c9tulp45x0jvvuyxup6uxu8ptgxafpbn7klebgfq9xg9tbu47gdt1elgrjdun3dhqq3vzmnkexuewkkorluy3wjdmxttbvaa+pm9rneg6uspcadpr9nn8sd8ctdr7q3jqax8n4yejk1bam31mcptlf4lgpiekdvlfwvvbrgkxytjqh8zgo4mffagde2afshh1mfq20c5k+liw7eiyadictfzf2c2mzakjh0uer57knjsggkhw3qmwlcigqantsnb8+6gbfr3bwpcp52okfr7a6hh57sayvwdjfelnebj4rw+sdlicmi0uhcsjctcbwqxubjqgl9mkr5s0cvvv+yb6wg7qkniwk9rlep5koouvgzqnnjhuk9vr2uge2j/rt1sl/rantczst5/atwigpv2stwi+3hbuj3gaxftmhkkltrzrlngxng5lny2aam7hnvsupybhsot0ze5cqyirdphjunal6dgqtq5c2y5von9d7tz07glhkmpgozfegslp77tcq1toxjk8zj8lsjdjr8lqfivxogcaiiikiiqv1pmxx70oxrja28qwzj0orxlnp3p99a/cuzwkgazfqgompzrvu0xygzckvjtzq+2zc4b1qzvfbhs5r1chrt8t80f/wladvftsmc6yxwkyezkoqoq1b11alk40l80r5qs45jnwpzmheqqvmgz7oz8znzzccj27jxfgqumndwaysgbqfwqqaa/sf+ncpzb0xsyrfo/l+6lg0ylnefnnt06xui433e/z9wx8wv+zgbhhndcwre+djswbe583vic6w2c/crhuln0zmui6yuoo0w0cc4bwfbrgn7poxgvv6rszk60z8x5lr0cca7xa1g903klfeqr7my4sqb5y+rfqywkj+ktkf++pjo0r0we79mk09+nnyduyfdiwzdgp1lhctmu7dz9vrxfftt/ukuz0tplxhydc0qkecercwqdit3a9cd65++om38d/hyefwcnq1tnhb863anofbespzqhw4hd7pik7fez6aqsxvtmdup2sbu/v2ttajschvdl+i0yfdb56mnuf2dq9usaedky8dbe38ur5qhvvmxee6hupabiwe5xzsdeq4l1sreqlbpg7igexlobbzcml9xy/ievsfu5f9p48pqk1fmwdjhuaurvnxvvncazchc5hwasx98z2k1krxdnc9lfcymujcbbutjffpwpt3j3i2sum3lzuk8benoyv+dlp4twidk4c9xuhwr3okrfrpemtjdtlivwkq+2ly+yiruiuvk7sbpcvxtkonucrccyi/ww2/fdbcv4tgplymqt825+qusa54hibrboprbu/vfugaxvw9rfaypcb8yggiyxgc1rky hwqhfefztqvldvyfnx6mg6h3nwq9w8lzihvrs6tom8qw67tt24ghoc7b2vuiah67qetce4rvtcztu5r4ek3b+2ce81ru9ulvmdocp2sp117y9wzv/kfnxze+rybgmm9/4rzmj809k7i8v7u2o9nzhf39/45m/n38+8+bab9b+shz7bbt2p7vp1p6sha89xwtv/fpwv2/959z/3//f/v/f/+3932nog7caml+t9x7uv/n/igns0g==</latexit> <latexit sha1_base64="lvj5f1yayqj0pimo0ef8s222bnc=">aaa1fxicnvtbc922ezz6tdvb0j72ha2kqr1lqo4tj5mknoliq3yniqracm6irafjhb5ivbkaz8um+wp7d/ov+tq+tlsazyejgjiczdgigg8xi8xuynfrucrmyp39f63/4ic/+vfpfvrwzzz+/otf/urxb7/zmy9fxvkqvgjzjodfb0tqhgx0hwqyov8xnji0sohxwdujbp9qsrlgexyqfwu9t0mcsteliysqi3fww+0skuyffeyfcrgcjreecelc+ma2nxi54wm1bkjv+fmaympe7+gpx06ojpvd76hnex6nzh5rlwr/e2/bz5mibk7vuy1vltsr5px2ehmlha/p73t2rvsw9ztlsks0bokvn7y3tne98cvlv/a73j2jyp8q5axewz6vaud5xsxbm/t7++rhsz9gzcfmwvnzcvhoe+t+lidlsjopdha22i/keuw4zgfc6w2/flqg4rwj6rl8zisl4rxs5lb721ateeocw79meqq2s1grvihfggasnslmnqx0tz2vcvzrecwyopq0c3vh4zlxzo6hbxkr46cfzaefjoqmzpxcceeklgcbg71ucraxybk8n5rk0rkm87to18acfbmwzvu1azlixvnzvzbmpyylzz71q8cp8c0z2a+lmd+ue0knzmgmyiym2mig/gij9pikkxtn393xvhzww4z73gwenxgtmqvelnscieecl6byrmmmcpvqtseuwt+d2pweoun7daimxiw2r4st3qviuutf9ofngb6etkyagr0lcq7zm0nvg0mpje5nrczavhdqiengejc1wrwqkww5zjdczozk3oedkvjocr6gho9wmcmqlymyykzkiojmeumjee7lqfkifh/jwdfbwhnkked9of4hfwmtshy+apgw+ssros2jm5pnqzqmhapqpk+8kpmy1juf0jhllzrcyk9iameuevaapgkfc5puvpo6wttvpk3rbomimleifkba33tgttkxtcfcsazuqx0srtifrmfxnrrgzfqyekona7wjjvdnxxuxfsn46rprocqo5towdvwqlbrhnfa0crwzb3pk7e5+fvlkvnhgiktgxbtyot54fm4jqzgfyq+ebfmcxrnjomz61ugx0dnrcak+fxo9kqkienx4papupb64oueapxfdhr73uf88scrpyoygtr6/cktnjc1gebtgcagp58bxy3/+qorbb5ka+gp8fo1qqfpw1d4/fe7h1ebiyluwbwfctxcwg1/5mpyvsvojietixkedcs/ncmhqk59ltkbyylbqq1pmfw4py8jcyb5dxc8nqu4onvjlfgupi5baxpc05b1hmllgntmujtt1k8gsn+yrtfp0upyimeqj6hmy1anjhvulqgppkwgxwzxbbwgsyroi4oqkflx+ffndcyfacx3irgmjaablmulyryodam5yfosgd0uog1xmrvic4wx7er2zjgemj7wxz1pv+odg0zfxkilyj201gynbymimoofkmbjgneu8dygtnemc/pacwzowreakqv0vkhg1fo8svo+8+d0bkdkywc+ro+xupusyybarjhgmlzyarriyqjcmc+uudxpnsd/yhf9fvozhqh6zwrmyfrmo2t3guwx3ulonhxfiembs387ikwc3e+dxfahn4zvzncvdu40tmg7tc+jefpypcnzf+6pysnuitr3tawmucnit6tfrow74tlp7zufkvnsitjb1ggm2onf2ihhva3o0nwdah18ch50stfsn2zlzkolgqcyrxbtnlrozfptqurnstjbiwp0qjg0n3dmzwgqthywyacilcsctbmomzeoglwhpi4ekeyp6huloaf3yvgf6fbci+dzak7sje8vabq9noy8gb8c0e/mnlioladqpujtzea20mdscaqava4twpnw5xioidp5k2p20btgegzmu5djsqevvwpogbgorlit7ps1dgdxnsfe1gjzqpk5m6lcfl5ddrvksdesmbdkovso7xr0hrxzheyoawkjyblvg/wsqmj0jjzs8ggz0asucoetxxf7hujfdf8wtfpxcwjakmrigwh4f2a4me9eeenwmhca6poinrja1ybywgtv8uwq+ddvd4txkrwqjsy0bzi58elfvrcotgvdtbt4ppeosgbuzudxbamnmoyox+8rrmjpbdnlvygkzicvsrbyeboyc1vc9ycplcmuhllfr+5z/9ru3wiao1itonccgltw6pjj52ovri66d3wqtnpn8jaoyhi3crg0qq/x/kaxsb5quo7swdbry2xvqsisijt5a6d47onyb/twowrvnf9my9b/dom5imnvnqaaly/widqtzwotlljkjcu4racdotdp37rn0qpkbew3zy0gksxcfedbgveji7fmk4xtwpov9yzmse6t81uolxcgl7rndxjnnlg3xzq7gzkeddratnms+hrkr+9irgaqt4gv1tttfxdxs6s50wmvndoheqw64uludnl13al9xo2adtb3wm9pa8xd/d32ubyrlzjnrjazwplu8tczbcxiamrm5ygrlllpzwkbmxg4y2qpsgtlt28j6wnping4yme0ujogfdvqhezzsiq7bnw57oxs+7jcd6jp1rkbqd5as3cg8isnmf0befenu0de/13aqv3powe/xn5uirucshxoysis2h/g5pesr8kudfgmocc8aoscvusv9bqvcqqxok1yifqnkbiiv3gcaorvzovgrkkdcgongazzvv5wmqx6wuq4eku1ve7gn8zjhmk9ftnkcagdusyzsy6zyz3mtdm57ec5avxy8zcbz/xv5c1a8prt/biqwivhlavo+4h7tjhiwlhkhz5njjcej87o/bgljmpudhus9hsxtxnadk3smi4exnizsmug9houaa+yriaa3k6/apklgdyfknkrmbx3gazv6qwm8evc3pkgeg4uatqaq3ykwgcktyywowu+pk/fwxg5jhhi+odsr+uhrtnxr9zojdtsxlhqtvamkey2sdbbxkuk6lcrvq6sbrg2uje0luvmlbtw9jk2y4jgatnessfjdgekjqi7tcxxp2rqw2afwks+gfpzbwibqvjtcup7rlumefrnrv461sswwyyzdqnafwt66svsenig2kb1sh5bmumyd1jaeiagp+wip/gnhr3kova0ngaraml2z76jq7xbxviichthajbjrtwpb54gvvzsgwdowsfslcturavqza82yiixnusuyxrbuf61v1ny2odip8rm5uizlbzfvreu4q3m7gavzmesdtryl5gugkdp2ljg77b5aobcyh5aaafp2moktlwh84efxo9cqcomqaebukdcw2jtb6cfz7hxazdcozdin8lpsmrosbkjvvq6cvwsiow7bzbn/fdgmkpuw5foqizgnrrpbthvyhj2yu1zieitdxpxeij9pcti51lml16wkm7njuxotbzy5/slxmonbvmftuafvoho7chnrkiu5jn/zuub425t5bl1eszooejbttlixzcc2wvqfvd5uap/rovdc3ms4upbrflpcnuzyxap762ged+obislx25p+twmyshedxz4mpg9q3vr3wkulmsnruobmxzichxerhxhhh0hz9jlvnnauzppqmu5amalanbrcxnjnpwuciizjn0eeyzy6ihuocnwils3ciaqgbbn1/vq4g3wf28h3oodpjxyc2+os+u3bmrxiyrrbzhoreezsrxe4gjgulh0mihaxgcctbgswiczyi0wwtwj41vsl+8ju6cp6sijusxcfzqlkfbk1j5y7vijy79vjvtk+6rdbj/+xdrtn7uyc27dibt5eku1qvdwsbo56g8t25yrkjk290prxljvvslaumgfu4z75hqecrbdn9mxmqeiiew6r4lhxpooxqe0oxdosb6lfqo83du5g4jlb4km03xvc6u61w6hw61wwvme8jrovsg/a034kcjvlgcmijoqelntzmf89f12ywtrgl8w+dkv6zt1z//cn3lsyp4gkruibpj+2vrgtc4cspvq08qjv2wmbd0a5xwczoq9rh6zflfnf/ztthrz5btpmsv5qmreyj0lmngxdq5wsnjbme+wpoucwvj86hxgrrppnedqfgted3qqcucvzyufijxufm7iaanxqqqio7bfgqz+w9cyskrsnyvutx9ii5xjtte9nl7qpnn5p2pdtfcd8hhuxxq3xma/tny8lgxmdn1cnottmpwuz7rxemjnrumbcqdmma3gg1366xta6g15ohogxmzctg/edz0zfa2u2w9umdt9irnkj+tive6oewpoqackci/qpzbfsqyfuknehew/ctpbgz9s7nbfqmc4wjnvbqrrqug8/v6913u8/1znm9uyycowfxdoobnfiwmkmyq92vwrdvf7pj97gf48nn9vl6duyyw/nt2p7dm7pkwcb1ef2+cclop2j8nrmzxlzonmnjqvc3bevtulecnl3+bwndhx682ssobk3e3vdohhmerwgaoi39yxyqgfqctfxgac2ype452bnumk9pfrns9jycrahzcodtzhxsov7ehhx2y/yvrw8e3jybfmaarj3mhctyoqvyhlegamdw7qkkomp7ywljr22g5ulvjbjfbii2ggluxrworw/wbovlb9t2vkej2kilfqlmmk/fehyoezpqrj6nsnlpuqdwe6yqxarsopyz5krx4zelxxzod3kgyljgj3djrfkepwzbvjlwxh6jqq3twjq/by5v6emdcxjafxzb1x3z7/uq9io5uohsslsa+ydcxndxrcolpc993aafzrkqiic4jpwze+6om0+zgrv72joe7cc50uu1jjwnzwv59vjp7yhd3oeg7xzrqf8afxxgyyur51a2jeq9ycyhporduazruib2jw3a3zcvm645u3dqvn7nlpqfl8hw2dy/my/mhmzfyjjf3x5f2+0vzf6+/ubn/65+fozt9z+t/ahtttro7up1z5de7p2svzilvz/5/q/1/+z/t/3//dg+8hogz0n/cf6q/pbtd7pgw//d9gm8z4=</latexit> Modeling, Inference, Learning inference: solve _ modeling: define score function learning: choose _ 7

8 Applications of our Classifier Framework task input (x) output (y) output space ( ) size of text classification a sentence gold standard label for x pre-defined, small label set (e.g., {positive, negative}) 2-10 word sense disambiguation instance of a particular word (e.g., bass) with its context gold standard word sense of target word pre-defined sense inventory from WordNet for bass 2-30 learning skipgram word embeddings instance of a word in a corpus a word in the context of x in a corpus vocabulary V part-of-speech tagging a sentence gold standard part-of-speech tags for x all possible part-ofspeech tag sequences with same length as x P x 8

9 Applications of our Classifier Framework task input (x) output (y) output space ( ) size of text classification word sense disambiguation learning skipgram word embeddings part-of-speech tagging a sentence instance of a particular word (e.g., bass) with its context instance of a word in a corpus a sentence gold standard label for x gold standard word sense of target word pre-defined, small label set (e.g., {positive, negative}) pre-defined sense inventory from WordNet for bass exponential in size of input! a word in the structured prediction context of x in vocabulary V a corpus gold standard part-of-speech tags for x all possible part-ofspeech tag sequences with same length as x P x 9

10 Applications of Classifier Framework (continued) task input (x) output (y) output space ( ) size of named entity recognition a sentence gold standard named entity labels for x (BIO tags) all possible BIO label sequences with same length as x P x constituency parsing a sentence gold standard constituent parse (labeled bracketing) of x all possible labeled bracketings of x exponential in length of x (Catalan number) dependency parsing a sentence gold standard dependency parse (labeled directed spanning tree) of x all possible labeled directed spanning trees of x exponential in length of x machine translation a sentence a translation of x all possible translations of x potentially infinite 10

11 <latexit sha1_base64="u0xwf3yzhfhquldgyizfugbiywg=">aaa1bxicnvtbc922ezbtw6reknb61e6hrasjhuuqjhonmaseiwkrdiekqtpybqksaukchki8gqdpxqz7w/pp+tbx9qv/obsazyejgjiczdgigg8xi8xuynfrucrmyl29f91643vf/8epf/tmj9d/8tof/fwxb739yy9exvkqpg/zjodfbutqhgx0uwqyov8vnji0soixwevdbp9ysrlgexyifwu9s0mcsteliysq87dvpdvkkml04udembah2hjhizakhk9v+smre55wy4b6tj+loazm9bb+8owesllf8r54pufxsubn1al2p/a2fjypaubuhdzy1bityc5ph5e32pb6/na/u8+6zvepctdy/5lqemnrfwvhg59fdgw94901qvwjqwqqosuzmg0oxz9/a2nvd0/9epbhqpnywgt+js/ffu+wh+vhmdjmqkgfjvykevyrllmy0hrdlwutshhjynoknxljqtir1ptx3hburn445/avk56q7vjujbvikqaarb0isw0rxw2nprx/efaxrcglzuld0bhmpjl7aetexdhoivnabwk5a1m9cei4csvy3hqvmwjssris7w2lknquwzwt+ruxj8wehfn+bvomkvf81q8n8ynlypln/epxcnzltpzrkcwp5ursgzmahhglozaaof/bqd0macpe0xe3vruh/jbhrjcdqsfeheypl+weagiqxwuonfgakck+ulmwpvdhodynqu7sfp0ogzglzdhihpeqec5rsx3gkyydoa2rrmbhszzd/ezse8bqa6ptgrelmbucdmg4mb5mbryhrapbdhkkn5mhkvdhp6suk5wvkohbhoyixmuidjipkqkgwctqcj5ruz88jmvfmnbosnyegqpqp6hxyevcqyyhl1o8qd72amjl6ixmuxalq4eqmuktl0smjnxlbzrmwaumt/itekdwraqugglcwzwmla+mjhz1tuftutugcgysap+p9nbvm610lfmissyre1cfrjhmh1hyvq0fglowrf5x4wdvsanu+zdfjn8wnvp0dazbsrkotemvjcxfuy09rrzv1nsyets7nx0/pmu0seksefgaxi72n51zwhboqfl9p5s+z/feeg6zvnlwzht2aokob55cjyoooh4dxi0qvy/pr+mrzmlcvwdhpdtnjy3xkxgjak3hl8lqkuklwa6tculo4zlw3psfvsxh0jsmgl4eh0u7voo8eln7d5/tcbuxslivrueyv9odxebxvozlc6xwqui6evfpp8lzuqka2spnmzmrnhis2qqwwz9byriwf7lhuphzizdbig1+uuvbyoilnvfdtxl3miyusu22o8l23gswxqd5rjm+ha69sayjgfoebvxawgfvidda8ytcxtbp0buchjgugrgikuwtp19ud10i0fwhsmnciprbsqa4fopgojobft84ymos4ycxarcwydhbxkthlgoyuqlvzppuo9rff3jofasg/cnbzwbksliyg04goyysuwc5t0ngk+cqb785z9akzrg4qlbxbwradd/bc+9bb35naiqmlpcl6ja7uw+rljhfpamgssunojeshham41w5rypexti3nuhx5xv7mkpfznaetf+hq3yxeftz3druy0ccegsw1lezealyrr54pf9a8/j2fhtxx25jc6zb+4i4kqvnkpx10b6qhfk1sfph9eccqxzephp42i4dvq3wxvohstueapnlvecadc61psle29wybq6y9shnr4fhs9n+txbmngsiezdjshftecverauldo6efc2kmhanbenbsxd2jhzzywclbobiscjjc4gyczksycvc2mkgwssifkesu1q/ugeaxsytar6n1itt6msydshdwc4jymexzct8cytisqd1axrlnkqdlsynpwboxccifu91ltcgip0/gxy/bqeowcamczkoo+psnzy+iihccsvink3zecdpgoxmdqynuqkte/rtryvkt6u8jmoscygmy5xktnhhqaud0e6iphystluv0f+cckxnwgknlyedvdw0jyhrflh/mw7k0b/zhewlsbymsaaegbbh+7ydw140gqqfq6gxlql4g5fn7bkthuy1x6yal1znkdg32avc6gzlhpkln97wo5fkiwb8pih3q486pwdtdc5pe8cw2bbkxt72gqzmej30o5hyjkijvjgtbxu7ydvur5tymujzcets1l7ln/3urtciazuhes2xguu1lk9ivnjh1klryldaq80kh4kchmelt1edxcr/h2qb+5um6w6tjr2f3hyflcyjqim3frppdw+eo381alzf08u0rv1vn6izmgzduxdowlg9ysncftb6ektmqpxtbmajgpponfssnvdmxl7b/gkq4sjfqyq1ojwqwps8ytgfvglhx1so6wqpf7vssrfyxpveefdms2vdfjmdmxpoqipfjoeyt2hgyj72mqdcjhhrxs5ncuc1u7ozhxbxnqmi0zadlm7sge3xhdqv3q7yag0h/lq2vjzc/10dzqfhgdnkcimdvgkvtyxzcikcruzedhjzeststparubedrrycm0b2xdaypty0sicdrma7i2pgj4n+6aqpu6jr8cfd3uigdzvkolpohcuo2gokbdckj0mk+z+rn8wqg/tn75ud5jwee9bt9wemacsq66egbtkizwf8zs/6cnzhaf+yy8kzhs5xre5kv1splxcr0yqxoswokohic/ezhk7tne78siqqn8a40ybn1swnyaohrk6rb0r7wx5uqzzmscztl2uegacaeywj+5ip3hpepj1bhp4luc8flxnixh9d3iavr0/vnrkjjgl8otq4p7jhnkochutgynlkeupj6lzuj0fckkzlqmdsj2fxszfcw8oew2jgda0jewxsj2ghxgd7ailo9corme8qwn0huqkr01sh+fqnxtayt8tvjsgqwwahgc0orhcidiziaxgjstzt6kq8fzebgaeelyjnqj76gm11dpxkab1nlcvdr9uosv6hjipfvuqsdixjmyrpolebjultse5usvtnl2mrjnikpu2djcxwd0b6ckrjdejderygllybnoqzqj/lfjcgmhwnsjrnteu6a19g9jvjbvij7lljkjbdobc3riw9iu2glwsh1cgkma5zbzukquho6r7aa/9y2guea1vdawybkgzh5jsovdvclzuih2g0diou9a3c1nlg5runklaodbjvadxopbr91eczlgjhc1qjrhce1v9rltwfbox0inxmrprj2vtmvde67urcdmdbxcz5z63fgnmbqyq2hqvmdrshea6e7cgkrsc8paj4awsax3h43og1bmi4bjordcq1itznmhrinhsfmdmmkjnoi/ystiwuc5tolzwrybuzyqhbmppex1+0qeqmlpmu6iloietsmfodngip5pqvkgrjn1c9tulp45x0jvvuyxup6uxu8ptgxafpbn7klebgfq9xg9tbu47gdt1elgrjdun3dhqq3vzmnkexuewkkorluy3wjdmxttbvaa+pm9rneg6uspcadpr9nn8sd8ctdr7q3jqax8n4yejk1bam31mcptlf4lgpiekdvlfwvvbrgkxytjqh8zgo4mffagde2afshh1mfq20c5k+liw7eiyadictfzf2c2mzakjh0uer57knjsggkhw3qmwlcigqantsnb8+6gbfr3bwpcp52okfr7a6hh57sayvwdjfelnebj4rw+sdlicmi0uhcsjctcbwqxubjqgl9mkr5s0cvvv+yb6wg7qkniwk9rlep5koouvgzqnnjhuk9vr2uge2j/rt1sl/rantczst5/atwigpv2stwi+3hbuj3gaxftmhkkltrzrlngxng5lny2aam7hnvsupybhsot0ze5cqyirdphjunal6dgqtq5c2y5von9d7tz07glhkmpgozfegslp77tcq1toxjk8zj8lsjdjr8lqfivxogcaiiikiiqv1pmxx70oxrja28qwzj0orxlnp3p99a/cuzwkgazfqgompzrvu0xygzckvjtzq+2zc4b1qzvfbhs5r1chrt8t80f/wladvftsmc6yxwkyezkoqoq1b11alk40l80r5qs45jnwpzmheqqvmgz7oz8znzzccj27jxfgqumndwaysgbqfwqqaa/sf+ncpzb0xsyrfo/l+6lg0ylnefnnt06xui433e/z9wx8wv+zgbhhndcwre+djswbe583vic6w2c/crhuln0zmui6yuoo0w0cc4bwfbrgn7poxgvv6rszk60z8x5lr0cca7xa1g903klfeqr7my4sqb5y+rfqywkj+ktkf++pjo0r0we79mk09+nnyduyfdiwzdgp1lhctmu7dz9vrxfftt/ukuz0tplxhydc0qkecercwqdit3a9cd65++om38d/hyefwcnq1tnhb863anofbespzqhw4hd7pik7fez6aqsxvtmdup2sbu/v2ttajschvdl+i0yfdb56mnuf2dq9usaedky8dbe38ur5qhvvmxee6hupabiwe5xzsdeq4l1sreqlbpg7igexlobbzcml9xy/ievsfu5f9p48pqk1fmwdjhuaurvnxvvncazchc5hwasx98z2k1krxdnc9lfcymujcbbutjffpwpt3j3i2sum3lzuk8benoyv+dlp4twidk4c9xuhwr3okrfrpemtjdtlivwkq+2ly+yiruiuvk7sbpcvxtkonucrccyi/ww2/fdbcv4tgplymqt825+qusa54hibrboprbu/vfugaxvw9rfaypcb8yggiyxgc1rky hwqhfefztqvldvyfnx6mg6h3nwq9w8lzihvrs6tom8qw67tt24ghoc7b2vuiah67qetce4rvtcztu5r4ek3b+2ce81ru9ulvmdocp2sp117y9wzv/kfnxze+rybgmm9/4rzmj809k7i8v7u2o9nzhf39/45m/n38+8+bab9b+shz7bbt2p7vp1p6sha89xwtv/fpwv2/959z/3//f/v/f/+3932nog7caml+t9x7uv/n/igns0g==</latexit> <latexit sha1_base64="lvj5f1yayqj0pimo0ef8s222bnc=">aaa1fxicnvtbc922ezz6tdvb0j72ha2kqr1lqo4tj5mknoliq3yniqracm6irafjhb5ivbkaz8um+wp7d/ov+tq+tlsazyejgjiczdgigg8xi8xuynfrucrmyp39f63/4ic/+vfpfvrwzzz+/otf/urxb7/zmy9fxvkqvgjzjodfb0tqhgx0hwqyov8xnji0sohxwdujbp9qsrlgexyqfwu9t0mcsteliysqi3fww+0skuyffeyfcrgcjreecelc+ma2nxi54wm1bkjv+fmaympe7+gpx06ojpvd76hnex6nzh5rlwr/e2/bz5mibk7vuy1vltsr5px2ehmlha/p73t2rvsw9ztlsks0bokvn7y3tne98cvlv/a73j2jyp8q5axewz6vaud5xsxbm/t7++rhsz9gzcfmwvnzcvhoe+t+lidlsjopdha22i/keuw4zgfc6w2/flqg4rwj6rl8zisl4rxs5lb721ateeocw79meqq2s1grvihfggasnslmnqx0tz2vcvzrecwyopq0c3vh4zlxzo6hbxkr46cfzaefjoqmzpxcceeklgcbg71ucraxybk8n5rk0rkm87to18acfbmwzvu1azlixvnzvzbmpyylzz71q8cp8c0z2a+lmd+ue0knzmgmyiym2mig/gij9pikkxtn393xvhzww4z73gwenxgtmqvelnscieecl6byrmmmcpvqtseuwt+d2pweoun7daimxiw2r4st3qviuutf9ofngb6etkyagr0lcq7zm0nvg0mpje5nrczavhdqiengejc1wrwqkww5zjdczozk3oedkvjocr6gho9wmcmqlymyykzkiojmeumjee7lqfkifh/jwdfbwhnkked9of4hfwmtshy+apgw+ssros2jm5pnqzqmhapqpk+8kpmy1juf0jhllzrcyk9iameuevaapgkfc5puvpo6wttvpk3rbomimleifkba33tgttkxtcfcsazuqx0srtifrmfxnrrgzfqyekona7wjjvdnxxuxfsn46rprocqo5towdvwqlbrhnfa0crwzb3pk7e5+fvlkvnhgiktgxbtyot54fm4jqzgfyq+ebfmcxrnjomz61ugx0dnrcak+fxo9kqkienx4papupb64oueapxfdhr73uf88scrpyoygtr6/cktnjc1gebtgcagp58bxy3/+qorbb5ka+gp8fo1qqfpw1d4/fe7h1ebiyluwbwfctxcwg1/5mpyvsvojietixkedcs/ncmhqk59ltkbyylbqq1pmfw4py8jcyb5dxc8nqu4onvjlfgupi5baxpc05b1hmllgntmujtt1k8gsn+yrtfp0upyimeqj6hmy1anjhvulqgppkwgxwzxbbwgsyroi4oqkflx+ffndcyfacx3irgmjaablmulyryodam5yfosgd0uog1xmrvic4wx7er2zjgemj7wxz1pv+odg0zfxkilyj201gynbymimoofkmbjgneu8dygtnemc/pacwzowreakqv0vkhg1fo8svo+8+d0bkdkywc+ro+xupusyybarjhgmlzyarriyqjcmc+uudxpnsd/yhf9fvozhqh6zwrmyfrmo2t3guwx3ulonhxfiembs387ikwc3e+dxfahn4zvzncvdu40tmg7tc+jefpypcnzf+6pysnuitr3tawmucnit6tfrow74tlp7zufkvnsitjb1ggm2onf2ihhva3o0nwdah18ch50stfsn2zlzkolgqcyrxbtnlrozfptqurnstjbiwp0qjg0n3dmzwgqthywyacilcsctbmomzeoglwhpi4ekeyp6huloaf3yvgf6fbci+dzak7sje8vabq9noy8gb8c0e/mnlioladqpujtzea20mdscaqava4twpnw5xioidp5k2p20btgegzmu5djsqevvwpogbgorlit7ps1dgdxnsfe1gjzqpk5m6lcfl5ddrvksdesmbdkovso7xr0hrxzheyoawkjyblvg/wsqmj0jjzs8ggz0asucoetxxf7hujfdf8wtfpxcwjakmrigwh4f2a4me9eeenwmhca6poinrja1ybywgtv8uwq+ddvd4txkrwqjsy0bzi58elfvrcotgvdtbt4ppeosgbuzudxbamnmoyox+8rrmjpbdnlvygkzicvsrbyeboyc1vc9ycplcmuhllfr+5z/9ru3wiao1itonccgltw6pjj52ovri66d3wqtnpn8jaoyhi3crg0qq/x/kaxsb5quo7swdbry2xvqsisijt5a6d47onyb/twowrvnf9my9b/dom5imnvnqaaly/widqtzwotlljkjcu4racdotdp37rn0qpkbew3zy0gksxcfedbgveji7fmk4xtwpov9yzmse6t81uolxcgl7rndxjnnlg3xzq7gzkeddratnms+hrkr+9irgaqt4gv1tttfxdxs6s50wmvndoheqw64uludnl13al9xo2adtb3wm9pa8xd/d32ubyrlzjnrjazwplu8tczbcxiamrm5ygrlllpzwkbmxg4y2qpsgtlt28j6wnping4yme0ujogfdvqhezzsiq7bnw57oxs+7jcd6jp1rkbqd5as3cg8isnmf0befenu0de/13aqv3powe/xn5uirucshxoysis2h/g5pesr8kudfgmocc8aoscvusv9bqvcqqxok1yifqnkbiiv3gcaorvzovgrkkdcgongazzvv5wmqx6wuq4eku1ve7gn8zjhmk9ftnkcagdusyzsy6zyz3mtdm57ec5avxy8zcbz/xv5c1a8prt/biqwivhlavo+4h7tjhiwlhkhz5njjcej87o/bgljmpudhus9hsxtxnadk3smi4exnizsmug9houaa+yriaa3k6/apklgdyfknkrmbx3gazv6qwm8evc3pkgeg4uatqaq3ykwgcktyywowu+pk/fwxg5jhhi+odsr+uhrtnxr9zojdtsxlhqtvamkey2sdbbxkuk6lcrvq6sbrg2uje0luvmlbtw9jk2y4jgatnessfjdgekjqi7tcxxp2rqw2afwks+gfpzbwibqvjtcup7rlumefrnrv461sswwyyzdqnafwt66svsenig2kb1sh5bmumyd1jaeiagp+wip/gnhr3kova0ngaraml2z76jq7xbxviichthajbjrtwpb54gvvzsgwdowsfslcturavqza82yiixnusuyxrbuf61v1ny2odip8rm5uizlbzfvreu4q3m7gavzmesdtryl5gugkdp2ljg77b5aobcyh5aaafp2moktlwh84efxo9cqcomqaebukdcw2jtb6cfz7hxazdcozdin8lpsmrosbkjvvq6cvwsiow7bzbn/fdgmkpuw5foqizgnrrpbthvyhj2yu1zieitdxpxeij9pcti51lml16wkm7njuxotbzy5/slxmonbvmftuafvoho7chnrkiu5jn/zuub425t5bl1eszooejbttlixzcc2wvqfvd5uap/rovdc3ms4upbrflpcnuzyxap762ged+obislx25p+twmyshedxz4mpg9q3vr3wkulmsnruobmxzichxerhxhhh0hz9jlvnnauzppqmu5amalanbrcxnjnpwuciizjn0eeyzy6ihuocnwils3ciaqgbbn1/vq4g3wf28h3oodpjxyc2+os+u3bmrxiyrrbzhoreezsrxe4gjgulh0mihaxgcctbgswiczyi0wwtwj41vsl+8ju6cp6sijusxcfzqlkfbk1j5y7vijy79vjvtk+6rdbj/+xdrtn7uyc27dibt5eku1qvdwsbo56g8t25yrkjk290prxljvvslaumgfu4z75hqecrbdn9mxmqeiiew6r4lhxpooxqe0oxdosb6lfqo83du5g4jlb4km03xvc6u61w6hw61wwvme8jrovsg/a034kcjvlgcmijoqelntzmf89f12ywtrgl8w+dkv6zt1z//cn3lsyp4gkruibpj+2vrgtc4cspvq08qjv2wmbd0a5xwczoq9rh6zflfnf/ztthrz5btpmsv5qmreyj0lmngxdq5wsnjbme+wpoucwvj86hxgrrppnedqfgted3qqcucvzyufijxufm7iaanxqqqio7bfgqz+w9cyskrsnyvutx9ii5xjtte9nl7qpnn5p2pdtfcd8hhuxxq3xma/tny8lgxmdn1cnottmpwuz7rxemjnrumbcqdmma3gg1366xta6g15ohogxmzctg/edz0zfa2u2w9umdt9irnkj+tive6oewpoqackci/qpzbfsqyfuknehew/ctpbgz9s7nbfqmc4wjnvbqrrqug8/v6913u8/1znm9uyycowfxdoobnfiwmkmyq92vwrdvf7pj97gf48nn9vl6duyyw/nt2p7dm7pkwcb1ef2+cclop2j8nrmzxlzonmnjqvc3bevtulecnl3+bwndhx682ssobk3e3vdohhmerwgaoi39yxyqgfqctfxgac2ype452bnumk9pfrns9jycrahzcodtzhxsov7ehhx2y/yvrw8e3jybfmaarj3mhctyoqvyhlegamdw7qkkomp7ywljr22g5ulvjbjfbii2ggluxrworw/wbovlb9t2vkej2kilfqlmmk/fehyoezpqrj6nsnlpuqdwe6yqxarsopyz5krx4zelxxzod3kgyljgj3djrfkepwzbvjlwxh6jqq3twjq/by5v6emdcxjafxzb1x3z7/uq9io5uohsslsa+ydcxndxrcolpc993aafzrkqiic4jpwze+6om0+zgrv72joe7cc50uu1jjwnzwv59vjp7yhd3oeg7xzrqf8afxxgyyur51a2jeq9ycyhporduazruib2jw3a3zcvm645u3dqvn7nlpqfl8hw2dy/my/mhmzfyjjf3x5f2+0vzf6+/ubn/65+fozt9z+t/ahtttro7up1z5de7p2svzilvz/5/q/1/+z/t/3//dg+8hogz0n/cf6q/pbtd7pgw//d9gm8z4=</latexit> Modeling, Inference, Learning inference: solve _ modeling: define score function learning: choose _ Working definition of structured prediction: size of output space is exponential in size of input or is unbounded (e.g., machine translation) (we can t just enumerate all possible outputs) 11

12 What is Structured Prediction? however, just because the output is a structured object does not necessarily mean we are doing structured prediction we can model many structured output spaces with traditional local or unstructured predictors today we will aim to make this more formal in short, we may be predicting structures but we might not necessarily be using a structured predictor 12

13 Example NLP Tasks we ll go through some examples of NLP tasks that involve predicting output structures 13

14 Sequence Labeling (e.g., Part-of-Speech Tagging) determiner verb (past) prep. proper proper poss. adj. noun determiner verb (past) prep. noun noun poss. adj. noun Some questioned if Tim Cook s first product modal verb det. adjective noun prep. proper punc. modal verb det. adjective noun prep. noun punc. would be a breakaway hit for Apple. 14

15 Unlabeled Segmentations (Chinese Word Segmentation) some languages are written without whitespace task: insert spaces to form words 莎拉波娃现在居住在美国东南部的佛罗里达 莎拉波娃现在居住在美国东南部的佛罗里达 Sharapova now lives in US southeastern Florida J&M/SLP3

16 Labeled Segmentations (Named Entity Recognition) Some questioned if Tim Cook s first product would be a breakaway hit for Apple. PERSON ORGANIZATION 16

17 Labeled Segmentations (Entity Linking) Some questioned if Tim Cook s first product would be a breakaway hit for Apple. 17

18 Labeled Segmentation as Sequence Labeling O O O B-PERSON I-PERSON O O O Some questioned if Tim Cook s first product O O O O O O B-ORGANIZATION O would be a breakaway hit for Apple. B = begin I = inside O = outside 18

19 Trees (Constituency Parsing) (S (NP the man) (VP walked (PP to (NP the park)))) S VP NP PP NP DT NN VBD IN DT NN the man walked to the park Key: S = sentence NP = noun phrase VP = verb phrase PP = prepositional phrase DT = determiner NN = noun VBD = verb (past tense) IN = preposition 19

20 Unlabeled Segmentation + Clustering (Coreference Resolution)

21 Generation there are many language generation tasks that involve predicting a phrase, sentence, document, or some other textual sequence 21

22 Answers (Question Answering)

23 Sentences (Machine Translation)

24 Summaries (Summarization) The Apple Watch has drawbacks. There are other smartwatches that offer more capabilities. 24

25 Structured Prediction what is and is not structured prediction? we use the term structured prediction when: we use a structured score function or a structured loss function a structured score/loss function does not decompose across minimal parts of output to apply this definition we need to define parts and minimal parts 25

26 parts: each part is a subcomponent of entire input/output pair e.g., a single word and its associated POS tag for POS tagging or a sequence of two words and their POS tags or a sequence of two POS tags determiner verb (past) prep. proper proper poss. adj. noun determiner verb (past) prep. noun noun poss. adj. noun Some questioned if Tim Cook s first product modal verb det. adjective noun prep. proper punc. modal verb det. adjective noun prep. noun punc. would be a breakaway hit for Apple. 26

27 parts: each part is a subcomponent of entire input/output pair parts function = decomposition of input/output pair into a set of parts parts functions defined for score/loss function, rather than for task (many parts functions possible for a task) parts may overlap determiner verb (past) prep. proper proper poss. adj. noun determiner verb (past) prep. noun noun poss. adj. noun Some questioned if Tim Cook s first product modal verb det. adjective noun prep. proper punc. modal verb det. adjective noun prep. noun punc. would be a breakaway hit for Apple. 27

28 parts: each part is a subcomponent of entire input/output pair parts function = decomposition of input/output pair into a set of parts parts functions defined for score/loss function, rather than for task (many parts functions possible for a task) parts may overlap minimal parts: smallest possible parts for the task minimal parts function defined for task (structured output space), not for structured score/loss function minimal parts are non-overlapping 28

29 determiner verb (past) prep. proper proper poss. adj. noun determiner verb (past) prep. noun noun poss. adj. noun Some questioned if Tim Cook s first product modal verb det. adjective noun prep. proper punc. modal verb det. adjective noun prep. noun punc. would be a breakaway hit for Apple. minimal parts: smallest possible parts for the task minimal parts function defined for task (structured output space), not for structured score/loss function minimal parts are non-overlapping 29

30 Categories of Structured Prediction Problems multi-label classification: each input can be labeled with multiple labels e.g., document classification where each document can have multiple labels 30

31 Categories of Structured Prediction Problems multi-label classification in NLP:

32 Categories of Structured Prediction Problems 1 Coarse genre 1.1 Company Business, Strategy, etc. (elaborate in Section 3 [Topics]) 1.2 Purely multi-label Personal classification in NLP: 1.3 Personal but in professional context (e.g., it was good working with you) 1.4 Logistic Arrangements (meeting scheduling, technical support, etc) 4 Emotional tone (if not neutral) 4.1 jubilation 4.2 hope / anticipation 4.3 humor 4.4 camaraderie 4.5 admiration 4.6 gratitude 4.7 friendship / affection

33 Categories of Structured Prediction Problems multi-label classification: each input can be labeled with multiple labels if there are N possible labels, output space has size? (Q1 on handout) 33

34 Categories of Structured Prediction Problems multi-label classification: each input can be labeled with multiple labels if there are N possible labels, output space has size 2 N what are the minimal parts? (Q2 on handout) 34

35 parts: each part is a subcomponent of entire input/output pair parts function = decomposition of input/output pair into a set of parts parts functions defined for score/loss function, rather than for task (many parts functions possible for a task) parts may overlap minimal parts: smallest possible parts for the task minimal parts function defined for task (structured output space), not for structured score/loss function minimal parts are non-overlapping 35

36 <latexit sha1_base64="cs2rlstvqqvviaakpmv2nv3uyg0=">aaa6jhicxvtldxy7cea1e8vhhvz1ltkgizlqfslw5pghkzrhtkriovexpvw4lzy1b92n6yhylwfozrranb+rbbljr8kuj4ts8ltsbfrmdwmykvlg1lmxdecrqlwhucg8gjypl+r4+h8/+973/+ip7/zjd/50+8/+/c/+8oc/+vzhx8miehf7exvpib4jqwqpz9kbxvxkvikfo1mysq/dy8fy/vuve5ix+wtvl+wio0nozzyicqqmn9/z3qn5wk4e/iquvci5/xdbrtvcze1wtvtbwksxrdp41dttxfkibe09nrwchr0d1nptoa0cjailzboym8eio9gctkse8odnqxn8mana7b16qpcuiisjy2ltyxv2etg0+w3bkuaa/xfdf86aettvbwnvlmrucdwwlgz367vbe2f7306vfqqtuxyzbvzbiguztb9i8xu7zbyinswvupfnyaepzq0uppo7uuuhdv19u08eeldeaw0+dht62hzy0+/f9l7fm4qdfsuaz22r3zb7y7trwzeivedgzfo7q70adthpb5cwjwtnt/eivc8jy6mizujxiwgqwwtq3sbazqtgil0ywoiy2uli661qd70m7iuq3rwlkb000mcoayho8ap4eovcnsyfh66+hch8zk+v13jtajzuuax+uobqkkihrgp/cq11g/33lvwnac5hij0kwy4vonn02ummjgrbinzcsuib6q9idubvhyujodoijjgjgjwtfoaqq34egwsvtfm6gwuqsellc+c1npjejxjecah86kjs2pxdthcssqmuffypzdmfgzozry1wa/o0wuoyn7trixarbqpnip+v6+xbyby3d4f/dsls+m7ye/rkucqyu4uxftvii7zkqia2pz/aot461j/e/zh0hztb3c/z9poffhbervrllfda4vpjcakuglhiejsydjuojctpdektdg6foc2yvgj0qtwspaijyawq8f+uik4dujq0kzghayk6srsnk31t55wa/fichr+sfmq/09gssokqcc6bjoycrjdw8eejwufwes0pejechxj71e1ju1go82kksqpyukufljcj2ktqcs6j5bg2q1lfrbey10bffc8tvetj6lkgfktcjwszja8tvdglm7qknemrnl5g2gbae65yxj+wnxyqnqf8vterwycgczknv4zkbzfadokqkkkfy7kmhjs7a9efncjs7ufq82xcj6twxhnbwxzwuzuofbp2ddzhleyab2qx+fgafda+8+y+pxpodbqgsgzxqaujlsd1ygjzjkrkunkoml3ncfviddunlzoxomzuhagyixavpyrktknkkuuxryd4xdrj+piw/5zddadvtzghbpodeawqu9bkbxyx8lhzkltqlrmfy69albwnvnmnn3xfctlvmybkcc8bpbhnkniqcj5eqagypgzcad4eeuhy2tbhlguhdzpgwwoym83x0c/svozzliao2he0ohgsis0o7kqfauhsjgu3tr4wozikdcflmea36noety4g0umpa8pqhgnfukszmcyh52rnqg4pvzh7dtfzy80lp/ituzyevlpwhkgcgeqnl3fnekafjpru01orsw9zrh33/hpuzbd15on1qer3+oaghlmejw3z9hse+vkzi36ayartjf4yal7sriqs3tetqp8ogonu8op9burv2gd2huyo+qe259p3r/8acdzrdiyoq6oskrd6upme5lwgevuwq7wqui7e4mpqauutbtrwkha5lxe2wkz6qp7zmfvgia2vasrq8wuovaeadx75ruhkmgcu8t1dew8zga0sl+zqkb36ywcxzidbs8d0upyimsqu+hmytbt0hvulrgfvs1oydwfk7smcudktzgs+3zb3faiw3abyaenilsoyprl85+ggmzsw33nio0qgkqscbkugzh5kvdoec9xwsjitruzot04ev4anz4lpsgnmccvywsylu9jtxo7wesgycniqjctb76bcgbq8hqkqtk05byko735nphly5/ihymgbq54iaeqmkpo7rizgm6ksfcysjxccqc/1pojqbyv6ziccyob5o0n7nncg5optoot3geet38tsmw7eoweis30/iq9bziih4nmyhuyzzkymwx638zqb1regxmqpua5nq3sgkzezwmazz3ggwdut3qz6/klxa76d1lhzc+2riwqmy2bb0ucc2h8rt3z3jrsbxpvpl6cvzlau/yns7dhjzscgblbm0r23xc56umqwxkjzq8q58ekqtb90b2eyzns+wlaaslml0byhqnmgxfhijslwy6dcxshxr0ow1r69b4hejz0cvq3wnbvyxpf2kajfiwliwthnxpwji3jfapubkkp8ew202dscaaaxa4topdw5xieitf5k+f1vd/ainuncqlk0mjeucexblrrwob736qqiapkkwxbtyvdana1iwrj9tilsdp2x5fiyizbxrfpzk9xxsozwcjrhmyoe7vzj9v8zielmngfrjwsgl7v2akwal/apxwr6phypjyvplbikxleowp6eubmy9qipjpgycq11sccbjgx6q+4snlr53br5na8zeucue9uyk225mhvhm2cdxqs2ogv/3chhouefpfb+bfdneedy/ednyg9jx9qoovnxb3phbrlvorknydmfrixqbvsuwygh4yin1nfm57h7jwyiojberdkucgxw1cnknz6mxnq97nzrtz3ki1himwzhnmodsc7/n7ka/u3x9ydwky5bbrsgkp7gzmy7c91vxjx9g/orm7mu2lpmypt/3qz9ftotaxocxvirr2j5so2sl4ninsq7raccozoz3hnm0gvlfuw1zn9tphjno6dsuu4njm4+t3hbag972lfohpwc9hx1ukkfcmrmxcau3gxln/hlhsbcywmdlvncqdgfhsvh2muqszvgbxo5csvd3ezrzp6a9u0teik2tcd61how63pa+61/anzadwj+2zorhu7/ro+yghccbh6nkwhe9ornjjt4qv4n8yi3onkky122jpp5srucba22ney562rjro1kzzc4mttdpiq/3jgpvednu9gn/ovns9ep3zwhn5rrtwcf1xuajojg4rnnmp+z8qyecoox+31wg7y2c8phtv7cfm0n4ipubkv6tk3gf+6or8hvpob3tk541ja4rih86xcv5qridjowuvyyxbiqrek/0zcpnxcsv68oxi0zbjrgu3upwlruaavb5pg2/h7bbqjhhofevakgegia7rgs7gohc89ll3buetwxyksyrriowny9uwur0z7fv7asgm34bbnzx3npwm4ej1amklpnjmgt0wu71kfbkszuho6v0ag+nq43shj1qdfo0dfydvbgh1rraptqktnnxisxtyp5oyc5dre3wwf4wmsvwyin4vohbxrbyigblrjedcithkhrosnknjlqwrw1l9s4b54vamvqpsyw7w10/eayg80di91e1rljxwmki8v1rliei6nbgukmutsjkdti/ltjem0oyyvnekrm/j2mpnf3ruyiajdppl707b1yhga4lrzqvygcymgmkxzjqcjzkptivywyg8fzpflyzdnnqg6fwt6gsoyetigukansj5b2uuyc1nauizft+xip/bpprnkevasngaramknvhkjqvbprl5eex+j9rt44t0rx5qgtv3sg0dkwshwldturgfrlc83zmefdusw+lqibfzeprm53jt2jjnultkvryqqlznhx4hyac/yykqzrlrbscwxa9u1yshfyi4d0inqioqwcxu0jw1tbshhn8lirdatsograuh3ktys+ttf6qekqephzdpvbpxfbxjlol0ddtmqrdbcklth0lzh94m8u2nrye1xiipkijonvbddxnzxjd/aqlyqg6tbxpxpiz7ocdo71dml3kelnboqmjdqdt29/ypuvmc0+4tzogpcdjb3wezgapwxo7+0opklvzt6dlupes1rkx3d7re+ym5fgsge8me7kpdfzoks71/cg3kp5brk7m/ppunvnab4p44ehrdbbmnfpkbjkeiphpjzmdnddovdazq+yydftnrmpdqq/k9c78ni9kby4z9brwewhew6lsweei6dbsc9fzvxceiaa6jimersf6qmjsoekz41q1sm8okrwnzvnr0+gwfejg3xpc5enysgpay5p3h6swiuwbbhkyhsbf9s1+oalionjmmcccb8tcnz6rmafwok7worfj4bvs19yl+w2xuxp61k/balylivbkff3wnnohiljst8e2+2tcbtz8p+yqnvdzisfeytw4umv1axwqy3hzms0qexftuhk0ryry7jgefegzpeyu3ax98z3bjm8hj0nstmbbylmtikuj7rmzyhba3my0uzff/029h5j5x4gphksptryixxwd68drq1e5/ap7lx62mzwin0ct/eryouvxxgrc1mmtnbpwwl90hpfauubx3h3obtptfcsg7/p4otshgaalskdmpnyxen2bquvz5js5nhftxmc74akgq8ytf6jd1k/rvlf4kmtd9789hnmzcy1majzryflfvvtq5ecnjyuuz1ttc5hrx5scrrrrrpnecafmxwd3qqc+ixzyufig/ufm6oeanxqqqmo7bf0x13gy1ky46li6ftkhwjpykiotdgjhi59h2m9n3dv28aa5q03yvun1zef3j2pi4f9nbfujzp7dj+1xffbbjjsdv1yqc1ph4ekx2s/0+i63q0vne7mi0zi1q34jiojo4ez2tf6bidvjooignxzvcntdywdils0xqj5kjkeb/ovcvcinx/9lmpaaj975omyhi5xhefxsoiy0wwffq5f6/qffuonmfqzmbmac7umsqvi0jbhxcvr638venj900+8jf89nnzurybxyis9dd+67rvms6inc5got8mneszy30uiryovutoz/yzwulvvx2vtmjb67zy2dlkjhgbzzk05arr79umggex5pcboejf1hfloz+oan5gxhmzik1khwm+ghhtjvzpxsoutia7iviv6mw+qngjmim62x7b7oxn57gmzgxgg4nyburtr1nx1lvajtngshlvcyxz8xlia0ms7ufnka5lcquieg6200m9a9oxp8wxln+hbdjxhexzbc/sshvc3ju6savz0ev2/zlgq0awnwehzyx6rsxuxv0kvn3ca9pbvhzkml2msspgllolqgudpccovhk1wxkjw0ysgm2+bc33k2dyicwfqhsdupthux5astor+ikwxx4d5uyojlxmivcvi0u8puv9mlivqbme1a7rfbfo6+zixtl+jurjbslguhdrlwnvevv40z4paetwqraheluob+pg6bgrngv479baxujycybiulgeyl6ziborx3ahwsv+57ru3nztdx9a0tff7ohggw+43/thmxp4tgffjq/thk+ojye/+cefx/9t9+cwptv rcmw7/y+vxw862zrtdb0z33d/7tzr/f+y8h//ngvx7894p/mddvfdbr/nxw6ofb//0/vhkhjq==</latexit> Categories of Structured Prediction Problems multi-label classification: each input can be labeled with multiple labels if there are N possible labels, output space has size 2 N what are the minimal parts? individual labels the mp(y) function defines the set of minimal parts of the structured output y 36

37 <latexit sha1_base64="cs2rlstvqqvviaakpmv2nv3uyg0=">aaa6jhicxvtldxy7cea1e8vhhvz1ltkgizlqfslw5pghkzrhtkriovexpvw4lzy1b92n6yhylwfozrranb+rbbljr8kuj4ts8ltsbfrmdwmykvlg1lmxdecrqlwhucg8gjypl+r4+h8/+973/+ip7/zjd/50+8/+/c/+8oc/+vzhx8miehf7exvpib4jqwqpz9kbxvxkvikfo1mysq/dy8fy/vuve5ix+wtvl+wio0nozzyicqqmn9/z3qn5wk4e/iquvci5/xdbrtvcze1wtvtbwksxrdp41dttxfkibe09nrwchr0d1nptoa0cjailzboym8eio9gctkse8odnqxn8mana7b16qpcuiisjy2ltyxv2etg0+w3bkuaa/xfdf86aettvbwnvlmrucdwwlgz367vbe2f7306vfqqtuxyzbvzbiguztb9i8xu7zbyinswvupfnyaepzq0uppo7uuuhdv19u08eeldeaw0+dht62hzy0+/f9l7fm4qdfsuaz22r3zb7y7trwzeivedgzfo7q70adthpb5cwjwtnt/eivc8jy6mizujxiwgqwwtq3sbazqtgil0ywoiy2uli661qd70m7iuq3rwlkb000mcoayho8ap4eovcnsyfh66+hch8zk+v13jtajzuuax+uobqkkihrgp/cq11g/33lvwnac5hij0kwy4vonn02ummjgrbinzcsuib6q9idubvhyujodoijjgjgjwtfoaqq34egwsvtfm6gwuqsellc+c1npjejxjecah86kjs2pxdthcssqmuffypzdmfgzozry1wa/o0wuoyn7trixarbqpnip+v6+xbyby3d4f/dsls+m7ye/rkucqyu4uxftvii7zkqia2pz/aot461j/e/zh0hztb3c/z9poffhbervrllfda4vpjcakuglhiejsydjuojctpdektdg6foc2yvgj0qtwspaijyawq8f+uik4dujq0kzghayk6srsnk31t55wa/fichr+sfmq/09gssokqcc6bjoycrjdw8eejwufwes0pejechxj71e1ju1go82kksqpyukufljcj2ktqcs6j5bg2q1lfrbey10bffc8tvetj6lkgfktcjwszja8tvdglm7qknemrnl5g2gbae65yxj+wnxyqnqf8vterwycgczknv4zkbzfadokqkkkfy7kmhjs7a9efncjs7ufq82xcj6twxhnbwxzwuzuofbp2ddzhleyab2qx+fgafda+8+y+pxpodbqgsgzxqaujlsd1ygjzjkrkunkoml3ncfviddunlzoxomzuhagyixavpyrktknkkuuxryd4xdrj+piw/5zddadvtzghbpodeawqu9bkbxyx8lhzkltqlrmfy69albwnvnmnn3xfctlvmybkcc8bpbhnkniqcj5eqagypgzcad4eeuhy2tbhlguhdzpgwwoym83x0c/svozzliao2he0ohgsis0o7kqfauhsjgu3tr4wozikdcflmea36noety4g0umpa8pqhgnfukszmcyh52rnqg4pvzh7dtfzy80lp/ituzyevlpwhkgcgeqnl3fnekafjpru01orsw9zrh33/hpuzbd15on1qer3+oaghlmejw3z9hse+vkzi36ayartjf4yal7sriqs3tetqp8ogonu8op9burv2gd2huyo+qe259p3r/8acdzrdiyoq6oskrd6upme5lwgevuwq7wqui7e4mpqauutbtrwkha5lxe2wkz6qp7zmfvgia2vasrq8wuovaeadx75ruhkmgcu8t1dew8zga0sl+zqkb36ywcxzidbs8d0upyimsqu+hmytbt0hvulrgfvs1oydwfk7smcudktzgs+3zb3faiw3abyaenilsoyprl85+ggmzsw33nio0qgkqscbkugzh5kvdoec9xwsjitruzot04ev4anz4lpsgnmccvywsylu9jtxo7wesgycniqjctb76bcgbq8hqkqtk05byko735nphly5/ihymgbq54iaeqmkpo7rizgm6ksfcysjxccqc/1pojqbyv6ziccyob5o0n7nncg5optoot3geet38tsmw7eoweis30/iq9bziih4nmyhuyzzkymwx638zqb1regxmqpua5nq3sgkzezwmazz3ggwdut3qz6/klxa76d1lhzc+2riwqmy2bb0ucc2h8rt3z3jrsbxpvpl6cvzlau/yns7dhjzscgblbm0r23xc56umqwxkjzq8q58ekqtb90b2eyzns+wlaaslml0byhqnmgxfhijslwy6dcxshxr0ow1r69b4hejz0cvq3wnbvyxpf2kajfiwliwthnxpwji3jfapubkkp8ew202dscaaaxa4topdw5xieitf5k+f1vd/ainuncqlk0mjeucexblrrwob736qqiapkkwxbtyvdana1iwrj9tilsdp2x5fiyizbxrfpzk9xxsozwcjrhmyoe7vzj9v8zielmngfrjwsgl7v2akwal/apxwr6phypjyvplbikxleowp6eubmy9qipjpgycq11sccbjgx6q+4snlr53br5na8zeucue9uyk225mhvhm2cdxqs2ogv/3chhouefpfb+bfdneedy/ednyg9jx9qoovnxb3phbrlvorknydmfrixqbvsuwygh4yin1nfm57h7jwyiojberdkucgxw1cnknz6mxnq97nzrtz3ki1himwzhnmodsc7/n7ka/u3x9ydwky5bbrsgkp7gzmy7c91vxjx9g/orm7mu2lpmypt/3qz9ftotaxocxvirr2j5so2sl4ninsq7raccozoz3hnm0gvlfuw1zn9tphjno6dsuu4njm4+t3hbag972lfohpwc9hx1ukkfcmrmxcau3gxln/hlhsbcywmdlvncqdgfhsvh2muqszvgbxo5csvd3ezrzp6a9u0teik2tcd61how63pa+61/anzadwj+2zorhu7/ro+yghccbh6nkwhe9ornjjt4qv4n8yi3onkky122jpp5srucba22ney562rjro1kzzc4mttdpiq/3jgpvednu9gn/ovns9ep3zwhn5rrtwcf1xuajojg4rnnmp+z8qyecoox+31wg7y2c8phtv7cfm0n4ipubkv6tk3gf+6or8hvpob3tk541ja4rih86xcv5qridjowuvyyxbiqrek/0zcpnxcsv68oxi0zbjrgu3upwlruaavb5pg2/h7bbqjhhofevakgegia7rgs7gohc89ll3buetwxyksyrriowny9uwur0z7fv7asgm34bbnzx3npwm4ej1amklpnjmgt0wu71kfbkszuho6v0ag+nq43shj1qdfo0dfydvbgh1rraptqktnnxisxtyp5oyc5dre3wwf4wmsvwyin4vohbxrbyigblrjedcithkhrosnknjlqwrw1l9s4b54vamvqpsyw7w10/eayg80di91e1rljxwmki8v1rliei6nbgukmutsjkdti/ltjem0oyyvnekrm/j2mpnf3ruyiajdppl707b1yhga4lrzqvygcymgmkxzjqcjzkptivywyg8fzpflyzdnnqg6fwt6gsoyetigukansj5b2uuyc1nauizft+xip/bpprnkevasngaramknvhkjqvbprl5eex+j9rt44t0rx5qgtv3sg0dkwshwldturgfrlc83zmefdusw+lqibfzeprm53jt2jjnultkvryqqlznhx4hyac/yykqzrlrbscwxa9u1yshfyi4d0inqioqwcxu0jw1tbshhn8lirdatsograuh3ktys+ttf6qekqephzdpvbpxfbxjlol0ddtmqrdbcklth0lzh94m8u2nrye1xiipkijonvbddxnzxjd/aqlyqg6tbxpxpiz7ocdo71dml3kelnboqmjdqdt29/ypuvmc0+4tzogpcdjb3wezgapwxo7+0opklvzt6dlupes1rkx3d7re+ym5fgsge8me7kpdfzoks71/cg3kp5brk7m/ppunvnab4p44ehrdbbmnfpkbjkeiphpjzmdnddovdazq+yydftnrmpdqq/k9c78ni9kby4z9brwewhew6lsweei6dbsc9fzvxceiaa6jimersf6qmjsoekz41q1sm8okrwnzvnr0+gwfejg3xpc5enysgpay5p3h6swiuwbbhkyhsbf9s1+oalionjmmcccb8tcnz6rmafwok7worfj4bvs19yl+w2xuxp61k/balylivbkff3wnnohiljst8e2+2tcbtz8p+yqnvdzisfeytw4umv1axwqy3hzms0qexftuhk0ryry7jgefegzpeyu3ax98z3bjm8hj0nstmbbylmtikuj7rmzyhba3my0uzff/029h5j5x4gphksptryixxwd68drq1e5/ap7lx62mzwin0ct/eryouvxxgrc1mmtnbpwwl90hpfauubx3h3obtptfcsg7/p4otshgaalskdmpnyxen2bquvz5js5nhftxmc74akgq8ytf6jd1k/rvlf4kmtd9789hnmzcy1majzryflfvvtq5ecnjyuuz1ttc5hrx5scrrrrrpnecafmxwd3qqc+ixzyufig/ufm6oeanxqqqmo7bf0x13gy1ky46li6ftkhwjpykiotdgjhi59h2m9n3dv28aa5q03yvun1zef3j2pi4f9nbfujzp7dj+1xffbbjjsdv1yqc1ph4ekx2s/0+i63q0vne7mi0zi1q34jiojo4ez2tf6bidvjooignxzvcntdywdils0xqj5kjkeb/ovcvcinx/9lmpaaj975omyhi5xhefxsoiy0wwffq5f6/qffuonmfqzmbmac7umsqvi0jbhxcvr638venj900+8jf89nnzurybxyis9dd+67rvms6inc5got8mneszy30uiryovutoz/yzwulvvx2vtmjb67zy2dlkjhgbzzk05arr79umggex5pcboejf1hfloz+oan5gxhmzik1khwm+ghhtjvzpxsoutia7iviv6mw+qngjmim62x7b7oxn57gmzgxgg4nyburtr1nx1lvajtngshlvcyxz8xlia0ms7ufnka5lcquieg6200m9a9oxp8wxln+hbdjxhexzbc/sshvc3ju6savz0ev2/zlgq0awnwehzyx6rsxuxv0kvn3ca9pbvhzkml2msspgllolqgudpccovhk1wxkjw0ysgm2+bc33k2dyicwfqhsdupthux5astor+ikwxx4d5uyojlxmivcvi0u8puv9mlivqbme1a7rfbfo6+zixtl+jurjbslguhdrlwnvevv40z4paetwqraheluob+pg6bgrngv479baxujycybiulgeyl6ziborx3ahwsv+57ru3nztdx9a0tff7ohggw+43/thmxp4tgffjq/thk+ojye/+cefx/9t9+cwptv rcmw7/y+vxw862zrtdb0z33d/7tzr/f+y8h//ngvx7894p/mddvfdbr/nxw6ofb//0/vhkhjq==</latexit> minimal parts: Multi-Label Classification if score & loss functions factor across minimal parts, then we are not doing structured prediction e.g., we could build N binary classifiers, one for each label, and use them to independently predict each label for each input this would not be considered structured prediction 37