Compressing the information contained in the different indexes is crucial for performance when implementing an IR system

Transkrypt

1 4.2 Compression Compressing the information contained in the different indexes is crucial for performance when implementing an IR system on current hardware it is typically much faster to read compressed data from secondary storage and decompress it on-the-fly than to read the same data in uncompressed form with compression more data fits into the available primary storage We will now see different ways how we can assign a code (i.e., a sequence of bits or bytes) to symbols (e.g., terms, document identifiers, positions) that need to be represented a

2 Byte-Level vs. Bit-Level Codes Byte-level codes assign each symbol a variable number of bytes and concatenate codes for symbols in a sequence, so that each byte contains information about one symbol Example: With Variable-Byte Encoding the integer sequence 1, 2, 4, 3, 1 is represented as the byte sequence Symbol: Code: Each byte contains information about a single symbol 25

3 Byte-Level vs. Bit-Level Codes Bit-level codes assign each symbol a variable number of bits and concatenate codes for symbols in a sequence, so that a byte can contain bits belonging to different symbols Example: With unary coding the integer sequence 1, 2, 4, 3, 1 is represented as the byte sequence Symbol: Code: Each byte can contain information about multiple symbols 26

4 Prefix Codes In order to be able to interpret a given sequence of bits and reconstruct the original information unambiguously, a code needs to be prefix free, i.e., there must not be two codes, so that one is a prefix of the other Example: With non-prefix-free code (don t use!) the integer sequence 1, 3, 1, 3, 2 would be represented as the byte sequence Symbol: Code: which we would also obtain when encoding 3, 1, 3, 1, 2 27

5 <latexit sha1_base64="hkazymd72ptjjmeirc54g3/0g+4=">aaacmxicdzdnthsxfixv8fnoskua7roznruq2pljqbqwsjg6yymeeofizgjkcs/bij22be+vyjq34e3ysauvwa7qfh4cn4ffinrkxedzfc3duyjoje2s+2bufmhxzdly28a7lfcfvptr62dgvppij0oudb8gbjkrswez5dhxgokooj4xw+9/8/ofqa2t5akdk8wegztsklfivzu3o8pn/lz4eh4nu1oj3i3cljvhvw5tpd2otr+kxa7yjpte67zzsqlo/l6yvr++hnautntqfosjjvpmy/pd0kpgasknxly0e2uzr7rllgpdscuditahgeafsrrrqdm3qcnq6vfs7lekak3mjmvphi4iy8aiegukyq9mzulkoswf+edze7mxovaqymjjftzbf/emly8efqj9yfhrom6vn/xojube4igwhgppio91w9fz0kh4fzjrrg3pjzut7u60j1igt/aznqen36alh3ampabwdbdwb7+cm+a++b38mt6dc55nnmbgwcmt8ger7a==</latexit> <latexit sha1_base64="hkazymd72ptjjmeirc54g3/0g+4=">aaacmxicdzdnthsxfixv8fnoskua7roznruq2pljqbqwsjg6yymeeofizgjkcs/bij22be+vyjq34e3ysauvwa7qfh4cn4ffinrkxedzfc3duyjoje2s+2bufmhxzdly28a7lfcfvptr62dgvppij0oudb8gbjkrswez5dhxgokooj4xw+9/8/ofqa2t5akdk8wegztsklfivzu3o8pn/lz4eh4nu1oj3i3cljvhvw5tpd2otr+kxa7yjpte67zzsqlo/l6yvr++hnautntqfosjjvpmy/pd0kpgasknxly0e2uzr7rllgpdscuditahgeafsrrrqdm3qcnq6vfs7lekak3mjmvphi4iy8aiegukyq9mzulkoswf+edze7mxovaqymjjftzbf/emly8efqj9yfhrom6vn/xojube4igwhgppio91w9fz0kh4fzjrrg3pjzut7u60j1igt/aznqen36alh3ampabwdbdwb7+cm+a++b38mt6dc55nnmbgwcmt8ger7a==</latexit> <latexit sha1_base64="hkazymd72ptjjmeirc54g3/0g+4=">aaacmxicdzdnthsxfixv8fnoskua7roznruq2pljqbqwsjg6yymeeofizgjkcs/bij22be+vyjq34e3ysauvwa7qfh4cn4ffinrkxedzfc3duyjoje2s+2bufmhxzdly28a7lfcfvptr62dgvppij0oudb8gbjkrswez5dhxgokooj4xw+9/8/ofqa2t5akdk8wegztsklfivzu3o8pn/lz4eh4nu1oj3i3cljvhvw5tpd2otr+kxa7yjpte67zzsqlo/l6yvr++hnautntqfosjjvpmy/pd0kpgasknxly0e2uzr7rllgpdscuditahgeafsrrrqdm3qcnq6vfs7lekak3mjmvphi4iy8aiegukyq9mzulkoswf+edze7mxovaqymjjftzbf/emly8efqj9yfhrom6vn/xojube4igwhgppio91w9fz0kh4fzjrrg3pjzut7u60j1igt/aznqen36alh3ampabwdbdwb7+cm+a++b38mt6dc55nnmbgwcmt8ger7a==</latexit> <latexit sha1_base64="2ngedvnqqr1yt5mwugex2umapvo=">aaacmxicdzdlsgmxgiuz3q23qks3g90o6my0ipefilhxiyhyltjdkim/ntszhcqjlwhewjdx51zfwp241ycw1rqo4igll+fkd/wnk4xqe0uv3sjo2pje5nr0zwz2bn6hurh0ouwhcdsjyek1mqyb0ryahhoglaka84zbzdy5+sov70bpkvjz05oqcnzo6q0l2dgrrqbha611/8df9gnd8nr2/zjmfqv0y6lst4w3yibaacn+x8u0woucxv5etlxv/4v6epvvqwodptwp+fqqgknucmnax9ujarkllageqvmjcw0skw5uwxwojjagetsgwcgo3ndsmmccdgl7aw+ffnotezz7y3jsbofntiiowzn+9bm52usszwvhiccu66mnm9qvf3yyuo7q8kqxptizbicdmmygg3jmlncbw7li6vnpwp8flhpb3fhzdu1wz1duffpbq2gn1deuokth6bq1euh36be9owfvwxvxxr2376cj3mbmgq3je/8ed2orma==</latexit> Entropy Entropy, as proposed by Claude E. Shannon, measures the uncertainty of a discrete random variable X and is defined as H(X) = ÿ xœx P[x ] log 2 P[x ] Quelle: en.wikipedia.org with x as a possible outcome of the random variable and P[ x ] as the probability of observing this outcome 28

6 <latexit sha1_base64="azdswq439r934hmxcslnfk3bhxs=">aaactnicdzdpaxnbfmffxqox/krvwy+luxjk7laxzueiecmi0eltfrpledt5iunmdqyzs9iw7b/lf+jnvondszfr6bzcy/exhw/f73cevg+pbbcutb9gnvsbt+/c7d7r3x/w8nhj/uati6tqw2jclfdmperlglc0cdwjotggujacjsvl2wv/+cmzy1v16faacomlis85qxekaf9dro3pjboprvo9wpltmvhnnm8nmj9qfjy2cx5w4yy+hjxelq4hszy47q/s4fzon305im9cnkzbgyyfqtv70/6pfkzylalytkc1p1mqxehrom4enb28tqsrlxfbp5hq1gqkvyalyznvuh2wqkm28g0na6zhae1kljfei2vwxvkppcps/rpczxclzytdo6py8fr0ycsgzpnlssy8nrxfjvmdhhcttqq6ok8kmqpsmgdtc/x87sd+pxxtd7pab68g49expuextua5viamdmame7ape2dwcb7an/gefy5+rr+i35frtnt15ymstaf7b531ucm=</latexit> <latexit sha1_base64="azdswq439r934hmxcslnfk3bhxs=">aaactnicdzdpaxnbfmffxqox/krvwy+luxjk7laxzueiecmi0eltfrpledt5iunmdqyzs9iw7b/lf+jnvondszfr6bzcy/exhw/f73cevg+pbbcutb9gnvsbt+/c7d7r3x/w8nhj/uati6tqw2jclfdmperlglc0cdwjotggujacjsvl2wv/+cmzy1v16faacomlis85qxekaf9dro3pjboprvo9wpltmvhnnm8nmj9qfjy2cx5w4yy+hjxelq4hszy47q/s4fzon305im9cnkzbgyyfqtv70/6pfkzylalytkc1p1mqxehrom4enb28tqsrlxfbp5hq1gqkvyalyznvuh2wqkm28g0na6zhae1kljfei2vwxvkppcps/rpczxclzytdo6py8fr0ycsgzpnlssy8nrxfjvmdhhcttqq6ok8kmqpsmgdtc/x87sd+pxxtd7pab68g49expuextua5viamdmame7ape2dwcb7an/gefy5+rr+i35frtnt15ymstaf7b531ucm=</latexit> <latexit sha1_base64="azdswq439r934hmxcslnfk3bhxs=">aaactnicdzdpaxnbfmffxqox/krvwy+luxjk7laxzueiecmi0eltfrpledt5iunmdqyzs9iw7b/lf+jnvondszfr6bzcy/exhw/f73cevg+pbbcutb9gnvsbt+/c7d7r3x/w8nhj/uati6tqw2jclfdmperlglc0cdwjotggujacjsvl2wv/+cmzy1v16faacomlis85qxekaf9dro3pjboprvo9wpltmvhnnm8nmj9qfjy2cx5w4yy+hjxelq4hszy47q/s4fzon305im9cnkzbgyyfqtv70/6pfkzylalytkc1p1mqxehrom4enb28tqsrlxfbp5hq1gqkvyalyznvuh2wqkm28g0na6zhae1kljfei2vwxvkppcps/rpczxclzytdo6py8fr0ycsgzpnlssy8nrxfjvmdhhcttqq6ok8kmqpsmgdtc/x87sd+pxxtd7pab68g49expuextua5viamdmame7ape2dwcb7an/gefy5+rr+i35frtnt15ymstaf7b531ucm=</latexit> <latexit sha1_base64="a+jafw1c2e/mkqxcco1ku7d+6/i=">aaactnicdzdbsxtbfmzn01zjtda2ry+luxjk7qq05iaivqhsudaquet4o3mjq2z2pjozyhj2j/i/6u282xvp3kthgmfufmzhx/d98+b9uelm2di+cwofpn6am68vnbaxpi9/aa58ptay1bs7vhkpt3mwyfmbxcssx1oleuto8sqf/xzyty5qgyaliztwmakyfmzakfgv9zr7qdiufwdptxb7ch1tvefoma40unepxbjxyfq7hh74ongejl8ojpngr9mk2xud7xize76fpb1ppkwmc9br/kv7kpycc0s5ghowxmpmdrrllgpvseudcugihnggsqkfonndlakths/ahgsqadi36wtgdccmgyv8jfh0zayyszmykjv/ltvbduzyouqlbfxebf3unb7zajsj9o9fv8zrt3nb32qidhyviwfus9p2wdv8ps8dho/d8uy78xy41dr9pi2qtlbjglkncflbdskeosbdqskvusa35g/wj7gl7ooh52gtmp75rmamvn8ei1g3zw==</latexit> <latexit sha1_base64="muhbr37amjdp/6qmvfdkiaurt+e=">aaacthicdza/bxnbemxntaij+wccxzotbqh8dwzbuksyrbmkpeskk0i5kzw3hpuvd2+x3t2etbrpxdeho6cbl0bjf0xkxkmkgivrfj+993akeauw3lo0/rm17q3dx3+wsdl++ojxk6edz1vhvtwg0zapocxpizyer2joubn0qg2hlawdllp3v/7jfzkwq+riztuveqcvn3cglkijzodcg59ldj+m9puey9s08v6ctwwynzw+38t55xrh8d3cexkxkht1ummvv7utvt6nvyhrpyvpdl7ayg5gnt/5wlfauuwyqgvpsls7wqnxnalq2nltssob4ztomnwoyrr+skqsm/nlo2cfkmzhf40smr6ltxnzrohxxyylpvizh6x9z7mb7bsev7p2vlhgldanqxt6twylmfb48ngejhqqwk02eejoaykrgwv7azt2qoe2g/j/cnzvzyep33qhb697gg3yhpfwcjj4bwpyhwmyaonv8an+we/oe/q3oo8urqot6obpc1ia1volfj63qq==</latexit> <latexit sha1_base64="muhbr37amjdp/6qmvfdkiaurt+e=">aaacthicdza/bxnbemxntaij+wccxzotbqh8dwzbuksyrbmkpeskk0i5kzw3hpuvd2+x3t2etbrpxdeho6cbl0bjf0xkxkmkgivrfj+993akeauw3lo0/rm17q3dx3+wsdl++ojxk6edz1vhvtwg0zapocxpizyer2joubn0qg2hlawdllp3v/7jfzkwq+riztuveqcvn3cglkijzodcg59ldj+m9puey9s08v6ctwwynzw+38t55xrh8d3cexkxkht1ummvv7utvt6nvyhrpyvpdl7ayg5gnt/5wlfauuwyqgvpsls7wqnxnalq2nltssob4ztomnwoyrr+skqsm/nlo2cfkmzhf40smr6ltxnzrohxxyylpvizh6x9z7mb7bsev7p2vlhgldanqxt6twylmfb48ngejhqqwk02eejoaykrgwv7azt2qoe2g/j/cnzvzyep33qhb697gg3yhpfwcjj4bwpyhwmyaonv8an+we/oe/q3oo8urqot6obpc1ia1volfj63qq==</latexit> <latexit sha1_base64="muhbr37amjdp/6qmvfdkiaurt+e=">aaacthicdza/bxnbemxntaij+wccxzotbqh8dwzbuksyrbmkpeskk0i5kzw3hpuvd2+x3t2etbrpxdeho6cbl0bjf0xkxkmkgivrfj+993akeauw3lo0/rm17q3dx3+wsdl++ojxk6edz1vhvtwg0zapocxpizyer2joubn0qg2hlawdllp3v/7jfzkwq+riztuveqcvn3cglkijzodcg59ldj+m9puey9s08v6ctwwynzw+38t55xrh8d3cexkxkht1ummvv7utvt6nvyhrpyvpdl7ayg5gnt/5wlfauuwyqgvpsls7wqnxnalq2nltssob4ztomnwoyrr+skqsm/nlo2cfkmzhf40smr6ltxnzrohxxyylpvizh6x9z7mb7bsev7p2vlhgldanqxt6twylmfb48ngejhqqwk02eejoaykrgwv7azt2qoe2g/j/cnzvzyep33qhb697gg3yhpfwcjj4bwpyhwmyaonv8an+we/oe/q3oo8urqot6obpc1ia1volfj63qq==</latexit> <latexit sha1_base64="ioaifbsgffkv4qykxteyn2gnpva=">aaacthicdzdnthsxfiu9afklf4eu2yzihlvmjiakcyqknnsbrcucsmwouupcbcv22lu9imiaz+jnuuuim/yfwhzxiwhsidvexhnx6zzjk92tk86mjeofqe3dx7n5hcwl+vlk6tp6y2pz0shsu+xqyaw+zsegzwv2llmcr5vgednhq3x48ujf3ae2tbyxdqqwezaowj9rsf7qnr6ksrtugl3vwp0i9exvhudh2tdaxvk5dhwm30rohf/nlodxmvy30yxb7codepcwniwkfy+nsszz3m08pj1js4gfprymuuliztmh2jlksaqnpuefdagdvifygukduqfkgvappm2pbqg0mrs3mmu6emamrd4jvhwzlezsdi3k5s1y2z/ihctuabgg3hujizrg9xonc9t+sehsfhwvf/xnjujg8t4sqlu0ly9v3dfz2kh4ply2w4nnr3vn4/1juytki2ythzkqz+synjjz0igupjaf5bf5hxwp/gr/g6d/0vow+fojte1t/hkb+rdv</latexit> <latexit sha1_base64="wsjiahmjrcx+sbux77/in/b/kto=">aaackxicdzdnbhmxfixvbepl+gla2leznrjiuwvm0qptfkir2gsdvkqmjzqzrxecm2dfhlu2uzuazrvwjuzywkuwa7zseqjclc5c1cmvpp9jw74n14jbf8c/g9qdh1uptnce1588ffz8t7g3p7bqyrj1mrlkdho0jhhbfcedoke2hdixdjnp393kl1dklfffhvtqyitocj7ldj23xo3xw/btmjaprpfryljholhvyxi3v0aubjvw40yzbru7z/frj9yepbwv1oy+hjxox40/6usxhatcmyhwjpjyu6xe4zgtvnxthswnbi4zgmgsuzpjyhkpsc4s12opbuqywbkady0suvq7lpmgefp/dtnxau4wt/897qznwcklvxbumj+tsiz61pcwzxmyfvho/tiaa2/4mwwk0nf1jjezzvseq7qv566d8h4ytfuj5w/hze7jbu+wa6/gan5aaqfqhr6cqx8yfiiv8bw+bz+d78gp4nft0vrw784lwfpw+y+imqny</latexit> <latexit sha1_base64="wsjiahmjrcx+sbux77/in/b/kto=">aaackxicdzdnbhmxfixvbepl+gla2leznrjiuwvm0qptfkir2gsdvkqmjzqzrxecm2dfhlu2uzuazrvwjuzywkuwa7zseqjclc5c1cmvpp9jw74n14jbf8c/g9qdh1uptnce1588ffz8t7g3p7bqyrj1mrlkdho0jhhbfcedoke2hdixdjnp393kl1dklfffhvtqyitocj7ldj23xo3xw/btmjaprpfryljholhvyxi3v0aubjvw40yzbru7z/frj9yepbwv1oy+hjxox40/6usxhatcmyhwjpjyu6xe4zgtvnxthswnbi4zgmgsuzpjyhkpsc4s12opbuqywbkady0suvq7lpmgefp/dtnxau4wt/897qznwcklvxbumj+tsiz61pcwzxmyfvho/tiaa2/4mwwk0nf1jjezzvseq7qv566d8h4ytfuj5w/hze7jbu+wa6/gan5aaqfqhr6cqx8yfiiv8bw+bz+d78gp4nft0vrw784lwfpw+y+imqny</latexit> <latexit sha1_base64="wsjiahmjrcx+sbux77/in/b/kto=">aaackxicdzdnbhmxfixvbepl+gla2leznrjiuwvm0qptfkir2gsdvkqmjzqzrxecm2dfhlu2uzuazrvwjuzywkuwa7zseqjclc5c1cmvpp9jw74n14jbf8c/g9qdh1uptnce1588ffz8t7g3p7bqyrj1mrlkdho0jhhbfcedoke2hdixdjnp393kl1dklfffhvtqyitocj7ldj23xo3xw/btmjaprpfryljholhvyxi3v0aubjvw40yzbru7z/frj9yepbwv1oy+hjxox40/6usxhatcmyhwjpjyu6xe4zgtvnxthswnbi4zgmgsuzpjyhkpsc4s12opbuqywbkady0suvq7lpmgefp/dtnxau4wt/897qznwcklvxbumj+tsiz61pcwzxmyfvho/tiaa2/4mwwk0nf1jjezzvseq7qv566d8h4ytfuj5w/hze7jbu+wa6/gan5aaqfqhr6cqx8yfiiv8bw+bz+d78gp4nft0vrw784lwfpw+y+imqny</latexit> <latexit sha1_base64="71apnbk7xvasn2psbx0+ltea46o=">aaackxicdzdlsgmxgiuz3q23qks3gwvxiz2pipefilhxi1roa8ezyj/xt4ymk5ckyhnmdxwtd271jdypw7c+hgmtiyoesvhythlyn0rxzmwyvnlj4xotu9mzs4w5+yxfpelysspirqzyp5jl3uzaigcp1i2zhjtki4ie40xsoennf7eodznpzfyuxglakbtmfkyzwswnpn/kr1kkwn5okz0ixjl8y//z14bxk0d5q1gky9uhb+hoof8xkuvwobizqtoqfkzxknyfppzymoayeiobz6atoxzzqtq1qib2oi2xecpqqoosjvkg1b3r2gekak2cdcydctmqxvre8sfs/3/utktswejmr8ft9ugcsvr1labuzqpmgrpxpqwdblvbldjrbs3lddetcthyvaseuc1n2wfecpx8dod/d43tcsxx+w7peg9y1axzi+tkk1tipjkmp6rk6osse/jinsiz9+c9ek/e+/frmw94z5wmypv4aieoqyq=</latexit> <latexit sha1_base64="wsjiahmjrcx+sbux77/in/b/kto=">aaackxicdzdnbhmxfixvbepl+gla2leznrjiuwvm0qptfkir2gsdvkqmjzqzrxecm2dfhlu2uzuazrvwjuzywkuwa7zseqjclc5c1cmvpp9jw74n14jbf8c/g9qdh1uptnce1588ffz8t7g3p7bqyrj1mrlkdho0jhhbfcedoke2hdixdjnp393kl1dklfffhvtqyitocj7ldj23xo3xw/btmjaprpfryljholhvyxi3v0aubjvw40yzbru7z/frj9yepbwv1oy+hjxox40/6usxhatcmyhwjpjyu6xe4zgtvnxthswnbi4zgmgsuzpjyhkpsc4s12opbuqywbkady0suvq7lpmgefp/dtnxau4wt/897qznwcklvxbumj+tsiz61pcwzxmyfvho/tiaa2/4mwwk0nf1jjezzvseq7qv566d8h4ytfuj5w/hze7jbu+wa6/gan5aaqfqhr6cqx8yfiiv8bw+bz+d78gp4nft0vrw784lwfpw+y+imqny</latexit> <latexit sha1_base64="wsjiahmjrcx+sbux77/in/b/kto=">aaackxicdzdnbhmxfixvbepl+gla2leznrjiuwvm0qptfkir2gsdvkqmjzqzrxecm2dfhlu2uzuazrvwjuzywkuwa7zseqjclc5c1cmvpp9jw74n14jbf8c/g9qdh1uptnce1588ffz8t7g3p7bqyrj1mrlkdho0jhhbfcedoke2hdixdjnp393kl1dklfffhvtqyitocj7ldj23xo3xw/btmjaprpfryljholhvyxi3v0aubjvw40yzbru7z/frj9yepbwv1oy+hjxox40/6usxhatcmyhwjpjyu6xe4zgtvnxthswnbi4zgmgsuzpjyhkpsc4s12opbuqywbkady0suvq7lpmgefp/dtnxau4wt/897qznwcklvxbumj+tsiz61pcwzxmyfvho/tiaa2/4mwwk0nf1jjezzvseq7qv566d8h4ytfuj5w/hze7jbu+wa6/gan5aaqfqhr6cqx8yfiiv8bw+bz+d78gp4nft0vrw784lwfpw+y+imqny</latexit> <latexit sha1_base64="wsjiahmjrcx+sbux77/in/b/kto=">aaackxicdzdnbhmxfixvbepl+gla2leznrjiuwvm0qptfkir2gsdvkqmjzqzrxecm2dfhlu2uzuazrvwjuzywkuwa7zseqjclc5c1cmvpp9jw74n14jbf8c/g9qdh1uptnce1588ffz8t7g3p7bqyrj1mrlkdho0jhhbfcedoke2hdixdjnp393kl1dklfffhvtqyitocj7ldj23xo3xw/btmjaprpfryljholhvyxi3v0aubjvw40yzbru7z/frj9yepbwv1oy+hjxox40/6usxhatcmyhwjpjyu6xe4zgtvnxthswnbi4zgmgsuzpjyhkpsc4s12opbuqywbkady0suvq7lpmgefp/dtnxau4wt/897qznwcklvxbumj+tsiz61pcwzxmyfvho/tiaa2/4mwwk0nf1jjezzvseq7qv566d8h4ytfuj5w/hze7jbu+wa6/gan5aaqfqhr6cqx8yfiiv8bw+bz+d78gp4nft0vrw784lwfpw+y+imqny</latexit> <latexit sha1_base64="71apnbk7xvasn2psbx0+ltea46o=">aaackxicdzdlsgmxgiuz3q23qks3gwvxiz2pipefilhxi1roa8ezyj/xt4ymk5ckyhnmdxwtd271jdypw7c+hgmtiyoesvhythlyn0rxzmwyvnlj4xotu9mzs4w5+yxfpelysspirqzyp5jl3uzaigcp1i2zhjtki4ie40xsoennf7eodznpzfyuxglakbtmfkyzwswnpn/kr1kkwn5okz0ixjl8y//z14bxk0d5q1gky9uhb+hoof8xkuvwobizqtoqfkzxknyfppzymoayeiobz6atoxzzqtq1qib2oi2xecpqqoosjvkg1b3r2gekak2cdcydctmqxvre8sfs/3/utktswejmr8ft9ugcsvr1labuzqpmgrpxpqwdblvbldjrbs3lddetcthyvaseuc1n2wfecpx8dod/d43tcsxx+w7peg9y1axzi+tkk1tipjkmp6rk6osse/jinsiz9+c9ek/e+/frmw94z5wmypv4aieoqyq=</latexit> <latexit sha1_base64="nzkdwr8bn+sizapz0aba8gn6pn8=">aaacdhicdzc9sgnbfixv+m/8iz+dttbnbhy3mzikeai2nokcq4fkcbpjnq67szpmtmsw5bxsbpul7mtwd/adfajhqmuqhqb4oofegxsimtbtfp/nmzqemz2bx1gslc2vrk4v1zfotrgqigevivcdighmwiqbysbbjlriejtgrrqffuyxn6g0e+mzgukmormk7ipryqzvpap0dkshjd+tn/rfsu/wwk1/r1x6c1xxn6jc3okjtvrf996loeooqaej0bpb9aujm6imowmoc72hrklotabyjb4kelwydvbwngqujy2mhkmos8lfutajxosrj/6ynjjrvbkjerss6v+fm6tmmlfudg2m1gytzlxa2168oejlh/oor15fwspepl2eglz1okfkanfiugdr+emg9d+c19yq5dn6ub3/1rmswdbsqawq0ia2hmejbebbwd08wknz5zw5z87l1+iu872zctk5rx+h9jxj</latexit> <latexit sha1_base64="nzkdwr8bn+sizapz0aba8gn6pn8=">aaacdhicdzc9sgnbfixv+m/8iz+dttbnbhy3mzikeai2nokcq4fkcbpjnq67szpmtmsw5bxsbpul7mtwd/adfajhqmuqhqb4oofegxsimtbtfp/nmzqemz2bx1gslc2vrk4v1zfotrgqigevivcdighmwiqbysbbjlriejtgrrqffuyxn6g0e+mzgukmormk7ipryqzvpap0dkshjd+tn/rfsu/wwk1/r1x6c1xxn6jc3okjtvrf996loeooqaej0bpb9aujm6imowmoc72hrklotabyjb4kelwydvbwngqujy2mhkmos8lfutajxosrj/6ynjjrvbkjerss6v+fm6tmmlfudg2m1gytzlxa2168oejlh/oor15fwspepl2eglz1okfkanfiugdr+emg9d+c19yq5dn6ub3/1rmswdbsqawq0ia2hmejbebbwd08wknz5zw5z87l1+iu872zctk5rx+h9jxj</latexit> <latexit sha1_base64="nzkdwr8bn+sizapz0aba8gn6pn8=">aaacdhicdzc9sgnbfixv+m/8iz+dttbnbhy3mzikeai2nokcq4fkcbpjnq67szpmtmsw5bxsbpul7mtwd/adfajhqmuqhqb4oofegxsimtbtfp/nmzqemz2bx1gslc2vrk4v1zfotrgqigevivcdighmwiqbysbbjlriejtgrrqffuyxn6g0e+mzgukmormk7ipryqzvpap0dkshjd+tn/rfsu/wwk1/r1x6c1xxn6jc3okjtvrf996loeooqaej0bpb9aujm6imowmoc72hrklotabyjb4kelwydvbwngqujy2mhkmos8lfutajxosrj/6ynjjrvbkjerss6v+fm6tmmlfudg2m1gytzlxa2168oejlh/oor15fwspepl2eglz1okfkanfiugdr+emg9d+c19yq5dn6ub3/1rmswdbsqawq0ia2hmejbebbwd08wknz5zw5z87l1+iu872zctk5rx+h9jxj</latexit> <latexit sha1_base64="mo3mkv5aejat1zdydyqqkmupgpk=">aaacdhicdzdntgixfiu7+if4h7p0m5enbmygjailexi3bewwcyaejqrtl9gwntztx0gmvii7t/os7oxb38f38cgsiasknntx5zx7m9wtiogq7xkfvmzldw19i7uz29re2d3l7x+0fu8kaz/wimtuibvenazfux1bv0jaliyge44vvvpohuhfexytjwichkcxhvkctbf6zwl3xd63padsheqlnlou17ztur0mjcebqydmag3yn/0bthigssyrvqpx8oqouiw1jrfmc/1egcbkjefqw57aamsqjoaz0hkygbummqmvplolfsium6umlfwygda3i2bi+vjjup35xa9rqupjkwiiiclmmlq+mr244xckeds9nlgdyqxzk3ijrohezzhirhyd05yp57cd+39ol52s4atkoxe2lyqljtaxkqisqqigaqiw8hfbhd2ij/rspvgv1qv19joasey7h2hb1vsxdvccdw==</latexit> <latexit sha1_base64="6rcna8joelge7tmx5egk6ly1r+c=">aaaccxicdzc7sgnbfibpxlumt3jpbijpyppdjekleai2nokc0ybzwux4eoed2rlmjmjy8gr2tvosdmlru/goporj1givf6b4+p9zbs4fk86mdyi3rzaxotu9u5wtzc0vlc6vl1fojrxoii0qudttmbjklmwwzzzjw2kkiuz4esehn/nfdwrdzhpmhwojqfop6zfkrlpar7x2zuwgenbl1ade2n8ltvyrfygsb2nvm2sw1km3/n65knqgmlwue2muw0dzkcpamspxvoomdcpce9lhsxioolbhwr+lqkuh+dhhsgsakbufkwsziowzivipkyi9zpuxliklsfn1ue3trrll1cbisl02xsxvgvekn8so3wp+8ddvkme4m4zpicvbxxcqpak7hjvcpt8dvp6h80y9dhy6xw3ufpuervihdahbclvqhcm4grzq4hapd/do3xlp3rp38jva8l53vien7/udhygbrq==</latexit> <latexit sha1_base64="6rcna8joelge7tmx5egk6ly1r+c=">aaaccxicdzc7sgnbfibpxlumt3jpbijpyppdjekleai2nokc0ybzwux4eoed2rlmjmjy8gr2tvosdmlru/goporj1givf6b4+p9zbs4fk86mdyi3rzaxotu9u5wtzc0vlc6vl1fojrxoii0qudttmbjklmwwzzzjw2kkiuz4esehn/nfdwrdzhpmhwojqfop6zfkrlpar7x2zuwgenbl1ade2n8ltvyrfygsb2nvm2sw1km3/n65knqgmlwue2muw0dzkcpamspxvoomdcpce9lhsxioolbhwr+lqkuh+dhhsgsakbufkwsziowzivipkyi9zpuxliklsfn1ue3trrll1cbisl02xsxvgvekn8so3wp+8ddvkme4m4zpicvbxxcqpak7hjvcpt8dvp6h80y9dhy6xw3ufpuervihdahbclvqhcm4grzq4hapd/do3xlp3rp38jva8l53vien7/udhygbrq==</latexit> <latexit sha1_base64="6rcna8joelge7tmx5egk6ly1r+c=">aaaccxicdzc7sgnbfibpxlumt3jpbijpyppdjekleai2nokc0ybzwux4eoed2rlmjmjy8gr2tvosdmlru/goporj1givf6b4+p9zbs4fk86mdyi3rzaxotu9u5wtzc0vlc6vl1fojrxoii0qudttmbjklmwwzzzjw2kkiuz4esehn/nfdwrdzhpmhwojqfop6zfkrlpar7x2zuwgenbl1ade2n8ltvyrfygsb2nvm2sw1km3/n65knqgmlwue2muw0dzkcpamspxvoomdcpce9lhsxioolbhwr+lqkuh+dhhsgsakbufkwsziowzivipkyi9zpuxliklsfn1ue3trrll1cbisl02xsxvgvekn8so3wp+8ddvkme4m4zpicvbxxcqpak7hjvcpt8dvp6h80y9dhy6xw3ufpuervihdahbclvqhcm4grzq4hapd/do3xlp3rp38jva8l53vien7/udhygbrq==</latexit> <latexit sha1_base64="+zciipbhwrhrajqlv/tjurujnxk=">aaaccxicdzdntgixfiu7+if4h7p0q2sdg2ygjclchmqngxnmbcabcenucza006ytxsmej3dnvl/cnxhru/gopoqvcyheky6+nhnvk3siyag2nvfhzjawv1bxsuu5jc2t7z387l5bi7ei0ckccrvewaojmbqmnqwcqqdzieengl185507ujqk+nokekkohzeduikntyjgktgqnbf8fr7olsu1qndckyycx/amkqkzmv38z+9gkdgh2bcgte76njrhipwhhmek1xtrkjim8bc62jnyggrtiqgorixzscuyc9bhoj1nlkwx1zrh0yljsbmdnymhrgzh+s/nzlanuxrlsygy2gykqdvsthr3figyj7qxidux1ra3azdha/cux0qjxby4ydl6fjso/a/tstm3fhvsrj/oisqia3sisshhz6iogqijwogghh7re3p2hpwx59v5+xnnolodftqn5/0los6bwq==</latexit> Entropy Example: Fair coin X = {Heads, Tails} P [ Heads ] = 1 2 P [ Tails ] = 1 2 H(X) =1 Example: Biased coin X = {Heads, Tails} P [ Heads ] = 9 10 H(X) =0.47 P [ Tails ] = 1 10 There is more uncertainty when tossing a fair coin 29

7 <latexit sha1_base64="hix2siev+fa9ho42inc5khn6zr0=">aaacghicdzc9tisxeivn+bsq/sjfrrorhiq7c1wufeiragiqqckarfbrrjkek+u1zttxrna+cb0tvaqdoqxjhxgitiaiii5cfdpnxtkcvgxc2ch6cuzgx8yn/kxolaznzufmywulp0b2nkmgk5nu5ykaynhodcttrudke4o0o7o0u/+en/0nbbjmt2xfuskwk/m2z2i91sqvnjv220vlr9jsa2tub+hiqgivq1fty3cn2tyt/is4fg1urs/dqeet8mvzurkeonyydi25icnle4facpzruwr2dclkxezqbuykfenedugksro/hhvmuzbj3oc6odchmkyv0h+mqhs1bkzsdi2m5tvntr2toj6rnqwc+wyalmwy31hytun7x8pdfths3va3mtbds9ehqkalqxkssr6erw4qv8ppri32flxvrw9/9astsaprsa4x/im6hmarnicbg1u4g/vgjngihoonj9gr4hnncyyupl8bgycika==</latexit> <latexit sha1_base64="hix2siev+fa9ho42inc5khn6zr0=">aaacghicdzc9tisxeivn+bsq/sjfrrorhiq7c1wufeiragiqqckarfbrrjkek+u1zttxrna+cb0tvaqdoqxjhxgitiaiii5cfdpnxtkcvgxc2ch6cuzgx8yn/kxolaznzufmywulp0b2nkmgk5nu5ykaynhodcttrudke4o0o7o0u/+en/0nbbjmt2xfuskwk/m2z2i91sqvnjv220vlr9jsa2tub+hiqgivq1fty3cn2tyt/is4fg1urs/dqeet8mvzurkeonyydi25icnle4facpzruwr2dclkxezqbuykfenedugksro/hhvmuzbj3oc6odchmkyv0h+mqhs1bkzsdi2m5tvntr2toj6rnqwc+wyalmwy31hytun7x8pdfths3va3mtbds9ehqkalqxkssr6erw4qv8ppri32flxvrw9/9astsaprsa4x/im6hmarnicbg1u4g/vgjngihoonj9gr4hnncyyupl8bgycika==</latexit> <latexit sha1_base64="hix2siev+fa9ho42inc5khn6zr0=">aaacghicdzc9tisxeivn+bsq/sjfrrorhiq7c1wufeiragiqqckarfbrrjkek+u1zttxrna+cb0tvaqdoqxjhxgitiaiii5cfdpnxtkcvgxc2ch6cuzgx8yn/kxolaznzufmywulp0b2nkmgk5nu5ykaynhodcttrudke4o0o7o0u/+en/0nbbjmt2xfuskwk/m2z2i91sqvnjv220vlr9jsa2tub+hiqgivq1fty3cn2tyt/is4fg1urs/dqeet8mvzurkeonyydi25icnle4facpzruwr2dclkxezqbuykfenedugksro/hhvmuzbj3oc6odchmkyv0h+mqhs1bkzsdi2m5tvntr2toj6rnqwc+wyalmwy31hytun7x8pdfths3va3mtbds9ehqkalqxkssr6erw4qv8ppri32flxvrw9/9astsaprsa4x/im6hmarnicbg1u4g/vgjngihoonj9gr4hnncyyupl8bgycika==</latexit> <latexit sha1_base64="jmqif4cu4qfhrd3qrh4wtkj+oyw=">aaacghicdzc9sgnbfivn/y3xlypwnse0vtndqfeliwbjiyiyreiwche8iun2doazirigfre7w30jo7g18x18cmcyiygepvg4596be2kzmg2c4n2bmp6znzvpleqxl5zxvgtr6w0tbopinypeqksyncysxbphjserqrb4ngaz7p985c1bvjqj9nimjuyceinrmgrgwz3czlsqw82kx8v2vwg1+5kng6xtkaxlytfhshtu/athoriprmy67xq+2tecdjimhiagdssmpikskmnoglm+pdaogfahhy0ijehuke2h4gjucdj2majhhdnrdrohba71kmd/ta7mztkmhegbipwvz033mlislqodkxxzck1f164jvx+jco/5z0o/i53hbtj+agbvfa5ucv12movdpt8dfp+hrqucor7yk9wq46jyzitskx0skgnsi6fknnqjjzy8kefy5n17z96l9/o9ouwndzbihly3t57nodq=</latexit> <latexit sha1_base64="/wqqthaziedc8b8crr4+48q9gei=">aaacmnicdzdnthsxfixv0pixwggtu25gznnvzozw/emvinxtdrkvcerirtg1uumtjmew7abe1jwgb9idw/oq7q6x5sewgs5s1cmvpp/ja+kepkthxzr+jhzevv5cwl5zbay9ebu+0dx8d2rv2hdqcluq02noqrqvdz1wjfw0izsspdm2+vqyn12ssujvj26qqza4rmrachtb6jetxbs/qemvct4wyh1w+906zg/zwzixte38qjn68tqjc1hfvx6zlbz3dvbttwfxs8ja6uytzhbmdnxv3ucxio8lvy6xao15lmpxedro8jlqrj62pjgpcejnmgruzao/jcxjmel8hlbcsbbws73nqo/s2qlkl0yj7se8yzqaowt2n8/dyl/wotjjrxup2qx90rwhvwteyiqjkqnp0tfbcdvzperhk0qin8q2a9anum/fdul/w+loowv8/xors/vue6zab9igj5dbhntggxxdfzhcwtxcwk/oz/qnuo3unp4urm8z72fo0f0dfcmsaw==</latexit> <latexit sha1_base64="/wqqthaziedc8b8crr4+48q9gei=">aaacmnicdzdnthsxfixv0pixwggtu25gznnvzozw/emvinxtdrkvcerirtg1uumtjmew7abe1jwgb9idw/oq7q6x5sewgs5s1cmvpp/ja+kepkthxzr+jhzevv5cwl5zbay9ebu+0dx8d2rv2hdqcluq02noqrqvdz1wjfw0izsspdm2+vqyn12ssujvj26qqza4rmrachtb6jetxbs/qemvct4wyh1w+906zg/zwzixte38qjn68tqjc1hfvx6zlbz3dvbttwfxs8ja6uytzhbmdnxv3ucxio8lvy6xao15lmpxedro8jlqrj62pjgpcejnmgruzao/jcxjmel8hlbcsbbws73nqo/s2qlkl0yj7se8yzqaowt2n8/dyl/wotjjrxup2qx90rwhvwteyiqjkqnp0tfbcdvzperhk0qin8q2a9anum/fdul/w+loowv8/xors/vue6zab9igj5dbhntggxxdfzhcwtxcwk/oz/qnuo3unp4urm8z72fo0f0dfcmsaw==</latexit> <latexit sha1_base64="/wqqthaziedc8b8crr4+48q9gei=">aaacmnicdzdnthsxfixv0pixwggtu25gznnvzozw/emvinxtdrkvcerirtg1uumtjmew7abe1jwgb9idw/oq7q6x5sewgs5s1cmvpp/ja+kepkthxzr+jhzevv5cwl5zbay9ebu+0dx8d2rv2hdqcluq02noqrqvdz1wjfw0izsspdm2+vqyn12ssujvj26qqza4rmrachtb6jetxbs/qemvct4wyh1w+906zg/zwzixte38qjn68tqjc1hfvx6zlbz3dvbttwfxs8ja6uytzhbmdnxv3ucxio8lvy6xao15lmpxedro8jlqrj62pjgpcejnmgruzao/jcxjmel8hlbcsbbws73nqo/s2qlkl0yj7se8yzqaowt2n8/dyl/wotjjrxup2qx90rwhvwteyiqjkqnp0tfbcdvzperhk0qin8q2a9anum/fdul/w+loowv8/xors/vue6zab9igj5dbhntggxxdfzhcwtxcwk/oz/qnuo3unp4urm8z72fo0f0dfcmsaw==</latexit> <latexit sha1_base64="xzxvquiveq6ue5h+i3ddbzgtw2m=">aaacmnicdzc9timxfiu9/gbdx2dlbuakocrmaojpqopesw0ssaqimapo2twek+oxztsoktwpwzvq0cjd7hzowx5itqhfqby5+hyor6v7qmq5sxh8j5iznztfwkz8qc4tr6yu1dy3lo0caiytjnop2xqm5rzaluu2x7bscilmeex7j2/51r1qw2vxyuckmwg9gnc5a+utti1klxbdmjwo064g5pls7zvhepqehamgcui6updv12gy8ijsd2r1ulf9ebdvhizfiwney9xjrged2mt6i9laygfzdszcj7gymqntocuxrkydgwpyh3p4dbechtpzpzqcrr5nxx4leggyn957knqgjbkj+suuyg+ntspl3wi1nz633ypm8uinlbbmz2n0ucv49qi+re0pj05huud5w+9kohwsdimbtevt8fhwft0fhytfw+v2i/f8vltv7k2kqpbfzjnskytskyb5tc5iizbytx7je3kohok/wuvw7/3ptdcz+ummflz+b5swrbc=</latexit> <latexit sha1_base64="3wsbnpyne/ptvcdmoe5aurypmuc=">aaacfnicdzc9sgnbfixv+hvjx/zpbbbtweh2n4qaqgjy2agkrgpuembhaxwyszpmtmrlyxvy2epl2imtre/gqzhglaj4moljnhsh7kkvz8ag4zs3nj4xotvdminpzs0vlfawls+m7guklsq51o2ugoqsw5zllmnbasqi5xie9g4+8/mb1ibj7ntmchnbuhm7ypryz3uqk21/34+lanop+aw0xt+jb51knazvg3vhvsp/c1ethkraxiwhjjuv9/hs0r7azfjojlmiqmwtgmjlkmdboe4bvit2sbcvskiiqp0uxzqcrc5hy4czewisynjbsfgqyuwu0j+mipz61eyl7fmsml+f26u9pgcz6lvmqmugwaqt4xokeilq91hwlacd5qx3kwk4sxgbcek1ndwhg7kr56cd/384q9cixyfb1ebov09qgjvyhw2iybeacajh0aikodzdazx6d96t9+y9fi2oed87kzai7/udml6ghw==</latexit> <latexit sha1_base64="3wsbnpyne/ptvcdmoe5aurypmuc=">aaacfnicdzc9sgnbfixv+hvjx/zpbbbtweh2n4qaqgjy2agkrgpuembhaxwyszpmtmrlyxvy2epl2imtre/gqzhglaj4moljnhsh7kkvz8ag4zs3nj4xotvdminpzs0vlfawls+m7guklsq51o2ugoqsw5zllmnbasqi5xie9g4+8/mb1ibj7ntmchnbuhm7ypryz3uqk21/34+lanop+aw0xt+jb51knazvg3vhvsp/c1ethkraxiwhjjuv9/hs0r7azfjojlmiqmwtgmjlkmdboe4bvit2sbcvskiiqp0uxzqcrc5hy4czewisynjbsfgqyuwu0j+mipz61eyl7fmsml+f26u9pgcz6lvmqmugwaqt4xokeilq91hwlacd5qx3kwk4sxgbcek1ndwhg7kr56cd/384q9cixyfb1ebov09qgjvyhw2iybeacajh0aikodzdazx6d96t9+y9fi2oed87kzai7/udml6ghw==</latexit> <latexit sha1_base64="3wsbnpyne/ptvcdmoe5aurypmuc=">aaacfnicdzc9sgnbfixv+hvjx/zpbbbtweh2n4qaqgjy2agkrgpuembhaxwyszpmtmrlyxvy2epl2imtre/gqzhglaj4moljnhsh7kkvz8ag4zs3nj4xotvdminpzs0vlfawls+m7guklsq51o2ugoqsw5zllmnbasqi5xie9g4+8/mb1ibj7ntmchnbuhm7ypryz3uqk21/34+lanop+aw0xt+jb51knazvg3vhvsp/c1ethkraxiwhjjuv9/hs0r7azfjojlmiqmwtgmjlkmdboe4bvit2sbcvskiiqp0uxzqcrc5hy4czewisynjbsfgqyuwu0j+mipz61eyl7fmsml+f26u9pgcz6lvmqmugwaqt4xokeilq91hwlacd5qx3kwk4sxgbcek1ndwhg7kr56cd/384q9cixyfb1ebov09qgjvyhw2iybeacajh0aikodzdazx6d96t9+y9fi2oed87kzai7/udml6ghw==</latexit> <latexit sha1_base64="cehjlm76zmlbnfldnv3sa1y+9es=">aaacfnicdzc9tsmwfiwd8l/+aowsev0yujmakmvaqmjhqqkjqiusry57w6zasww7icjqe7cxwkuwivzw3oghwjqyhiojd5/oudfspalkvjsg+haqu9mzs3pzc9xfpewvvxdt/uqlgslqioij1u6xbkyzablqglslasxtbtdp/+q7v74dpanilk0uiea4l9eujdhyk3e32t6rfxxhjhexjjdaa0tdxk0f9d3dzrb36e1cwa9gqqgxzhp3m+oimucqgckw1jdhie1cyguoytcsrgmnepm+7sendiswookib4kduxk5tphhdjourrevwgjzrxoetpgcm9uymqrrnzjvfz433wzc0ewodgtezims/ja2dfn9fjr91d/l/uraw96kfyyn3psceyv03ekwauv57cd7h65266hli/3acwnc1dzarftog4xoab2ju3sowoighd2ij/tspdgvzqvz9jnaccy7g6gk5/0lt6ugmw==</latexit> <latexit sha1_base64="dz+efi10aajekl5c28i+jcawglo=">aaacdhicdzc7tgjbfibp4g3xhpfohkiddbugqshmsgxotdcrswibmjsecmlozmrmmjanr2bnqy9hz2x9b9/bh3aeldd4z4ov/3/ojocpzmi08bx3j7g0vlk6llxpbwxube+kd/eawowuxqyvovdtgggmwyqnw0yibamq8cdevjc8+mpbd6g0e9g1muj0orlerm8omdbq1hlj48x5ppgvlhrprjcvvsressxzfwp5b6zs9qbmqvfsh90bqucci0ndonwn4enjx0qzrkocprojjzlqirlgh3issfr+pedb0ajjymwxihy1h88uwghjwrwe8ocpyym5xtqdiyagbprx56zf9mmwyzhbinpshrhb0lyxdxigso+5lxo3j61hb9jusayoxu6oejpvczqy9fx0kpkfmrzjy1en2wpp3hmk4rcoiacfoimq1kaodaag4aee4cm5d56df+d1pppwvnf2yuho2ye5e5yb</latexit> <latexit sha1_base64="dz+efi10aajekl5c28i+jcawglo=">aaacdhicdzc7tgjbfibp4g3xhpfohkiddbugqshmsgxotdcrswibmjsecmlozmrmmjanr2bnqy9hz2x9b9/bh3aeldd4z4ov/3/ojocpzmi08bx3j7g0vlk6llxpbwxube+kd/eawowuxqyvovdtgggmwyqnw0yibamq8cdevjc8+mpbd6g0e9g1muj0orlerm8omdbq1hlj48x5ppgvlhrprjcvvsressxzfwp5b6zs9qbmqvfsh90bqucci0ndonwn4enjx0qzrkocprojjzlqirlgh3issfr+pedb0ajjymwxihy1h88uwghjwrwe8ocpyym5xtqdiyagbprx56zf9mmwyzhbinpshrhb0lyxdxigso+5lxo3j61hb9jusayoxu6oejpvczqy9fx0kpkfmrzjy1en2wpp3hmk4rcoiacfoimq1kaodaag4aee4cm5d56df+d1pppwvnf2yuho2ye5e5yb</latexit> <latexit sha1_base64="dz+efi10aajekl5c28i+jcawglo=">aaacdhicdzc7tgjbfibp4g3xhpfohkiddbugqshmsgxotdcrswibmjsecmlozmrmmjanr2bnqy9hz2x9b9/bh3aeldd4z4ov/3/ojocpzmi08bx3j7g0vlk6llxpbwxube+kd/eawowuxqyvovdtgggmwyqnw0yibamq8cdevjc8+mpbd6g0e9g1muj0orlerm8omdbq1hlj48x5ppgvlhrprjcvvsressxzfwp5b6zs9qbmqvfsh90bqucci0ndonwn4enjx0qzrkocprojjzlqirlgh3issfr+pedb0ajjymwxihy1h88uwghjwrwe8ocpyym5xtqdiyagbprx56zf9mmwyzhbinpshrhb0lyxdxigso+5lxo3j61hb9jusayoxu6oejpvczqy9fx0kpkfmrzjy1en2wpp3hmk4rcoiacfoimq1kaodaag4aee4cm5d56df+d1pppwvnf2yuho2ye5e5yb</latexit> <latexit sha1_base64="cdcjcl9kvdar9wc1ojpw5xybdxc=">aaacdhicdzdntgixfiu7+if4h7p0m5enbpgbdclchmqngxnmhccbcenuczzmp03bmzajr+dorb6eo+pwd/adfagr4miknntx5zx7m9wtijaq7bofvmzldw19i7uz29re2d3l7x+0fy8lay/wkmtugbwenajpux1cv0jalaihe4wvv/popuhfexsjpwj8hkcrhvkctbf6zelkxl6wk6vydzavukvkveae1u1lkjfcuqpoodyg/9m/5srmegksyqv6zvdop8fsuxlclnepfqhmxngepewkled6yqg4ay2n6dhghbkop5lflaotzjsasmdjzfjfpc2a87hggfrzur7w/irgityqezpnmxe8zxpxxgfi86hznxugwhjmjuwewmpeyzhirkogzzltz28h9v/qnk0avj4rnkqlorlocb2jiiqjc9ratdrchikio0f0hj6tb+vferxefkyz1mlnekvkvx8bpteclq==</latexit> <latexit sha1_base64="qf5nbglebgwhzxlye2dytf3enqu=">aaacdhicdzc7tsmwfizpyr3cymvjqehsliqttzigvwjhqqkjleptvdnmtfijy8t2evxek7cxwkuwivbegxfgitabhol45eht/59j6fyrtjg2vv/mfcymp6znzuek8wuls8ulldwwfknfmaaieaodey0jszewzctylgojjxk8iokjz/zigpvmij03i4khj4ou9rklxlqd4+rnvvmwxhd3g71sxxfrbw1/+6d8f2qun6vsxidcp73se/ds0chh1ncean2p+dkegvgg0qrvi92hrklotabyib4kelwydvbwngo0hltmcucdzvlfy2fgunyjhv0xotfx42ykrgxiph99bvqnmgophbpmqc1yzlxa2168oejlh/norl5pwspepl2eglzxokfkanfibdhw89nb+x9o1d2a5bodsnpvqyeyhq3yhcruyb+acayneaafaffwai/onfpkpdsvx6mf53tndcbkvh4awzochw==</latexit> <latexit sha1_base64="qf5nbglebgwhzxlye2dytf3enqu=">aaacdhicdzc7tsmwfizpyr3cymvjqehsliqttzigvwjhqqkjleptvdnmtfijy8t2evxek7cxwkuwivbegxfgitabhol45eht/59j6fyrtjg2vv/mfcymp6znzuek8wuls8ulldwwfknfmaaieaodey0jszewzctylgojjxk8iokjz/zigpvmij03i4khj4ou9rklxlqd4+rnvvmwxhd3g71sxxfrbw1/+6d8f2qun6vsxidcp73se/ds0chh1ncean2p+dkegvgg0qrvi92hrklotabyib4kelwydvbwngo0hltmcucdzvlfy2fgunyjhv0xotfx42ykrgxiph99bvqnmgophbpmqc1yzlxa2168oejlh/norl5pwspepl2eglzxokfkanfibdhw89nb+x9o1d2a5bodsnpvqyeyhq3yhcruyb+acayneaafaffwai/onfpkpdsvx6mf53tndcbkvh4awzochw==</latexit> <latexit sha1_base64="qf5nbglebgwhzxlye2dytf3enqu=">aaacdhicdzc7tsmwfizpyr3cymvjqehsliqttzigvwjhqqkjleptvdnmtfijy8t2evxek7cxwkuwivbegxfgitabhol45eht/59j6fyrtjg2vv/mfcymp6znzuek8wuls8ulldwwfknfmaaieaodey0jszewzctylgojjxk8iokjz/zigpvmij03i4khj4ou9rklxlqd4+rnvvmwxhd3g71sxxfrbw1/+6d8f2qun6vsxidcp73se/ds0chh1ncean2p+dkegvgg0qrvi92hrklotabyib4kelwydvbwngo0hltmcucdzvlfy2fgunyjhv0xotfx42ykrgxiph99bvqnmgophbpmqc1yzlxa2168oejlh/norl5pwspepl2eglzxokfkanfibdhw89nb+x9o1d2a5bodsnpvqyeyhq3yhcruyb+acayneaafaffwai/onfpkpdsvx6mf53tndcbkvh4awzochw==</latexit> <latexit sha1_base64="k9jw1zuxwmvowxl2vt2hbteqstw=">aaacdhicdzc7tgjbfizn8yz4qy1tntjgsxe8qwfcyknjgolcetiq2fgak93zmcwmbrlhfexs9sxsjk3v4dv4ei6ibrl/tphl/8+z5pyhiknsnvdhzzawv1bxsuu5jc2t7z387l5t8aek0ca85ridyguxtachqy6hlsrgfsbqcqpl77x1d1jrntzosyca4ufc+5rgbaxortg6si/sknna7uulnloqll3jir0ivunnvuaz1xv5z+4tj0mgisyxvqrje0ihkzaakhgmue5qgcakwgpoye9gatjib8azadmejw0mmiek0ulfc2gkmvjjfi6ydou7etpkpni4vh8+1/1yknjeddukxgrttn2gmr24uqjspopejd2emia5sbkx1jbygsask8fgjgfq+e3a/h+ajcc3fh1sqj7nisqia3siishh56ikaqiogoggjh7re3q2hqwx69v6+xnnwlodftqn6/0lsi+cmw==</latexit> <latexit sha1_base64="3wsbnpyne/ptvcdmoe5aurypmuc=">aaacfnicdzc9sgnbfixv+hvjx/zpbbbtweh2n4qaqgjy2agkrgpuembhaxwyszpmtmrlyxvy2epl2imtre/gqzhglaj4moljnhsh7kkvz8ag4zs3nj4xotvdminpzs0vlfawls+m7guklsq51o2ugoqsw5zllmnbasqi5xie9g4+8/mb1ibj7ntmchnbuhm7ypryz3uqk21/34+lanop+aw0xt+jb51knazvg3vhvsp/c1ethkraxiwhjjuv9/hs0r7azfjojlmiqmwtgmjlkmdboe4bvit2sbcvskiiqp0uxzqcrc5hy4czewisynjbsfgqyuwu0j+mipz61eyl7fmsml+f26u9pgcz6lvmqmugwaqt4xokeilq91hwlacd5qx3kwk4sxgbcek1ndwhg7kr56cd/384q9cixyfb1ebov09qgjvyhw2iybeacajh0aikodzdazx6d96t9+y9fi2oed87kzai7/udml6ghw==</latexit> <latexit sha1_base64="3wsbnpyne/ptvcdmoe5aurypmuc=">aaacfnicdzc9sgnbfixv+hvjx/zpbbbtweh2n4qaqgjy2agkrgpuembhaxwyszpmtmrlyxvy2epl2imtre/gqzhglaj4moljnhsh7kkvz8ag4zs3nj4xotvdminpzs0vlfawls+m7guklsq51o2ugoqsw5zllmnbasqi5xie9g4+8/mb1ibj7ntmchnbuhm7ypryz3uqk21/34+lanop+aw0xt+jb51knazvg3vhvsp/c1ethkraxiwhjjuv9/hs0r7azfjojlmiqmwtgmjlkmdboe4bvit2sbcvskiiqp0uxzqcrc5hy4czewisynjbsfgqyuwu0j+mipz61eyl7fmsml+f26u9pgcz6lvmqmugwaqt4xokeilq91hwlacd5qx3kwk4sxgbcek1ndwhg7kr56cd/384q9cixyfb1ebov09qgjvyhw2iybeacajh0aikodzdazx6d96t9+y9fi2oed87kzai7/udml6ghw==</latexit> <latexit sha1_base64="3wsbnpyne/ptvcdmoe5aurypmuc=">aaacfnicdzc9sgnbfixv+hvjx/zpbbbtweh2n4qaqgjy2agkrgpuembhaxwyszpmtmrlyxvy2epl2imtre/gqzhglaj4moljnhsh7kkvz8ag4zs3nj4xotvdminpzs0vlfawls+m7guklsq51o2ugoqsw5zllmnbasqi5xie9g4+8/mb1ibj7ntmchnbuhm7ypryz3uqk21/34+lanop+aw0xt+jb51knazvg3vhvsp/c1ethkraxiwhjjuv9/hs0r7azfjojlmiqmwtgmjlkmdboe4bvit2sbcvskiiqp0uxzqcrc5hy4czewisynjbsfgqyuwu0j+mipz61eyl7fmsml+f26u9pgcz6lvmqmugwaqt4xokeilq91hwlacd5qx3kwk4sxgbcek1ndwhg7kr56cd/384q9cixyfb1ebov09qgjvyhw2iybeacajh0aikodzdazx6d96t9+y9fi2oed87kzai7/udml6ghw==</latexit> <latexit sha1_base64="cehjlm76zmlbnfldnv3sa1y+9es=">aaacfnicdzc9tsmwfiwd8l/+aowsev0yujmakmvaqmjhqqkjqiusry57w6zasww7icjqe7cxwkuwivzw3oghwjqyhiojd5/oudfspalkvjsg+haqu9mzs3pzc9xfpewvvxdt/uqlgslqioij1u6xbkyzablqglslasxtbtdp/+q7v74dpanilk0uiea4l9eujdhyk3e32t6rfxxhjhexjjdaa0tdxk0f9d3dzrb36e1cwa9gqqgxzhp3m+oimucqgckw1jdhie1cyguoytcsrgmnepm+7sendiswookib4kduxk5tphhdjourrevwgjzrxoetpgcm9uymqrrnzjvfz433wzc0ewodgtezims/ja2dfn9fjr91d/l/uraw96kfyyn3psceyv03ekwauv57cd7h65266hli/3acwnc1dzarftog4xoab2ju3sowoighd2ij/tspdgvzqvz9jnaccy7g6gk5/0lt6ugmw==</latexit> <latexit sha1_base64="vzwrk/uckte7knqmn5jkbjhjnj4=">aaacqhicdzblbxmxfixvtdxkeixcjs2ibfhumznyaitvissgdvkrsfuphkxx7k2wyo8t20gjrpk1/sfdsyuv/4afemllcjefrag48uk759iw7ufwsr/k8mu2tn7t+o2bg7c6t+/cvxe/+2dz0juzezqurhl3zngtkg0ngwykjq0j1fzrez++vsippplz0jtvw8jsrxhsyleugji16u4z6+k8zfdznnyoytxgqmxztsf2cqa5mcexce3fom+zbhiwqy2mtk3wwy9yo+r2yv727k75bde/clw/xko3earlhyy639mpetnntrakvt+pshvqic5ioajtsjkni2kkezrb0qilv8cjgu3blvbjha1q8nvctrasrttelzs/ymomh1znbsw0ipf/fb7go3wujz0faktklhilou9dflnolh1zvf0ui5umtjmvfaaafxqfm76fso2kev52kp8fdrf7vej3z3udl5c9wqy8hifwfcp4bqn4awcwbafn8ak+w5fsppuw/ch+xl5dy/68eqgryn79bthtsbo=</latexit> <latexit sha1_base64="vzwrk/uckte7knqmn5jkbjhjnj4=">aaacqhicdzblbxmxfixvtdxkeixcjs2ibfhumznyaitvissgdvkrsfuphkxx7k2wyo8t20gjrpk1/sfdsyuv/4afemllcjefrag48uk759iw7ufwsr/k8mu2tn7t+o2bg7c6t+/cvxe/+2dz0juzezqurhl3zngtkg0ngwykjq0j1fzrez++vsippplz0jtvw8jsrxhsyleugji16u4z6+k8zfdznnyoytxgqmxztsf2cqa5mcexce3fom+zbhiwqy2mtk3wwy9yo+r2yv727k75bde/clw/xko3earlhyy639mpetnntrakvt+pshvqic5ioajtsjkni2kkezrb0qilv8cjgu3blvbjha1q8nvctrasrttelzs/ymomh1znbsw0ipf/fb7go3wujz0faktklhilou9dflnolh1zvf0ui5umtjmvfaaafxqfm76fso2kev52kp8fdrf7vej3z3udl5c9wqy8hifwfcp4bqn4awcwbafn8ak+w5fsppuw/ch+xl5dy/68eqgryn79bthtsbo=</latexit> <latexit sha1_base64="vzwrk/uckte7knqmn5jkbjhjnj4=">aaacqhicdzblbxmxfixvtdxkeixcjs2ibfhumznyaitvissgdvkrsfuphkxx7k2wyo8t20gjrpk1/sfdsyuv/4afemllcjefrag48uk759iw7ufwsr/k8mu2tn7t+o2bg7c6t+/cvxe/+2dz0juzezqurhl3zngtkg0ngwykjq0j1fzrez++vsippplz0jtvw8jsrxhsyleugji16u4z6+k8zfdznnyoytxgqmxztsf2cqa5mcexce3fom+zbhiwqy2mtk3wwy9yo+r2yv727k75bde/clw/xko3earlhyy639mpetnntrakvt+pshvqic5ioajtsjkni2kkezrb0qilv8cjgu3blvbjha1q8nvctrasrttelzs/ymomh1znbsw0ipf/fb7go3wujz0faktklhilou9dflnolh1zvf0ui5umtjmvfaaafxqfm76fso2kev52kp8fdrf7vej3z3udl5c9wqy8hifwfcp4bqn4awcwbafn8ak+w5fsppuw/ch+xl5dy/68eqgryn79bthtsbo=</latexit> <latexit sha1_base64="csxc8jrmbusfn59ucvq4jje/1hq=">aaacqhicdza/t1sxfmx9ofbikqq6dnlqlg4o7z3+fzcqkfi6vkjsa0j4kbo2n6kv+9myhztiep+gb8lwtz34bh0qoa6datj0cigjd797jm3phmakcd7p75k5+rcliy+xlhuvvl6vrjxxn86chlqoha6lthcmhepryccll/hcwatfjj6zwcldfn6n1gldfffjg6wcfiv6goopvrd5ri0nozo9smnpag9fhyq8tukhpuypynouetrwd+mopajkasg2qbzs3m3u0rrbboxtryp9fpsgfqpfo5+oray67tz/0svnhworzyu4d1nkxpcbrbdcyt2gq4cg+ad6eam5ayo2dh3ucr0dz8yrk1doyjbpysymojwbk/bevoc/zppm64eh5h597nv7zrcvgxqsemwmglkoi11ma4y2hpf9gmdde424k8skebxlcrjvrh2xbsr6/neqpg9nw+0i8ued1vhetkgl8pa8i+9jqt6qy/krnjio4esgfcpfyy/knvmz3ce//12ds6zv3pazjx/+al3jswy=</latexit> Entropy Example: Fair dice X = {1,...6} P[x ]= 1 6 x œ X H(x) =2.58 Example: Biased dice P[6]= 5 10 X = {1,...6} P[x ]= 1 10 H(x) =2.16 x œ {1,...,5} There is more uncertainty when tossing a fair dice 30

8 <latexit sha1_base64="98welh48otnrglx4uwp6aaz+rgg=">aaacuhicdzdnbhmxfebvhj+wleka7timykarzeydaltaissgtauikbrszxrduzeplfhy2b5ezm1t8sbsuoutd8coookilfrcwflrd2xl/pguhxvpehu1hjx89hhn90l77+n+s+edfy9hvtwg05crupllhpzkudhqcvfsptaekpv0wwyffv7ikxkrvpxzztuveqevmaiolktjzlmujccmfh/ne4pcz43p0ibov9r4hs8c++f6245v3xt3umnv6pqk7z/g9yhrpcvpdg5goefjzu/8wvfauuv4idzezal2hufjbc+paee1jy18hlo6wlsjjlp4ksljzsw3dcakjdncl1vzkb6ltxpj7ous3c1myjsaowr263e3osm8qhttqolbldenqxu6twamtfgiozsnyx2c8ceblojowykrg2v7azt2qgfdqfx/gb31sscf3nyh71y9ws68hjdwcbkcwwa+wjkmgcn3uiwf8cv6ef2j/rai1dh1dq9gy1rtowystja=</latexit> <latexit sha1_base64="98welh48otnrglx4uwp6aaz+rgg=">aaacuhicdzdnbhmxfebvhj+wleka7timykarzeydaltaissgtauikbrszxrduzeplfhy2b5ezm1t8sbsuoutd8coookilfrcwflrd2xl/pguhxvpehu1hjx89hhn90l77+n+s+edfy9hvtwg05crupllhpzkudhqcvfsptaekpv0wwyffv7ikxkrvpxzztuveqevmaiolktjzlmujccmfh/ne4pcz43p0ibov9r4hs8c++f6245v3xt3umnv6pqk7z/g9yhrpcvpdg5goefjzu/8wvfauuv4idzezal2hufjbc+paee1jy18hlo6wlsjjlp4ksljzsw3dcakjdncl1vzkb6ltxpj7ous3c1myjsaowr263e3osm8qhttqolbldenqxu6twamtfgiozsnyx2c8ceblojowykrg2v7azt2qgfdqfx/gb31sscf3nyh71y9ws68hjdwcbkcwwa+wjkmgcn3uiwf8cv6ef2j/rai1dh1dq9gy1rtowystja=</latexit> <latexit sha1_base64="98welh48otnrglx4uwp6aaz+rgg=">aaacuhicdzdnbhmxfebvhj+wleka7timykarzeydaltaissgtauikbrszxrduzeplfhy2b5ezm1t8sbsuoutd8coookilfrcwflrd2xl/pguhxvpehu1hjx89hhn90l77+n+s+edfy9hvtwg05crupllhpzkudhqcvfsptaekpv0wwyffv7ikxkrvpxzztuveqevmaiolktjzlmujccmfh/ne4pcz43p0ibov9r4hs8c++f6245v3xt3umnv6pqk7z/g9yhrpcvpdg5goefjzu/8wvfauuv4idzezal2hufjbc+paee1jy18hlo6wlsjjlp4ksljzsw3dcakjdncl1vzkb6ltxpj7ous3c1myjsaowr263e3osm8qhttqolbldenqxu6twamtfgiozsnyx2c8ceblojowykrg2v7azt2qgfdqfx/gb31sscf3nyh71y9ws68hjdwcbkcwwa+wjkmgcn3uiwf8cv6ef2j/rai1dh1dq9gy1rtowystja=</latexit> <latexit sha1_base64="ka+cops08cdynwavbxug1/jdoyc=">aaacuhicdzdnthsxfebvhp6etjsultmmmk1xmrlafbavirhpbgmkbpcyuxrtbokv8djynqqrnu/fm7bjs7d9go6ksufvgnplyuffss35y7ou1qxpbdraefhy1ev2aufn27v36933gydw1ybtkktsmtoglkpr0dajv9kznosslxtkpgcp/vq7gstu9c3nnbusj5uyc44urkpuya6nxyb+eudjg9xnjc/sjs6varyihxz757axhv++n+r20v7w/l66vr8/h6yfzqchj3m06v7klxsvjvwol2jtezzqv3g0tvcsmk5ew9lipzihc0w1ajkfn5cs5mxsuqesujit/lyvbelrwjut7fko0v0uhkypqunmlx53473ci0rxjioe3bx9mrsh22tkyiqlksnzmtihch+ysymofiqsuvg2h7dphhqeooj/dydb/szw8u5vsptyvbs24sn8ggw+wwc+whemgcm13mid/ixuot/rn1b09+jtdh9gyvqde4tftdw=</latexit> <latexit sha1_base64="itbmxkdqjxngqrpgp5hgxzn1xu8=">aaacnhicdzdnshxbfezva4xmnmnezjdn42xctxcb8wcrghcttudbucfuhts1dybfdhwvvdxiuprz+cbu3oozblil2qwfwbjncbpxuovdd6ok7perghsbx9+dufkxcy8xl161lldev3nbfrd6zgslgfwzlkq+ydfqwuvqw24lolgauoqfheetvqd/fe7acfke2qmitoc45cpo0ppo0e5spd2wdj+h6ugjc5u1s+i6tm8qhiypjv65phgddifubuzuxj92w6eqdonmor0p0mz+op07hupwcsotk9cy0yrwnnoolwcf1a20mqsqtxbmpxgrvkqznyypyorprpzyoictuwbzgelqgdmv+znqop02g+zstizm5r/p7wgnc7xulawsedegi/rg9xdnctl+8ojrnbooh/idtfsgpytiinpsdd3wlv/p3w7c5+foo5t4ptjs9lyee4il+ahrsa4jbempvsa+9ihbjvzdddwgv8gp4gfw6/hqxpdnzxuymeduhshkruy=</latexit> <latexit sha1_base64="itbmxkdqjxngqrpgp5hgxzn1xu8=">aaacnhicdzdnshxbfezva4xmnmnezjdn42xctxcb8wcrghcttudbucfuhts1dybfdhwvvdxiuprz+cbu3oozblil2qwfwbjncbpxuovdd6ok7perghsbx9+dufkxcy8xl161lldev3nbfrd6zgslgfwzlkq+ydfqwuvqw24lolgauoqfheetvqd/fe7acfke2qmitoc45cpo0ppo0e5spd2wdj+h6ugjc5u1s+i6tm8qhiypjv65phgddifubuzuxj92w6eqdonmor0p0mz+op07hupwcsotk9cy0yrwnnoolwcf1a20mqsqtxbmpxgrvkqznyypyorprpzyoictuwbzgelqgdmv+znqop02g+zstizm5r/p7wgnc7xulawsedegi/rg9xdnctl+8ojrnbooh/idtfsgpytiinpsdd3wlv/p3w7c5+foo5t4ptjs9lyee4il+ahrsa4jbempvsa+9ihbjvzdddwgv8gp4gfw6/hqxpdnzxuymeduhshkruy=</latexit> <latexit sha1_base64="itbmxkdqjxngqrpgp5hgxzn1xu8=">aaacnhicdzdnshxbfezva4xmnmnezjdn42xctxcb8wcrghcttudbucfuhts1dybfdhwvvdxiuprz+cbu3oozblil2qwfwbjncbpxuovdd6ok7perghsbx9+dufkxcy8xl161lldev3nbfrd6zgslgfwzlkq+ydfqwuvqw24lolgauoqfheetvqd/fe7acfke2qmitoc45cpo0ppo0e5spd2wdj+h6ugjc5u1s+i6tm8qhiypjv65phgddifubuzuxj92w6eqdonmor0p0mz+op07hupwcsotk9cy0yrwnnoolwcf1a20mqsqtxbmpxgrvkqznyypyorprpzyoictuwbzgelqgdmv+znqop02g+zstizm5r/p7wgnc7xulawsedegi/rg9xdnctl+8ojrnbooh/idtfsgpytiinpsdd3wlv/p3w7c5+foo5t4ptjs9lyee4il+ahrsa4jbempvsa+9ihbjvzdddwgv8gp4gfw6/hqxpdnzxuymeduhshkruy=</latexit> <latexit sha1_base64="xf10p1lix2hvjsj2vcgk4ioypvq=">aaacnhicdzdlahsxfiy16c110naallmz6k1wnhk3ppaiymgmm0ac9qu8gzkjh7vco5eqauqnmofim2txbfomhe5kdqhpumvxci7jqyup/5me589kzrsjot/e1rpnl16+qr2ub++8efvof7/b16jufhtu5einm9cyswj7hpkch1ih8czhqty/vvwd76g0e8uxs5cycpgvbmoogben/tiryk6q4hoqtbvqe1jzokqc5fsjk+dw4yoll27sn6jmq9oopnactyib0xiazdvny/8mmqhaciwmzuhrurxjk1pqhtecq3psapra5zddeuqsjkruzlbwngqxrh0wwfgndrn5mrtatv7wbcpkyl6uh5kqcwozfvs5mbztywpzgiyoc0u0yu+7/sj5hsodfp4uwrf0gdtjhzky/bfyoeropsoq7uq57yb4gvqtzuz4/ldrpvovvsp75am5idh5rlrkhjyrhqhkgvwkv+sxd+n98f5613dxt7zvmz2ynt6//0cmrpi=</latexit> <latexit sha1_base64="0ft7jxjmfuzoo3h4sesqtqbipem=">aaacfnicdzc9ahtbfixvok4ik39ylc7nejupgnzlh0qqdii0bgikrd+gxctd0zuyagznmbkfl4vew51b+yxcbbdu/q5+ieykufbedln8nhpvwd2zfty6ol4l9h7tp37ythjqffb8xctxtcpxa6uwhlgfkahmkenlgufud9wjgmldkdnbw2zx5u8+/engcpv/d4wmvoi85zpo0hlruqupwpmwkffdmiipcjakzdwpnelmuacdh3fcxwg147ua3tewvm9su0+mii0l5y4jthbcirvlszsom0grark0pjetce5jjdvqmmk5jyxjmwi79pijjjuw69u2whkltyxmdkyj7se2msm1cjjzfz53s3za8lwvhexmz2sso771duwljix/ppparbptdx+tjqq6ooskmqns0+oq6ut56cd8pwyomi3p3z42up82puef3si7ea8t+axdoiue9ifbardwcvfbexad/apunqn7wd+domwpup0nzp6g5g==</latexit> <latexit sha1_base64="0ft7jxjmfuzoo3h4sesqtqbipem=">aaacfnicdzc9ahtbfixvok4ik39ylc7nejupgnzlh0qqdii0bgikrd+gxctd0zuyagznmbkfl4vew51b+yxcbbdu/q5+ieykufbedln8nhpvwd2zfty6ol4l9h7tp37ythjqffb8xctxtcpxa6uwhlgfkahmkenlgufud9wjgmldkdnbw2zx5u8+/engcpv/d4wmvoi85zpo0hlruqupwpmwkffdmiipcjakzdwpnelmuacdh3fcxwg147ua3tewvm9su0+mii0l5y4jthbcirvlszsom0grark0pjetce5jjdvqmmk5jyxjmwi79pijjjuw69u2whkltyxmdkyj7se2msm1cjjzfz53s3za8lwvhexmz2sso771duwljix/ppparbptdx+tjqq6ooskmqns0+oq6ut56cd8pwyomi3p3z42up82puef3si7ea8t+axdoiue9ifbardwcvfbexad/apunqn7wd+domwpup0nzp6g5g==</latexit> <latexit sha1_base64="0ft7jxjmfuzoo3h4sesqtqbipem=">aaacfnicdzc9ahtbfixvok4ik39ylc7nejupgnzlh0qqdii0bgikrd+gxctd0zuyagznmbkfl4vew51b+yxcbbdu/q5+ieykufbedln8nhpvwd2zfty6ol4l9h7tp37ythjqffb8xctxtcpxa6uwhlgfkahmkenlgufud9wjgmldkdnbw2zx5u8+/engcpv/d4wmvoi85zpo0hlruqupwpmwkffdmiipcjakzdwpnelmuacdh3fcxwg147ua3tewvm9su0+mii0l5y4jthbcirvlszsom0grark0pjetce5jjdvqmmk5jyxjmwi79pijjjuw69u2whkltyxmdkyj7se2msm1cjjzfz53s3za8lwvhexmz2sso771duwljix/ppparbptdx+tjqq6ooskmqns0+oq6ut56cd8pwyomi3p3z42up82puef3si7ea8t+axdoiue9ifbardwcvfbexad/apunqn7wd+domwpup0nzp6g5g==</latexit> <latexit sha1_base64="tklpoqe+hrlpu0nbsl3g7issmuw=">aaacfnicdzc9sgnbfivn41+mf1flm8u0fpldrpgneai2nokciyhseu6o1zhkzmeymyjlkvews9wxsbnbw9/bh3cmsyjiyyqpc+4ducdrnbkbhu9eywp6znauof9awfxaximvrrwmhgiktsq51o0edhkwytmyy7gtnijiof4m/eov/piwtweyvbczwlhal2xxjij1vre83vap/cihbt/iv9iah6nht1wjq/xdg3dn0p8ltwo4uowmddytf0rxkg4eppzymkztc5wnc9cwuy7dujqwqid2oycdcbuo1hheqynq6mwydpicqbpno9smwhyemzli/pgc7m2kmujzt5cyx5/b64m4z6kaweypy0ayb03jggr6cwr3whcabv3ldhetcthyvaseuc1n1egw5or56cd/h1r1as3x+w6lstcuqkg2ycbzijwytxrkhjyrjqekiw/kktx5996z9+k9fo8wvphoopmq9/yju/qgkg==</latexit> <latexit sha1_base64="n6pej54tfpssdi4kdu8b81rwooo=">aaacdhicdzc7sgnbfibpeo3xfi+dttcnnrubeeskiwctrojgmkbcwux4jmpu7awzezesequ7w30jo7h1hxwhh8ix0skkp1n8/p85a+epzmk08f13z2p6znzupreqx1xaxlktrk1fanfxfamqeqhaedgysbqdw0ycbamq8cjbvhsffowtw1saiftcdcsgnprsds0omdbqnhbudovhxyrrh3qljd+t1kr+xq34f8qup1kpvgkjnbufj8srqfscu0mtonwn7estzkqzrhmc5i/7giwhmelhh/issfrh1kpb0ajbzgwxjrx1mi0umggzwrue8oipyym5mtqjiwjdiv3rc3nddtowyr7blnpshjkxanulf0eo7gpe6cdrsmvym7sxein3hiduce1ahoztpt8dfp+hi4pbtny2x6ofjnuchgzbnuxagy6gdg1oqgaubdzaizw5986z8+k8jkenno+ddziq8/yjtj6cfw==</latexit> <latexit sha1_base64="n6pej54tfpssdi4kdu8b81rwooo=">aaacdhicdzc7sgnbfibpeo3xfi+dttcnnrubeeskiwctrojgmkbcwux4jmpu7awzezesequ7w30jo7h1hxwhh8ix0skkp1n8/p85a+epzmk08f13z2p6znzupreqx1xaxlktrk1fanfxfamqeqhaedgysbqdw0ycbamq8cjbvhsffowtw1saiftcdcsgnprsds0omdbqnhbudovhxyrrh3qljd+t1kr+xq34f8qup1kpvgkjnbufj8srqfscu0mtonwn7estzkqzrhmc5i/7giwhmelhh/issfrh1kpb0ajbzgwxjrx1mi0umggzwrue8oipyym5mtqjiwjdiv3rc3nddtowyr7blnpshjkxanulf0eo7gpe6cdrsmvym7sxein3hiduce1ahoztpt8dfp+hi4pbtny2x6ofjnuchgzbnuxagy6gdg1oqgaubdzaizw5986z8+k8jkenno+ddziq8/yjtj6cfw==</latexit> <latexit sha1_base64="n6pej54tfpssdi4kdu8b81rwooo=">aaacdhicdzc7sgnbfibpeo3xfi+dttcnnrubeeskiwctrojgmkbcwux4jmpu7awzezesequ7w30jo7h1hxwhh8ix0skkp1n8/p85a+epzmk08f13z2p6znzupreqx1xaxlktrk1fanfxfamqeqhaedgysbqdw0ycbamq8cjbvhsffowtw1saiftcdcsgnprsds0omdbqnhbudovhxyrrh3qljd+t1kr+xq34f8qup1kpvgkjnbufj8srqfscu0mtonwn7estzkqzrhmc5i/7giwhmelhh/issfrh1kpb0ajbzgwxjrx1mi0umggzwrue8oipyym5mtqjiwjdiv3rc3nddtowyr7blnpshjkxanulf0eo7gpe6cdrsmvym7sxein3hiduce1ahoztpt8dfp+hi4pbtny2x6ofjnuchgzbnuxagy6gdg1oqgaubdzaizw5986z8+k8jkenno+ddziq8/yjtj6cfw==</latexit> <latexit sha1_base64="hy4198yi38ygz0udqfqax426auq=">aaacdhicdzdntgixfiu7+if4h7p0q2sdm5kb/2bhqukgjqkmdpdahhtqbzuztpu2gmiev3dnvl/cnxhro/gopoqvctest7r4cs69te4jresvdt0pk7o0vlk6ll3pbwxube/kd/daio8kay/wimtogbvenazpux1br0jaliighysx33n7hqsipl7rewe+w8oydijb2ljdrml8vlgovgz3tj8vunalvnwpa4vfknvuteu0v7of/+zdcjjiegssyaw6zvdop8fsuxlbnncbkrcyhhgixewkled6yra4ay0n6dhgjbkop5ldlaotzjsasgdbzfjfpc2a81djqp35xa+qfkjjmdiqe5pnmhe8zxpxwgckeds5mjh9yqxzk3iirghsmewkv7bbac7u89tb4x9oveyy4eutyv1sxlqwhabdvejldi7qqigayemecfsintcz9wc9wk/w289oxprv7kourpcvo5qckw==</latexit> Entropy Example: Text documents with terms a,, e P[a ]= 1 10 X = {a,... e} P[d ]= 4 10 P[b ]= 3 10 H(x) =2.05 P[e ]= 1 10 P[c ]= 1 10 What is the connection between entropy and compression? 31

9 Entropy and Compression Entropy provides us with a lower bound on the average number of bits (i.e., average code length) that any prefix-free encoding of the random variable s outcomes will require Any prefix-free encoding will thus require at least H(X ) bits on average to encode outcomes of X 32

10 Huffman Codes Huffman coding assigns shorter codes to more frequent symbols and longer codes to less frequent symbols It can be shown that Huffman codes are optimal, under certain assumptions, i.e., the average number of bits required corresponds to the entropy of the random variable 33

11 Huffman Tree Huffman codes are computed by constructing a Huffman tree each symbol corresponds to a leaf node in the tree edges to the left and right child of an inner node are labeled with 0 and 1, respectively the code of a symbol is obtained as the sequence of labels on the path from the root node to the corresponding leaf node 34

12 Huffman Tree Huffman tree is constructed in a bottom-up fashion as follows create a leaf node for each symbol x and assign probability P[x ] while there is more than one node left pick the two nodes x and y having the lowest probabilities replace the two nodes x and y by an inner node having probability P[x ] + P[y ] node x as left child with edge label 0 node y as right child with edge label 1 35

13 Huffman Codes Example: Let us assume that we have observed symbols a,,e with probabilities 1/ /2 1 Symbol: Code: 0 0 1/4 1 1/8 a 0 b 10 c 110 d 1110 e a b c d e 36

14 Huffman Codes Let us compute the expected number of bits per symbol For our example, this corresponds exactly to the entropy of the corresponding random variable, i.e. our code is optimal Huffman codes achieve an optimal average number of bits, whenever all probabilities are of the form 2 -i 37

15 Integer Encoding We now consider different methods to encode non-negative integers, since most symbols in the indexes of an IR system are integers (e.g., document identifiers or term frequencies) The common objective of these methods is to represent smaller integers using fewer bits or bytes One can also use these methods to encode symbols other than integers by assigning integer identifiers to them, so that more frequent symbols are represented by smaller integers 38

16 Variable-Byte Encoding Programming languages represent integers using a fixed number of bytes (e.g., 4 for int and 8 for long in Java), so that small numbers tend to have many leading 0s Variable-Byte Encoding (VBE) (also: 7-Bit Encoding) uses a variable number of bytes to represent a non-negative integer n and is based on two key ideas use only as many bytes as required to represent the integer use the first bit in every byte as a stop bit and the remaining seven bits as a payload to represent information stop bit payload 39

17 Variable-Byte Encoding Example: Using 32 bits the integer 13 is represented as With VBE it is represented as Example: Using 32 bits the integer 131 is represented as With VBE it is represented as

18 Variable-Byte Encoding Example: Using 32 bits the integer 1337 is represented as With VBE it is represented as

19 Variable-Byte Encoding In VBE, the stop bit thus indicates whether the current byte is the last one belonging to the current non-negative integer VBE requires bytes to represent the non-negative integer n 42

20 Gamma Encoding γ Encoding is a bit-level encoding for positive integers The key idea is to split the representation of an integer n into its length which corresponds to the most significant bit an offset which corresponds to the bits after the most significant bit in the binary representation of n 43

21 Gamma Encoding The length in γ Encoding is stored using unary encoding, which represents the non-negative integer n as a sequence of n 1s followed by a 0 Examples: With unary encoding 0 is represented as 0 1 is represented as 10 7 is represented as

22 Gamma Encoding Example: With γ Encoding the integer 2 is represented as 10 0 Example: With γ Encoding the integer 13 is represented as Example: With γ Encoding the integer 131 is represented as Example: With γ Encoding the integer 511 is represented as

23 Gamma Encoding The number of bits used for the offset always corresponds to the number of 1s in the unary encoding of the length This makes the code prefix free and allows us to interpret a sequence of bits unambiguously: when we have read the length, we know how many of the following bits belong to the same integer It can be shown that the average number of bits for γ Encoding is within a factor of 3 of an optimal encoding 46

24 Gap Encoding Whenever we have to encode a sequence of non-decreasing integers, we can apply gap encoding as follows represent the first integer explicitly represent all subsequent integers as the difference to their predecessor Example: With gap encoding, the integer sequence 1, 7, 128, 129, 131, 138 would be represented as 1, 6, 121, 1, 2, 7 47

25 Gap Encoding Note that we reconstruct the original integer sequence on-thefly while reading its gap-encoded representation first integer is used as is subsequent integers are added to the previous integer 1, 6, 121, 1, 2, 7 1, 7, 128, 129, 131, 138 Gap Encoding is only effective when used with another integer encoding method (e.g., VBE or Gamma Encoding) In the above example, when used with VBE, we can represent the numbers 128,, 138 using one instead of two bytes 48

26 Compressing Posting Lists How can we compress posting lists using these encodings? We will use VBE and Gap Encoding for illustration, and assume that we compress a document-ordered posting list Example: Document identifiers d 1 d 2 d 6 d 9 d 11 corresponds to the integer sequence which is encoded as 1, 1, 4, 5,

27 Compressing Posting Lists Example: Document identifiers and term frequencies d 2 5 d 4 7 d 7 1 d 9 3 Observe that only the document identifiers form a non-decreasing integer sequence, so that we can apply Gap Encoding only to those With Gap Encoding, we obtain the integer sequence 2, 5, 2, 7, 3, 1, 2, 3 which is represented as

28 Compressing Posting Lists Example: Document identifiers and positions d 2 [3, 7] d 7 [11] Observe that the document identifiers from different postings form a non-decreasing integer sequence; also the positions within each posting form a non-decreasing integer sequence With Gap-Encoding for both, we obtain the integer sequence 2, 2, 3, 4, 5, 1, 11 which also the lengths of position arrays and is encoded as

29 Compression Effectiveness / Decoding Speed Büttcher et al. [3] report compression effectiveness and decoding speeds on benchmark collections VBE tends to be less effective than Gamma Encoding, but allows for faster decoding speeds 52