Compressing the information contained in the different indexes is crucial for performance when implementing an IR system

4.2 Compression Compressing the information contained in the different indexes is crucial for performance when implementing an IR system on current hardware it is typically much faster to read compressed data from secondary storage and decompress it on-the-fly than to read the same data in uncompressed form with compression more data fits into the available primary storage We will now see different ways how we can assign a code (i.e., a sequence of bits or bytes) to symbols (e.g., terms, document identifiers, positions) that need to be represented a 1010 24

Byte-Level vs. Bit-Level Codes Byte-level codes assign each symbol a variable number of bytes and concatenate codes for symbols in a sequence, so that each byte contains information about one symbol Example: With Variable-Byte Encoding the integer sequence 1, 2, 4, 3, 1 is represented as the byte sequence 10000001 10000010 10000100 10000011 10000001 Symbol: Code: 1 10000001 2 10000010 3 10000011 4 10000100... Each byte contains information about a single symbol 25

Byte-Level vs. Bit-Level Codes Bit-level codes assign each symbol a variable number of bits and concatenate codes for symbols in a sequence, so that a byte can contain bits belonging to different symbols Example: With unary coding the integer sequence 1, 2, 4, 3, 1 is represented as the byte sequence 10110111 10111010 Symbol: Code: 1 10 2 110 3 1110 4 11110... Each byte can contain information about multiple symbols 26

Prefix Codes In order to be able to interpret a given sequence of bits and reconstruct the original information unambiguously, a code needs to be prefix free, i.e., there must not be two codes, so that one is a prefix of the other Example: With non-prefix-free code (don t use!) the integer sequence 1, 3, 1, 3, 2 would be represented as the byte sequence 11111110 Symbol: Code: 1 1 2 10 3 11 4 100... which we would also obtain when encoding 3, 1, 3, 1, 2 27

<latexit sha1_base64="hkazymd72ptjjmeirc54g3/0g+4=">aaacmxicdzdnthsxfixv8fnoskua7roznruq2pljqbqwsjg6yymeeofizgjkcs/bij22be+vyjq34e3ysauvwa7qfh4cn4ffinrkxedzfc3duyjoje2s+2bufmhxzdly28a7lfcfvptr62dgvppij0oudb8gbjkrswez5dhxgokooj4xw+9/8/ofqa2t5akdk8wegztsklfivzu3o8pn/lz4eh4nu1oj3i3cljvhvw5tpd2otr+kxa7yjpte67zzsqlo/l6yvr++hnautntqfosjjvpmy/pd0kpgasknxly0e2uzr7rllgpdscuditahgeafsrrrqdm3qcnq6vfs7lekak3mjmvphi4iy8aiegukyq9mzulkoswf+edze7mxovaqymjjftzbf/emly8efqj9yfhrom6vn/xojube4igwhgppio91w9fz0kh4fzjrrg3pjzut7u60j1igt/aznqen36alh3ampabwdbdwb7+cm+a++b38mt6dc55nnmbgwcmt8ger7a==</latexit> <latexit sha1_base64="hkazymd72ptjjmeirc54g3/0g+4=">aaacmxicdzdnthsxfixv8fnoskua7roznruq2pljqbqwsjg6yymeeofizgjkcs/bij22be+vyjq34e3ysauvwa7qfh4cn4ffinrkxedzfc3duyjoje2s+2bufmhxzdly28a7lfcfvptr62dgvppij0oudb8gbjkrswez5dhxgokooj4xw+9/8/ofqa2t5akdk8wegztsklfivzu3o8pn/lz4eh4nu1oj3i3cljvhvw5tpd2otr+kxa7yjpte67zzsqlo/l6yvr++hnautntqfosjjvpmy/pd0kpgasknxly0e2uzr7rllgpdscuditahgeafsrrrqdm3qcnq6vfs7lekak3mjmvphi4iy8aiegukyq9mzulkoswf+edze7mxovaqymjjftzbf/emly8efqj9yfhrom6vn/xojube4igwhgppio91w9fz0kh4fzjrrg3pjzut7u60j1igt/aznqen36alh3ampabwdbdwb7+cm+a++b38mt6dc55nnmbgwcmt8ger7a==</latexit> <latexit sha1_base64="hkazymd72ptjjmeirc54g3/0g+4=">aaacmxicdzdnthsxfixv8fnoskua7roznruq2pljqbqwsjg6yymeeofizgjkcs/bij22be+vyjq34e3ysauvwa7qfh4cn4ffinrkxedzfc3duyjoje2s+2bufmhxzdly28a7lfcfvptr62dgvppij0oudb8gbjkrswez5dhxgokooj4xw+9/8/ofqa2t5akdk8wegztsklfivzu3o8pn/lz4eh4nu1oj3i3cljvhvw5tpd2otr+kxa7yjpte67zzsqlo/l6yvr++hnautntqfosjjvpmy/pd0kpgasknxly0e2uzr7rllgpdscuditahgeafsrrrqdm3qcnq6vfs7lekak3mjmvphi4iy8aiegukyq9mzulkoswf+edze7mxovaqymjjftzbf/emly8efqj9yfhrom6vn/xojube4igwhgppio91w9fz0kh4fzjrrg3pjzut7u60j1igt/aznqen36alh3ampabwdbdwb7+cm+a++b38mt6dc55nnmbgwcmt8ger7a==</latexit> <latexit sha1_base64="2ngedvnqqr1yt5mwugex2umapvo=">aaacmxicdzdlsgmxgiuz3q23qks3g90o6my0ipefilhxiyhyltjdkim/ntszhcqjlwhewjdx51zfwp241ycw1rqo4igll+fkd/wnk4xqe0uv3sjo2pje5nr0zwz2bn6hurh0ouwhcdsjyek1mqyb0ryahhoglaka84zbzdy5+sov70bpkvjz05oqcnzo6q0l2dgrrqbha611/8df9gnd8nr2/zjmfqv0y6lst4w3yibaacn+x8u0woucxv5etlxv/4v6epvvqwodptwp+fqqgknucmnax9ujarkllageqvmjcw0skw5uwxwojjagetsgwcgo3ndsmmccdgl7aw+ffnotezz7y3jsbofntiiowzn+9bm52usszwvhiccu66mnm9qvf3yyuo7q8kqxptizbicdmmygg3jmlncbw7li6vnpwp8flhpb3fhzdu1wz1duffpbq2gn1deuokth6bq1euh36be9owfvwxvxxr2376cj3mbmgq3je/8ed2orma==</latexit> Entropy Entropy, as proposed by Claude E. Shannon, measures the uncertainty of a discrete random variable X and is defined as H(X) = ÿ xœx P[x ] log 2 P[x ] Quelle: en.wikipedia.org with x as a possible outcome of the random variable and P[ x ] as the probability of observing this outcome 28

<latexit sha1_base64="azdswq439r934hmxcslnfk3bhxs=">aaactnicdzdpaxnbfmffxqox/krvwy+luxjk7laxzueiecmi0eltfrpledt5iunmdqyzs9iw7b/lf+jnvondszfr6bzcy/exhw/f73cevg+pbbcutb9gnvsbt+/c7d7r3x/w8nhj/uati6tqw2jclfdmperlglc0cdwjotggujacjsvl2wv/+cmzy1v16faacomlis85qxekaf9dro3pjboprvo9wpltmvhnnm8nmj9qfjy2cx5w4yy+hjxelq4hszy47q/s4fzon305im9cnkzbgyyfqtv70/6pfkzylalytkc1p1mqxehrom4enb28tqsrlxfbp5hq1gqkvyalyznvuh2wqkm28g0na6zhae1kljfei2vwxvkppcps/rpczxclzytdo6py8fr0ycsgzpnlssy8nrxfjvmdhhcttqq6ok8kmqpsmgdtc/x87sd+pxxtd7pab68g49expuextua5viamdmame7ape2dwcb7an/gefy5+rr+i35frtnt15ymstaf7b531ucm=</latexit> <latexit sha1_base64="azdswq439r934hmxcslnfk3bhxs=">aaactnicdzdpaxnbfmffxqox/krvwy+luxjk7laxzueiecmi0eltfrpledt5iunmdqyzs9iw7b/lf+jnvondszfr6bzcy/exhw/f73cevg+pbbcutb9gnvsbt+/c7d7r3x/w8nhj/uati6tqw2jclfdmperlglc0cdwjotggujacjsvl2wv/+cmzy1v16faacomlis85qxekaf9dro3pjboprvo9wpltmvhnnm8nmj9qfjy2cx5w4yy+hjxelq4hszy47q/s4fzon305im9cnkzbgyyfqtv70/6pfkzylalytkc1p1mqxehrom4enb28tqsrlxfbp5hq1gqkvyalyznvuh2wqkm28g0na6zhae1kljfei2vwxvkppcps/rpczxclzytdo6py8fr0ycsgzpnlssy8nrxfjvmdhhcttqq6ok8kmqpsmgdtc/x87sd+pxxtd7pab68g49expuextua5viamdmame7ape2dwcb7an/gefy5+rr+i35frtnt15ymstaf7b531ucm=</latexit> <latexit sha1_base64="azdswq439r934hmxcslnfk3bhxs=">aaactnicdzdpaxnbfmffxqox/krvwy+luxjk7laxzueiecmi0eltfrpledt5iunmdqyzs9iw7b/lf+jnvondszfr6bzcy/exhw/f73cevg+pbbcutb9gnvsbt+/c7d7r3x/w8nhj/uati6tqw2jclfdmperlglc0cdwjotggujacjsvl2wv/+cmzy1v16faacomlis85qxekaf9dro3pjboprvo9wpltmvhnnm8nmj9qfjy2cx5w4yy+hjxelq4hszy47q/s4fzon305im9cnkzbgyyfqtv70/6pfkzylalytkc1p1mqxehrom4enb28tqsrlxfbp5hq1gqkvyalyznvuh2wqkm28g0na6zhae1kljfei2vwxvkppcps/rpczxclzytdo6py8fr0ycsgzpnlssy8nrxfjvmdhhcttqq6ok8kmqpsmgdtc/x87sd+pxxtd7pab68g49expuextua5viamdmame7ape2dwcb7an/gefy5+rr+i35frtnt15ymstaf7b531ucm=</latexit> <latexit sha1_base64="a+jafw1c2e/mkqxcco1ku7d+6/i=">aaactnicdzdbsxtbfmzn01zjtda2ry+luxjk7qq05iaivqhsudaquet4o3mjq2z2pjozyhj2j/i/6u282xvp3kthgmfufmzhx/d98+b9uelm2di+cwofpn6am68vnbaxpi9/aa58ptay1bs7vhkpt3mwyfmbxcssx1oleuto8sqf/xzyty5qgyaliztwmakyfmzakfgv9zr7qdiufwdptxb7ch1tvefoma40unepxbjxyfq7hh74ongejl8ojpngr9mk2xud7xize76fpb1ppkwmc9br/kv7kpycc0s5ghowxmpmdrrllgpvseudcugihnggsqkfonndlakths/ahgsqadi36wtgdccmgyv8jfh0zayyszmykjv/ltvbduzyouqlbfxebf3unb7zajsj9o9fv8zrt3nb32qidhyviwfus9p2wdv8ps8dho/d8uy78xy41dr9pi2qtlbjglkncflbdskeosbdqskvusa35g/wj7gl7ooh52gtmp75rmamvn8ei1g3zw==</latexit> <latexit sha1_base64="muhbr37amjdp/6qmvfdkiaurt+e=">aaacthicdza/bxnbemxntaij+wccxzotbqh8dwzbuksyrbmkpeskk0i5kzw3hpuvd2+x3t2etbrpxdeho6cbl0bjf0xkxkmkgivrfj+993akeauw3lo0/rm17q3dx3+wsdl++ojxk6edz1vhvtwg0zapocxpizyer2joubn0qg2hlawdllp3v/7jfzkwq+riztuveqcvn3cglkijzodcg59ldj+m9puey9s08v6ctwwynzw+38t55xrh8d3cexkxkht1ummvv7utvt6nvyhrpyvpdl7ayg5gnt/5wlfauuwyqgvpsls7wqnxnalq2nltssob4ztomnwoyrr+skqsm/nlo2cfkmzhf40smr6ltxnzrohxxyylpvizh6x9z7mb7bsev7p2vlhgldanqxt6twylmfb48ngejhqqwk02eejoaykrgwv7azt2qoe2g/j/cnzvzyep33qhb697gg3yhpfwcjj4bwpyhwmyaonv8an+we/oe/q3oo8urqot6obpc1ia1volfj63qq==</latexit> <latexit sha1_base64="muhbr37amjdp/6qmvfdkiaurt+e=">aaacthicdza/bxnbemxntaij+wccxzotbqh8dwzbuksyrbmkpeskk0i5kzw3hpuvd2+x3t2etbrpxdeho6cbl0bjf0xkxkmkgivrfj+993akeauw3lo0/rm17q3dx3+wsdl++ojxk6edz1vhvtwg0zapocxpizyer2joubn0qg2hlawdllp3v/7jfzkwq+riztuveqcvn3cglkijzodcg59ldj+m9puey9s08v6ctwwynzw+38t55xrh8d3cexkxkht1ummvv7utvt6nvyhrpyvpdl7ayg5gnt/5wlfauuwyqgvpsls7wqnxnalq2nltssob4ztomnwoyrr+skqsm/nlo2cfkmzhf40smr6ltxnzrohxxyylpvizh6x9z7mb7bsev7p2vlhgldanqxt6twylmfb48ngejhqqwk02eejoaykrgwv7azt2qoe2g/j/cnzvzyep33qhb697gg3yhpfwcjj4bwpyhwmyaonv8an+we/oe/q3oo8urqot6obpc1ia1volfj63qq==</latexit> <latexit sha1_base64="muhbr37amjdp/6qmvfdkiaurt+e=">aaacthicdza/bxnbemxntaij+wccxzotbqh8dwzbuksyrbmkpeskk0i5kzw3hpuvd2+x3t2etbrpxdeho6cbl0bjf0xkxkmkgivrfj+993akeauw3lo0/rm17q3dx3+wsdl++ojxk6edz1vhvtwg0zapocxpizyer2joubn0qg2hlawdllp3v/7jfzkwq+riztuveqcvn3cglkijzodcg59ldj+m9puey9s08v6ctwwynzw+38t55xrh8d3cexkxkht1ummvv7utvt6nvyhrpyvpdl7ayg5gnt/5wlfauuwyqgvpsls7wqnxnalq2nltssob4ztomnwoyrr+skqsm/nlo2cfkmzhf40smr6ltxnzrohxxyylpvizh6x9z7mb7bsev7p2vlhgldanqxt6twylmfb48ngejhqqwk02eejoaykrgwv7azt2qoe2g/j/cnzvzyep33qhb697gg3yhpfwcjj4bwpyhwmyaonv8an+we/oe/q3oo8urqot6obpc1ia1volfj63qq==</latexit> <latexit sha1_base64="ioaifbsgffkv4qykxteyn2gnpva=">aaacthicdzdnthsxfiu9afklf4eu2yzihlvmjiakcyqknnsbrcucsmwouupcbcv22lu9imiaz+jnuuuim/yfwhzxiwhsidvexhnx6zzjk92tk86mjeofqe3dx7n5hcwl+vlk6tp6y2pz0shsu+xqyaw+zsegzwv2llmcr5vgednhq3x48ujf3ae2tbyxdqqwezaowj9rsf7qnr6ksrtugl3vwp0i9exvhudh2tdaxvk5dhwm30rohf/nlodxmvy30yxb7codepcwniwkfy+nsszz3m08pj1js4gfprymuuliztmh2jlksaqnpuefdagdvifygukduqfkgvappm2pbqg0mrs3mmu6emamrd4jvhwzlezsdi3k5s1y2z/ihctuabgg3hujizrg9xonc9t+sehsfhwvf/xnjujg8t4sqlu0ly9v3dfz2kh4ply2w4nnr3vn4/1juytki2ythzkqz+synjjz0igupjaf5bf5hxwp/gr/g6d/0vow+fojte1t/hkb+rdv</latexit> <latexit sha1_base64="wsjiahmjrcx+sbux77/in/b/kto=">aaackxicdzdnbhmxfixvbepl+gla2leznrjiuwvm0qptfkir2gsdvkqmjzqzrxecm2dfhlu2uzuazrvwjuzywkuwa7zseqjclc5c1cmvpp9jw74n14jbf8c/g9qdh1uptnce1588ffz8t7g3p7bqyrj1mrlkdho0jhhbfcedoke2hdixdjnp393kl1dklfffhvtqyitocj7ldj23xo3xw/btmjaprpfryljholhvyxi3v0aubjvw40yzbru7z/frj9yepbwv1oy+hjxox40/6usxhatcmyhwjpjyu6xe4zgtvnxthswnbi4zgmgsuzpjyhkpsc4s12opbuqywbkady0suvq7lpmgefp/dtnxau4wt/897qznwcklvxbumj+tsiz61pcwzxmyfvho/tiaa2/4mwwk0nf1jjezzvseq7qv566d8h4ytfuj5w/hze7jbu+wa6/gan5aaqfqhr6cqx8yfiiv8bw+bz+d78gp4nft0vrw784lwfpw+y+imqny</latexit> <latexit sha1_base64="wsjiahmjrcx+sbux77/in/b/kto=">aaackxicdzdnbhmxfixvbepl+gla2leznrjiuwvm0qptfkir2gsdvkqmjzqzrxecm2dfhlu2uzuazrvwjuzywkuwa7zseqjclc5c1cmvpp9jw74n14jbf8c/g9qdh1uptnce1588ffz8t7g3p7bqyrj1mrlkdho0jhhbfcedoke2hdixdjnp393kl1dklfffhvtqyitocj7ldj23xo3xw/btmjaprpfryljholhvyxi3v0aubjvw40yzbru7z/frj9yepbwv1oy+hjxox40/6usxhatcmyhwjpjyu6xe4zgtvnxthswnbi4zgmgsuzpjyhkpsc4s12opbuqywbkady0suvq7lpmgefp/dtnxau4wt/897qznwcklvxbumj+tsiz61pcwzxmyfvho/tiaa2/4mwwk0nf1jjezzvseq7qv566d8h4ytfuj5w/hze7jbu+wa6/gan5aaqfqhr6cqx8yfiiv8bw+bz+d78gp4nft0vrw784lwfpw+y+imqny</latexit> <latexit sha1_base64="wsjiahmjrcx+sbux77/in/b/kto=">aaackxicdzdnbhmxfixvbepl+gla2leznrjiuwvm0qptfkir2gsdvkqmjzqzrxecm2dfhlu2uzuazrvwjuzywkuwa7zseqjclc5c1cmvpp9jw74n14jbf8c/g9qdh1uptnce1588ffz8t7g3p7bqyrj1mrlkdho0jhhbfcedoke2hdixdjnp393kl1dklfffhvtqyitocj7ldj23xo3xw/btmjaprpfryljholhvyxi3v0aubjvw40yzbru7z/frj9yepbwv1oy+hjxox40/6usxhatcmyhwjpjyu6xe4zgtvnxthswnbi4zgmgsuzpjyhkpsc4s12opbuqywbkady0suvq7lpmgefp/dtnxau4wt/897qznwcklvxbumj+tsiz61pcwzxmyfvho/tiaa2/4mwwk0nf1jjezzvseq7qv566d8h4ytfuj5w/hze7jbu+wa6/gan5aaqfqhr6cqx8yfiiv8bw+bz+d78gp4nft0vrw784lwfpw+y+imqny</latexit> <latexit sha1_base64="71apnbk7xvasn2psbx0+ltea46o=">aaackxicdzdlsgmxgiuz3q23qks3gwvxiz2pipefilhxi1roa8ezyj/xt4ymk5ckyhnmdxwtd271jdypw7c+hgmtiyoesvhythlyn0rxzmwyvnlj4xotu9mzs4w5+yxfpelysspirqzyp5jl3uzaigcp1i2zhjtki4ie40xsoennf7eodznpzfyuxglakbtmfkyzwswnpn/kr1kkwn5okz0ixjl8y//z14bxk0d5q1gky9uhb+hoof8xkuvwobizqtoqfkzxknyfppzymoayeiobz6atoxzzqtq1qib2oi2xecpqqoosjvkg1b3r2gekak2cdcydctmqxvre8sfs/3/utktswejmr8ft9ugcsvr1labuzqpmgrpxpqwdblvbldjrbs3lddetcthyvaseuc1n2wfecpx8dod/d43tcsxx+w7peg9y1axzi+tkk1tipjkmp6rk6osse/jinsiz9+c9ek/e+/frmw94z5wmypv4aieoqyq=</latexit> <latexit sha1_base64="wsjiahmjrcx+sbux77/in/b/kto=">aaackxicdzdnbhmxfixvbepl+gla2leznrjiuwvm0qptfkir2gsdvkqmjzqzrxecm2dfhlu2uzuazrvwjuzywkuwa7zseqjclc5c1cmvpp9jw74n14jbf8c/g9qdh1uptnce1588ffz8t7g3p7bqyrj1mrlkdho0jhhbfcedoke2hdixdjnp393kl1dklfffhvtqyitocj7ldj23xo3xw/btmjaprpfryljholhvyxi3v0aubjvw40yzbru7z/frj9yepbwv1oy+hjxox40/6usxhatcmyhwjpjyu6xe4zgtvnxthswnbi4zgmgsuzpjyhkpsc4s12opbuqywbkady0suvq7lpmgefp/dtnxau4wt/897qznwcklvxbumj+tsiz61pcwzxmyfvho/tiaa2/4mwwk0nf1jjezzvseq7qv566d8h4ytfuj5w/hze7jbu+wa6/gan5aaqfqhr6cqx8yfiiv8bw+bz+d78gp4nft0vrw784lwfpw+y+imqny</latexit> <latexit sha1_base64="wsjiahmjrcx+sbux77/in/b/kto=">aaackxicdzdnbhmxfixvbepl+gla2leznrjiuwvm0qptfkir2gsdvkqmjzqzrxecm2dfhlu2uzuazrvwjuzywkuwa7zseqjclc5c1cmvpp9jw74n14jbf8c/g9qdh1uptnce1588ffz8t7g3p7bqyrj1mrlkdho0jhhbfcedoke2hdixdjnp393kl1dklfffhvtqyitocj7ldj23xo3xw/btmjaprpfryljholhvyxi3v0aubjvw40yzbru7z/frj9yepbwv1oy+hjxox40/6usxhatcmyhwjpjyu6xe4zgtvnxthswnbi4zgmgsuzpjyhkpsc4s12opbuqywbkady0suvq7lpmgefp/dtnxau4wt/897qznwcklvxbumj+tsiz61pcwzxmyfvho/tiaa2/4mwwk0nf1jjezzvseq7qv566d8h4ytfuj5w/hze7jbu+wa6/gan5aaqfqhr6cqx8yfiiv8bw+bz+d78gp4nft0vrw784lwfpw+y+imqny</latexit> <latexit sha1_base64="wsjiahmjrcx+sbux77/in/b/kto=">aaackxicdzdnbhmxfixvbepl+gla2leznrjiuwvm0qptfkir2gsdvkqmjzqzrxecm2dfhlu2uzuazrvwjuzywkuwa7zseqjclc5c1cmvpp9jw74n14jbf8c/g9qdh1uptnce1588ffz8t7g3p7bqyrj1mrlkdho0jhhbfcedoke2hdixdjnp393kl1dklfffhvtqyitocj7ldj23xo3xw/btmjaprpfryljholhvyxi3v0aubjvw40yzbru7z/frj9yepbwv1oy+hjxox40/6usxhatcmyhwjpjyu6xe4zgtvnxthswnbi4zgmgsuzpjyhkpsc4s12opbuqywbkady0suvq7lpmgefp/dtnxau4wt/897qznwcklvxbumj+tsiz61pcwzxmyfvho/tiaa2/4mwwk0nf1jjezzvseq7qv566d8h4ytfuj5w/hze7jbu+wa6/gan5aaqfqhr6cqx8yfiiv8bw+bz+d78gp4nft0vrw784lwfpw+y+imqny</latexit> <latexit sha1_base64="71apnbk7xvasn2psbx0+ltea46o=">aaackxicdzdlsgmxgiuz3q23qks3gwvxiz2pipefilhxi1roa8ezyj/xt4ymk5ckyhnmdxwtd271jdypw7c+hgmtiyoesvhythlyn0rxzmwyvnlj4xotu9mzs4w5+yxfpelysspirqzyp5jl3uzaigcp1i2zhjtki4ie40xsoennf7eodznpzfyuxglakbtmfkyzwswnpn/kr1kkwn5okz0ixjl8y//z14bxk0d5q1gky9uhb+hoof8xkuvwobizqtoqfkzxknyfppzymoayeiobz6atoxzzqtq1qib2oi2xecpqqoosjvkg1b3r2gekak2cdcydctmqxvre8sfs/3/utktswejmr8ft9ugcsvr1labuzqpmgrpxpqwdblvbldjrbs3lddetcthyvaseuc1n2wfecpx8dod/d43tcsxx+w7peg9y1axzi+tkk1tipjkmp6rk6osse/jinsiz9+c9ek/e+/frmw94z5wmypv4aieoqyq=</latexit> <latexit sha1_base64="nzkdwr8bn+sizapz0aba8gn6pn8=">aaacdhicdzc9sgnbfixv+m/8iz+dttbnbhy3mzikeai2nokcq4fkcbpjnq67szpmtmsw5bxsbpul7mtwd/adfajhqmuqhqb4oofegxsimtbtfp/nmzqemz2bx1gslc2vrk4v1zfotrgqigevivcdighmwiqbysbbjlriejtgrrqffuyxn6g0e+mzgukmormk7ipryqzvpap0dkshjd+tn/rfsu/wwk1/r1x6c1xxn6jc3okjtvrf996loeooqaej0bpb9aujm6imowmoc72hrklotabyjb4kelwydvbwngqujy2mhkmos8lfutajxosrj/6ynjjrvbkjerss6v+fm6tmmlfudg2m1gytzlxa2168oejlh/oor15fwspepl2eglz1okfkanfiugdr+emg9d+c19yq5dn6ub3/1rmswdbsqawq0ia2hmejbebbwd08wknz5zw5z87l1+iu872zctk5rx+h9jxj</latexit> <latexit sha1_base64="nzkdwr8bn+sizapz0aba8gn6pn8=">aaacdhicdzc9sgnbfixv+m/8iz+dttbnbhy3mzikeai2nokcq4fkcbpjnq67szpmtmsw5bxsbpul7mtwd/adfajhqmuqhqb4oofegxsimtbtfp/nmzqemz2bx1gslc2vrk4v1zfotrgqigevivcdighmwiqbysbbjlriejtgrrqffuyxn6g0e+mzgukmormk7ipryqzvpap0dkshjd+tn/rfsu/wwk1/r1x6c1xxn6jc3okjtvrf996loeooqaej0bpb9aujm6imowmoc72hrklotabyjb4kelwydvbwngqujy2mhkmos8lfutajxosrj/6ynjjrvbkjerss6v+fm6tmmlfudg2m1gytzlxa2168oejlh/oor15fwspepl2eglz1okfkanfiugdr+emg9d+c19yq5dn6ub3/1rmswdbsqawq0ia2hmejbebbwd08wknz5zw5z87l1+iu872zctk5rx+h9jxj</latexit> <latexit sha1_base64="nzkdwr8bn+sizapz0aba8gn6pn8=">aaacdhicdzc9sgnbfixv+m/8iz+dttbnbhy3mzikeai2nokcq4fkcbpjnq67szpmtmsw5bxsbpul7mtwd/adfajhqmuqhqb4oofegxsimtbtfp/nmzqemz2bx1gslc2vrk4v1zfotrgqigevivcdighmwiqbysbbjlriejtgrrqffuyxn6g0e+mzgukmormk7ipryqzvpap0dkshjd+tn/rfsu/wwk1/r1x6c1xxn6jc3okjtvrf996loeooqaej0bpb9aujm6imowmoc72hrklotabyjb4kelwydvbwngqujy2mhkmos8lfutajxosrj/6ynjjrvbkjerss6v+fm6tmmlfudg2m1gytzlxa2168oejlh/oor15fwspepl2eglz1okfkanfiugdr+emg9d+c19yq5dn6ub3/1rmswdbsqawq0ia2hmejbebbwd08wknz5zw5z87l1+iu872zctk5rx+h9jxj</latexit> <latexit sha1_base64="mo3mkv5aejat1zdydyqqkmupgpk=">aaacdhicdzdntgixfiu7+if4h7p0m5enbmygjailexi3bewwcyaejqrtl9gwntztx0gmvii7t/os7oxb38f38cgsiasknntx5zx7m9wtiogq7xkfvmzldw19i7uz29re2d3l7x+0fu8kaz/wimtuibvenazfux1bv0jaliyge44vvvpohuhfexytjwichkcxhvkctbf6zwl3xd63padsheqlnlou17ztur0mjcebqydmag3yn/0bthigssyrvqpx8oqouiw1jrfmc/1egcbkjefqw57aamsqjoaz0hkygbummqmvplolfsium6umlfwygda3i2bi+vjjup35xa9rqupjkwiiiclmmlq+mr244xckeds9nlgdyqxzk3ijrohezzhirhyd05yp57cd+39ol52s4atkoxe2lyqljtaxkqisqqigaqiw8hfbhd2ij/rspvgv1qv19joasey7h2hb1vsxdvccdw==</latexit> <latexit sha1_base64="6rcna8joelge7tmx5egk6ly1r+c=">aaaccxicdzc7sgnbfibpxlumt3jpbijpyppdjekleai2nokc0ybzwux4eoed2rlmjmjy8gr2tvosdmlru/goporj1givf6b4+p9zbs4fk86mdyi3rzaxotu9u5wtzc0vlc6vl1fojrxoii0qudttmbjklmwwzzzjw2kkiuz4esehn/nfdwrdzhpmhwojqfop6zfkrlpar7x2zuwgenbl1ade2n8ltvyrfygsb2nvm2sw1km3/n65knqgmlwue2muw0dzkcpamspxvoomdcpce9lhsxioolbhwr+lqkuh+dhhsgsakbufkwsziowzivipkyi9zpuxliklsfn1ue3trrll1cbisl02xsxvgvekn8so3wp+8ddvkme4m4zpicvbxxcqpak7hjvcpt8dvp6h80y9dhy6xw3ufpuervihdahbclvqhcm4grzq4hapd/do3xlp3rp38jva8l53vien7/udhygbrq==</latexit> <latexit sha1_base64="6rcna8joelge7tmx5egk6ly1r+c=">aaaccxicdzc7sgnbfibpxlumt3jpbijpyppdjekleai2nokc0ybzwux4eoed2rlmjmjy8gr2tvosdmlru/goporj1givf6b4+p9zbs4fk86mdyi3rzaxotu9u5wtzc0vlc6vl1fojrxoii0qudttmbjklmwwzzzjw2kkiuz4esehn/nfdwrdzhpmhwojqfop6zfkrlpar7x2zuwgenbl1ade2n8ltvyrfygsb2nvm2sw1km3/n65knqgmlwue2muw0dzkcpamspxvoomdcpce9lhsxioolbhwr+lqkuh+dhhsgsakbufkwsziowzivipkyi9zpuxliklsfn1ue3trrll1cbisl02xsxvgvekn8so3wp+8ddvkme4m4zpicvbxxcqpak7hjvcpt8dvp6h80y9dhy6xw3ufpuervihdahbclvqhcm4grzq4hapd/do3xlp3rp38jva8l53vien7/udhygbrq==</latexit> <latexit sha1_base64="6rcna8joelge7tmx5egk6ly1r+c=">aaaccxicdzc7sgnbfibpxlumt3jpbijpyppdjekleai2nokc0ybzwux4eoed2rlmjmjy8gr2tvosdmlru/goporj1givf6b4+p9zbs4fk86mdyi3rzaxotu9u5wtzc0vlc6vl1fojrxoii0qudttmbjklmwwzzzjw2kkiuz4esehn/nfdwrdzhpmhwojqfop6zfkrlpar7x2zuwgenbl1ade2n8ltvyrfygsb2nvm2sw1km3/n65knqgmlwue2muw0dzkcpamspxvoomdcpce9lhsxioolbhwr+lqkuh+dhhsgsakbufkwsziowzivipkyi9zpuxliklsfn1ue3trrll1cbisl02xsxvgvekn8so3wp+8ddvkme4m4zpicvbxxcqpak7hjvcpt8dvp6h80y9dhy6xw3ufpuervihdahbclvqhcm4grzq4hapd/do3xlp3rp38jva8l53vien7/udhygbrq==</latexit> <latexit sha1_base64="+zciipbhwrhrajqlv/tjurujnxk=">aaaccxicdzdntgixfiu7+if4h7p0q2sdg2ygjclchmqngxnmbcabcenucza006ytxsmej3dnvl/cnxhru/gopoqvcyheky6+nhnvk3siyag2nvfhzjawv1bxsuu5jc2t7z387l5bi7ei0ckccrvewaojmbqmnqwcqqdzieengl185507ujqk+nokekkohzeduikntyjgktgqnbf8fr7olsu1qndckyycx/amkqkzmv38z+9gkdgh2bcgte76njrhipwhhmek1xtrkjim8bc62jnyggrtiqgorixzscuyc9bhoj1nlkwx1zrh0yljsbmdnymhrgzh+s/nzlanuxrlsygy2gykqdvsthr3figyj7qxidux1ra3azdha/cux0qjxby4ydl6fjso/a/tstm3fhvsrj/oisqia3sisshhz6iogqijwogghh7re3p2hpwx59v5+xnnolodftqn5/0los6bwq==</latexit> Entropy Example: Fair coin X = {Heads, Tails} P [ Heads ] = 1 2 P [ Tails ] = 1 2 H(X) =1 Example: Biased coin X = {Heads, Tails} P [ Heads ] = 9 10 H(X) =0.47 P [ Tails ] = 1 10 There is more uncertainty when tossing a fair coin 29

<latexit sha1_base64="hix2siev+fa9ho42inc5khn6zr0=">aaacghicdzc9tisxeivn+bsq/sjfrrorhiq7c1wufeiragiqqckarfbrrjkek+u1zttxrna+cb0tvaqdoqxjhxgitiaiii5cfdpnxtkcvgxc2ch6cuzgx8yn/kxolaznzufmywulp0b2nkmgk5nu5ykaynhodcttrudke4o0o7o0u/+en/0nbbjmt2xfuskwk/m2z2i91sqvnjv220vlr9jsa2tub+hiqgivq1fty3cn2tyt/is4fg1urs/dqeet8mvzurkeonyydi25icnle4facpzruwr2dclkxezqbuykfenedugksro/hhvmuzbj3oc6odchmkyv0h+mqhs1bkzsdi2m5tvntr2toj6rnqwc+wyalmwy31hytun7x8pdfths3va3mtbds9ehqkalqxkssr6erw4qv8ppri32flxvrw9/9astsaprsa4x/im6hmarnicbg1u4g/vgjngihoonj9gr4hnncyyupl8bgycika==</latexit> <latexit sha1_base64="hix2siev+fa9ho42inc5khn6zr0=">aaacghicdzc9tisxeivn+bsq/sjfrrorhiq7c1wufeiragiqqckarfbrrjkek+u1zttxrna+cb0tvaqdoqxjhxgitiaiii5cfdpnxtkcvgxc2ch6cuzgx8yn/kxolaznzufmywulp0b2nkmgk5nu5ykaynhodcttrudke4o0o7o0u/+en/0nbbjmt2xfuskwk/m2z2i91sqvnjv220vlr9jsa2tub+hiqgivq1fty3cn2tyt/is4fg1urs/dqeet8mvzurkeonyydi25icnle4facpzruwr2dclkxezqbuykfenedugksro/hhvmuzbj3oc6odchmkyv0h+mqhs1bkzsdi2m5tvntr2toj6rnqwc+wyalmwy31hytun7x8pdfths3va3mtbds9ehqkalqxkssr6erw4qv8ppri32flxvrw9/9astsaprsa4x/im6hmarnicbg1u4g/vgjngihoonj9gr4hnncyyupl8bgycika==</latexit> <latexit sha1_base64="hix2siev+fa9ho42inc5khn6zr0=">aaacghicdzc9tisxeivn+bsq/sjfrrorhiq7c1wufeiragiqqckarfbrrjkek+u1zttxrna+cb0tvaqdoqxjhxgitiaiii5cfdpnxtkcvgxc2ch6cuzgx8yn/kxolaznzufmywulp0b2nkmgk5nu5ykaynhodcttrudke4o0o7o0u/+en/0nbbjmt2xfuskwk/m2z2i91sqvnjv220vlr9jsa2tub+hiqgivq1fty3cn2tyt/is4fg1urs/dqeet8mvzurkeonyydi25icnle4facpzruwr2dclkxezqbuykfenedugksro/hhvmuzbj3oc6odchmkyv0h+mqhs1bkzsdi2m5tvntr2toj6rnqwc+wyalmwy31hytun7x8pdfths3va3mtbds9ehqkalqxkssr6erw4qv8ppri32flxvrw9/9astsaprsa4x/im6hmarnicbg1u4g/vgjngihoonj9gr4hnncyyupl8bgycika==</latexit> <latexit sha1_base64="jmqif4cu4qfhrd3qrh4wtkj+oyw=">aaacghicdzc9sgnbfivn/y3xlypwnse0vtndqfeliwbjiyiyreiwche8iun2doazirigfre7w30jo7g18x18cmcyiygepvg4596be2kzmg2c4n2bmp6znzvpleqxl5zxvgtr6w0tbopinypeqksyncysxbphjserqrb4ngaz7p985c1bvjqj9nimjuyceinrmgrgwz3czlsqw82kx8v2vwg1+5kng6xtkaxlytfhshtu/athoriprmy67xq+2tecdjimhiagdssmpikskmnoglm+pdaogfahhy0ijehuke2h4gjucdj2majhhdnrdrohba71kmd/ta7mztkmhegbipwvz033mlislqodkxxzck1f164jvx+jco/5z0o/i53hbtj+agbvfa5ucv12movdpt8dfp+hrqucor7yk9wq46jyzitskx0skgnsi6fknnqjjzy8kefy5n17z96l9/o9ouwndzbihly3t57nodq=</latexit> <latexit sha1_base64="/wqqthaziedc8b8crr4+48q9gei=">aaacmnicdzdnthsxfixv0pixwggtu25gznnvzozw/emvinxtdrkvcerirtg1uumtjmew7abe1jwgb9idw/oq7q6x5sewgs5s1cmvpp/ja+kepkthxzr+jhzevv5cwl5zbay9ebu+0dx8d2rv2hdqcluq02noqrqvdz1wjfw0izsspdm2+vqyn12ssujvj26qqza4rmrachtb6jetxbs/qemvct4wyh1w+906zg/zwzixte38qjn68tqjc1hfvx6zlbz3dvbttwfxs8ja6uytzhbmdnxv3ucxio8lvy6xao15lmpxedro8jlqrj62pjgpcejnmgruzao/jcxjmel8hlbcsbbws73nqo/s2qlkl0yj7se8yzqaowt2n8/dyl/wotjjrxup2qx90rwhvwteyiqjkqnp0tfbcdvzperhk0qin8q2a9anum/fdul/w+loowv8/xors/vue6zab9igj5dbhntggxxdfzhcwtxcwk/oz/qnuo3unp4urm8z72fo0f0dfcmsaw==</latexit> <latexit sha1_base64="/wqqthaziedc8b8crr4+48q9gei=">aaacmnicdzdnthsxfixv0pixwggtu25gznnvzozw/emvinxtdrkvcerirtg1uumtjmew7abe1jwgb9idw/oq7q6x5sewgs5s1cmvpp/ja+kepkthxzr+jhzevv5cwl5zbay9ebu+0dx8d2rv2hdqcluq02noqrqvdz1wjfw0izsspdm2+vqyn12ssujvj26qqza4rmrachtb6jetxbs/qemvct4wyh1w+906zg/zwzixte38qjn68tqjc1hfvx6zlbz3dvbttwfxs8ja6uytzhbmdnxv3ucxio8lvy6xao15lmpxedro8jlqrj62pjgpcejnmgruzao/jcxjmel8hlbcsbbws73nqo/s2qlkl0yj7se8yzqaowt2n8/dyl/wotjjrxup2qx90rwhvwteyiqjkqnp0tfbcdvzperhk0qin8q2a9anum/fdul/w+loowv8/xors/vue6zab9igj5dbhntggxxdfzhcwtxcwk/oz/qnuo3unp4urm8z72fo0f0dfcmsaw==</latexit> <latexit sha1_base64="/wqqthaziedc8b8crr4+48q9gei=">aaacmnicdzdnthsxfixv0pixwggtu25gznnvzozw/emvinxtdrkvcerirtg1uumtjmew7abe1jwgb9idw/oq7q6x5sewgs5s1cmvpp/ja+kepkthxzr+jhzevv5cwl5zbay9ebu+0dx8d2rv2hdqcluq02noqrqvdz1wjfw0izsspdm2+vqyn12ssujvj26qqza4rmrachtb6jetxbs/qemvct4wyh1w+906zg/zwzixte38qjn68tqjc1hfvx6zlbz3dvbttwfxs8ja6uytzhbmdnxv3ucxio8lvy6xao15lmpxedro8jlqrj62pjgpcejnmgruzao/jcxjmel8hlbcsbbws73nqo/s2qlkl0yj7se8yzqaowt2n8/dyl/wotjjrxup2qx90rwhvwteyiqjkqnp0tfbcdvzperhk0qin8q2a9anum/fdul/w+loowv8/xors/vue6zab9igj5dbhntggxxdfzhcwtxcwk/oz/qnuo3unp4urm8z72fo0f0dfcmsaw==</latexit> <latexit sha1_base64="xzxvquiveq6ue5h+i3ddbzgtw2m=">aaacmnicdzc9timxfiu9/gbdx2dlbuakocrmaojpqopesw0ssaqimapo2twek+oxztsoktwpwzvq0cjd7hzowx5itqhfqby5+hyor6v7qmq5sxh8j5iznztfwkz8qc4tr6yu1dy3lo0caiytjnop2xqm5rzaluu2x7bscilmeex7j2/51r1qw2vxyuckmwg9gnc5a+utti1klxbdmjwo064g5pls7zvhepqehamgcui6updv12gy8ijsd2r1ulf9ebdvhizfiwney9xjrged2mt6i9laygfzdszcj7gymqntocuxrkydgwpyh3p4dbechtpzpzqcrr5nxx4leggyn957knqgjbkj+suuyg+ntspl3wi1nz633ypm8uinlbbmz2n0ucv49qi+re0pj05huud5w+9kohwsdimbtevt8fhwft0fhytfw+v2i/f8vltv7k2kqpbfzjnskytskyb5tc5iizbytx7je3kohok/wuvw7/3ptdcz+ummflz+b5swrbc=</latexit> <latexit sha1_base64="3wsbnpyne/ptvcdmoe5aurypmuc=">aaacfnicdzc9sgnbfixv+hvjx/zpbbbtweh2n4qaqgjy2agkrgpuembhaxwyszpmtmrlyxvy2epl2imtre/gqzhglaj4moljnhsh7kkvz8ag4zs3nj4xotvdminpzs0vlfawls+m7guklsq51o2ugoqsw5zllmnbasqi5xie9g4+8/mb1ibj7ntmchnbuhm7ypryz3uqk21/34+lanop+aw0xt+jb51knazvg3vhvsp/c1ethkraxiwhjjuv9/hs0r7azfjojlmiqmwtgmjlkmdboe4bvit2sbcvskiiqp0uxzqcrc5hy4czewisynjbsfgqyuwu0j+mipz61eyl7fmsml+f26u9pgcz6lvmqmugwaqt4xokeilq91hwlacd5qx3kwk4sxgbcek1ndwhg7kr56cd/384q9cixyfb1ebov09qgjvyhw2iybeacajh0aikodzdazx6d96t9+y9fi2oed87kzai7/udml6ghw==</latexit> <latexit sha1_base64="3wsbnpyne/ptvcdmoe5aurypmuc=">aaacfnicdzc9sgnbfixv+hvjx/zpbbbtweh2n4qaqgjy2agkrgpuembhaxwyszpmtmrlyxvy2epl2imtre/gqzhglaj4moljnhsh7kkvz8ag4zs3nj4xotvdminpzs0vlfawls+m7guklsq51o2ugoqsw5zllmnbasqi5xie9g4+8/mb1ibj7ntmchnbuhm7ypryz3uqk21/34+lanop+aw0xt+jb51knazvg3vhvsp/c1ethkraxiwhjjuv9/hs0r7azfjojlmiqmwtgmjlkmdboe4bvit2sbcvskiiqp0uxzqcrc5hy4czewisynjbsfgqyuwu0j+mipz61eyl7fmsml+f26u9pgcz6lvmqmugwaqt4xokeilq91hwlacd5qx3kwk4sxgbcek1ndwhg7kr56cd/384q9cixyfb1ebov09qgjvyhw2iybeacajh0aikodzdazx6d96t9+y9fi2oed87kzai7/udml6ghw==</latexit> <latexit sha1_base64="3wsbnpyne/ptvcdmoe5aurypmuc=">aaacfnicdzc9sgnbfixv+hvjx/zpbbbtweh2n4qaqgjy2agkrgpuembhaxwyszpmtmrlyxvy2epl2imtre/gqzhglaj4moljnhsh7kkvz8ag4zs3nj4xotvdminpzs0vlfawls+m7guklsq51o2ugoqsw5zllmnbasqi5xie9g4+8/mb1ibj7ntmchnbuhm7ypryz3uqk21/34+lanop+aw0xt+jb51knazvg3vhvsp/c1ethkraxiwhjjuv9/hs0r7azfjojlmiqmwtgmjlkmdboe4bvit2sbcvskiiqp0uxzqcrc5hy4czewisynjbsfgqyuwu0j+mipz61eyl7fmsml+f26u9pgcz6lvmqmugwaqt4xokeilq91hwlacd5qx3kwk4sxgbcek1ndwhg7kr56cd/384q9cixyfb1ebov09qgjvyhw2iybeacajh0aikodzdazx6d96t9+y9fi2oed87kzai7/udml6ghw==</latexit> <latexit sha1_base64="cehjlm76zmlbnfldnv3sa1y+9es=">aaacfnicdzc9tsmwfiwd8l/+aowsev0yujmakmvaqmjhqqkjqiusry57w6zasww7icjqe7cxwkuwivzw3oghwjqyhiojd5/oudfspalkvjsg+haqu9mzs3pzc9xfpewvvxdt/uqlgslqioij1u6xbkyzablqglslasxtbtdp/+q7v74dpanilk0uiea4l9eujdhyk3e32t6rfxxhjhexjjdaa0tdxk0f9d3dzrb36e1cwa9gqqgxzhp3m+oimucqgckw1jdhie1cyguoytcsrgmnepm+7sendiswookib4kduxk5tphhdjourrevwgjzrxoetpgcm9uymqrrnzjvfz433wzc0ewodgtezims/ja2dfn9fjr91d/l/uraw96kfyyn3psceyv03ekwauv57cd7h65266hli/3acwnc1dzarftog4xoab2ju3sowoighd2ij/tspdgvzqvz9jnaccy7g6gk5/0lt6ugmw==</latexit> <latexit sha1_base64="dz+efi10aajekl5c28i+jcawglo=">aaacdhicdzc7tgjbfibp4g3xhpfohkiddbugqshmsgxotdcrswibmjsecmlozmrmmjanr2bnqy9hz2x9b9/bh3aeldd4z4ov/3/ojocpzmi08bx3j7g0vlk6llxpbwxube+kd/eawowuxqyvovdtgggmwyqnw0yibamq8cdevjc8+mpbd6g0e9g1muj0orlerm8omdbq1hlj48x5ppgvlhrprjcvvsressxzfwp5b6zs9qbmqvfsh90bqucci0ndonwn4enjx0qzrkocprojjzlqirlgh3issfr+pedb0ajjymwxihy1h88uwghjwrwe8ocpyym5xtqdiyagbprx56zf9mmwyzhbinpshrhb0lyxdxigso+5lxo3j61hb9jusayoxu6oejpvczqy9fx0kpkfmrzjy1en2wpp3hmk4rcoiacfoimq1kaodaag4aee4cm5d56df+d1pppwvnf2yuho2ye5e5yb</latexit> <latexit sha1_base64="dz+efi10aajekl5c28i+jcawglo=">aaacdhicdzc7tgjbfibp4g3xhpfohkiddbugqshmsgxotdcrswibmjsecmlozmrmmjanr2bnqy9hz2x9b9/bh3aeldd4z4ov/3/ojocpzmi08bx3j7g0vlk6llxpbwxube+kd/eawowuxqyvovdtgggmwyqnw0yibamq8cdevjc8+mpbd6g0e9g1muj0orlerm8omdbq1hlj48x5ppgvlhrprjcvvsressxzfwp5b6zs9qbmqvfsh90bqucci0ndonwn4enjx0qzrkocprojjzlqirlgh3issfr+pedb0ajjymwxihy1h88uwghjwrwe8ocpyym5xtqdiyagbprx56zf9mmwyzhbinpshrhb0lyxdxigso+5lxo3j61hb9jusayoxu6oejpvczqy9fx0kpkfmrzjy1en2wpp3hmk4rcoiacfoimq1kaodaag4aee4cm5d56df+d1pppwvnf2yuho2ye5e5yb</latexit> <latexit sha1_base64="dz+efi10aajekl5c28i+jcawglo=">aaacdhicdzc7tgjbfibp4g3xhpfohkiddbugqshmsgxotdcrswibmjsecmlozmrmmjanr2bnqy9hz2x9b9/bh3aeldd4z4ov/3/ojocpzmi08bx3j7g0vlk6llxpbwxube+kd/eawowuxqyvovdtgggmwyqnw0yibamq8cdevjc8+mpbd6g0e9g1muj0orlerm8omdbq1hlj48x5ppgvlhrprjcvvsressxzfwp5b6zs9qbmqvfsh90bqucci0ndonwn4enjx0qzrkocprojjzlqirlgh3issfr+pedb0ajjymwxihy1h88uwghjwrwe8ocpyym5xtqdiyagbprx56zf9mmwyzhbinpshrhb0lyxdxigso+5lxo3j61hb9jusayoxu6oejpvczqy9fx0kpkfmrzjy1en2wpp3hmk4rcoiacfoimq1kaodaag4aee4cm5d56df+d1pppwvnf2yuho2ye5e5yb</latexit> <latexit sha1_base64="cdcjcl9kvdar9wc1ojpw5xybdxc=">aaacdhicdzdntgixfiu7+if4h7p0m5enbpgbdclchmqngxnmhccbcenuczzmp03bmzajr+dorb6eo+pwd/adfagr4miknntx5zx7m9wtijaq7bofvmzldw19i7uz29re2d3l7x+0fy8lay/wkmtugbwenajpux1cv0jalaihe4wvv/popuhfexsjpwj8hkcrhvkctbf6zelkxl6wk6vydzavukvkveae1u1lkjfcuqpoodyg/9m/5srmegksyqv6zvdop8fsuxlclnepfqhmxngepewkled6yqg4ay2n6dhghbkop5lflaotzjsasmdjzfjfpc2a87hggfrzur7w/irgityqezpnmxe8zxpxxgfi86hznxugwhjmjuwewmpeyzhirkogzzltz28h9v/qnk0avj4rnkqlorlocb2jiiqjc9ratdrchikio0f0hj6tb+vferxefkyz1mlnekvkvx8bpteclq==</latexit> <latexit sha1_base64="qf5nbglebgwhzxlye2dytf3enqu=">aaacdhicdzc7tsmwfizpyr3cymvjqehsliqttzigvwjhqqkjleptvdnmtfijy8t2evxek7cxwkuwivbegxfgitabhol45eht/59j6fyrtjg2vv/mfcymp6znzuek8wuls8ulldwwfknfmaaieaodey0jszewzctylgojjxk8iokjz/zigpvmij03i4khj4ou9rklxlqd4+rnvvmwxhd3g71sxxfrbw1/+6d8f2qun6vsxidcp73se/ds0chh1ncean2p+dkegvgg0qrvi92hrklotabyib4kelwydvbwngo0hltmcucdzvlfy2fgunyjhv0xotfx42ykrgxiph99bvqnmgophbpmqc1yzlxa2168oejlh/norl5pwspepl2eglzxokfkanfibdhw89nb+x9o1d2a5bodsnpvqyeyhq3yhcruyb+acayneaafaffwai/onfpkpdsvx6mf53tndcbkvh4awzochw==</latexit> <latexit sha1_base64="qf5nbglebgwhzxlye2dytf3enqu=">aaacdhicdzc7tsmwfizpyr3cymvjqehsliqttzigvwjhqqkjleptvdnmtfijy8t2evxek7cxwkuwivbegxfgitabhol45eht/59j6fyrtjg2vv/mfcymp6znzuek8wuls8ulldwwfknfmaaieaodey0jszewzctylgojjxk8iokjz/zigpvmij03i4khj4ou9rklxlqd4+rnvvmwxhd3g71sxxfrbw1/+6d8f2qun6vsxidcp73se/ds0chh1ncean2p+dkegvgg0qrvi92hrklotabyib4kelwydvbwngo0hltmcucdzvlfy2fgunyjhv0xotfx42ykrgxiph99bvqnmgophbpmqc1yzlxa2168oejlh/norl5pwspepl2eglzxokfkanfibdhw89nb+x9o1d2a5bodsnpvqyeyhq3yhcruyb+acayneaafaffwai/onfpkpdsvx6mf53tndcbkvh4awzochw==</latexit> <latexit sha1_base64="qf5nbglebgwhzxlye2dytf3enqu=">aaacdhicdzc7tsmwfizpyr3cymvjqehsliqttzigvwjhqqkjleptvdnmtfijy8t2evxek7cxwkuwivbegxfgitabhol45eht/59j6fyrtjg2vv/mfcymp6znzuek8wuls8ulldwwfknfmaaieaodey0jszewzctylgojjxk8iokjz/zigpvmij03i4khj4ou9rklxlqd4+rnvvmwxhd3g71sxxfrbw1/+6d8f2qun6vsxidcp73se/ds0chh1ncean2p+dkegvgg0qrvi92hrklotabyib4kelwydvbwngo0hltmcucdzvlfy2fgunyjhv0xotfx42ykrgxiph99bvqnmgophbpmqc1yzlxa2168oejlh/norl5pwspepl2eglzxokfkanfibdhw89nb+x9o1d2a5bodsnpvqyeyhq3yhcruyb+acayneaafaffwai/onfpkpdsvx6mf53tndcbkvh4awzochw==</latexit> <latexit sha1_base64="k9jw1zuxwmvowxl2vt2hbteqstw=">aaacdhicdzc7tgjbfizn8yz4qy1tntjgsxe8qwfcyknjgolcetiq2fgak93zmcwmbrlhfexs9sxsjk3v4dv4ei6ibrl/tphl/8+z5pyhiknsnvdhzzawv1bxsuu5jc2t7z387l5t8aek0ca85ridyguxtachqy6hlsrgfsbqcqpl77x1d1jrntzosyca4ufc+5rgbaxortg6si/sknna7uulnloqll3jir0ivunnvuaz1xv5z+4tj0mgisyxvqrje0ihkzaakhgmue5qgcakwgpoye9gatjib8azadmejw0mmiek0ulfc2gkmvjjfi6ydou7etpkpni4vh8+1/1yknjeddukxgrttn2gmr24uqjspopejd2emia5sbkx1jbygsask8fgjgfq+e3a/h+ajcc3fh1sqj7nisqia3siishh56ikaqiogoggjh7re3q2hqwx69v6+xnnwlodftqn6/0lsi+cmw==</latexit> <latexit sha1_base64="3wsbnpyne/ptvcdmoe5aurypmuc=">aaacfnicdzc9sgnbfixv+hvjx/zpbbbtweh2n4qaqgjy2agkrgpuembhaxwyszpmtmrlyxvy2epl2imtre/gqzhglaj4moljnhsh7kkvz8ag4zs3nj4xotvdminpzs0vlfawls+m7guklsq51o2ugoqsw5zllmnbasqi5xie9g4+8/mb1ibj7ntmchnbuhm7ypryz3uqk21/34+lanop+aw0xt+jb51knazvg3vhvsp/c1ethkraxiwhjjuv9/hs0r7azfjojlmiqmwtgmjlkmdboe4bvit2sbcvskiiqp0uxzqcrc5hy4czewisynjbsfgqyuwu0j+mipz61eyl7fmsml+f26u9pgcz6lvmqmugwaqt4xokeilq91hwlacd5qx3kwk4sxgbcek1ndwhg7kr56cd/384q9cixyfb1ebov09qgjvyhw2iybeacajh0aikodzdazx6d96t9+y9fi2oed87kzai7/udml6ghw==</latexit> <latexit sha1_base64="3wsbnpyne/ptvcdmoe5aurypmuc=">aaacfnicdzc9sgnbfixv+hvjx/zpbbbtweh2n4qaqgjy2agkrgpuembhaxwyszpmtmrlyxvy2epl2imtre/gqzhglaj4moljnhsh7kkvz8ag4zs3nj4xotvdminpzs0vlfawls+m7guklsq51o2ugoqsw5zllmnbasqi5xie9g4+8/mb1ibj7ntmchnbuhm7ypryz3uqk21/34+lanop+aw0xt+jb51knazvg3vhvsp/c1ethkraxiwhjjuv9/hs0r7azfjojlmiqmwtgmjlkmdboe4bvit2sbcvskiiqp0uxzqcrc5hy4czewisynjbsfgqyuwu0j+mipz61eyl7fmsml+f26u9pgcz6lvmqmugwaqt4xokeilq91hwlacd5qx3kwk4sxgbcek1ndwhg7kr56cd/384q9cixyfb1ebov09qgjvyhw2iybeacajh0aikodzdazx6d96t9+y9fi2oed87kzai7/udml6ghw==</latexit> <latexit sha1_base64="3wsbnpyne/ptvcdmoe5aurypmuc=">aaacfnicdzc9sgnbfixv+hvjx/zpbbbtweh2n4qaqgjy2agkrgpuembhaxwyszpmtmrlyxvy2epl2imtre/gqzhglaj4moljnhsh7kkvz8ag4zs3nj4xotvdminpzs0vlfawls+m7guklsq51o2ugoqsw5zllmnbasqi5xie9g4+8/mb1ibj7ntmchnbuhm7ypryz3uqk21/34+lanop+aw0xt+jb51knazvg3vhvsp/c1ethkraxiwhjjuv9/hs0r7azfjojlmiqmwtgmjlkmdboe4bvit2sbcvskiiqp0uxzqcrc5hy4czewisynjbsfgqyuwu0j+mipz61eyl7fmsml+f26u9pgcz6lvmqmugwaqt4xokeilq91hwlacd5qx3kwk4sxgbcek1ndwhg7kr56cd/384q9cixyfb1ebov09qgjvyhw2iybeacajh0aikodzdazx6d96t9+y9fi2oed87kzai7/udml6ghw==</latexit> <latexit sha1_base64="cehjlm76zmlbnfldnv3sa1y+9es=">aaacfnicdzc9tsmwfiwd8l/+aowsev0yujmakmvaqmjhqqkjqiusry57w6zasww7icjqe7cxwkuwivzw3oghwjqyhiojd5/oudfspalkvjsg+haqu9mzs3pzc9xfpewvvxdt/uqlgslqioij1u6xbkyzablqglslasxtbtdp/+q7v74dpanilk0uiea4l9eujdhyk3e32t6rfxxhjhexjjdaa0tdxk0f9d3dzrb36e1cwa9gqqgxzhp3m+oimucqgckw1jdhie1cyguoytcsrgmnepm+7sendiswookib4kduxk5tphhdjourrevwgjzrxoetpgcm9uymqrrnzjvfz433wzc0ewodgtezims/ja2dfn9fjr91d/l/uraw96kfyyn3psceyv03ekwauv57cd7h65266hli/3acwnc1dzarftog4xoab2ju3sowoighd2ij/tspdgvzqvz9jnaccy7g6gk5/0lt6ugmw==</latexit> <latexit sha1_base64="vzwrk/uckte7knqmn5jkbjhjnj4=">aaacqhicdzblbxmxfixvtdxkeixcjs2ibfhumznyaitvissgdvkrsfuphkxx7k2wyo8t20gjrpk1/sfdsyuv/4afemllcjefrag48uk759iw7ufwsr/k8mu2tn7t+o2bg7c6t+/cvxe/+2dz0juzezqurhl3zngtkg0ngwykjq0j1fzrez++vsippplz0jtvw8jsrxhsyleugji16u4z6+k8zfdznnyoytxgqmxztsf2cqa5mcexce3fom+zbhiwqy2mtk3wwy9yo+r2yv727k75bde/clw/xko3earlhyy639mpetnntrakvt+pshvqic5ioajtsjkni2kkezrb0qilv8cjgu3blvbjha1q8nvctrasrttelzs/ymomh1znbsw0ipf/fb7go3wujz0faktklhilou9dflnolh1zvf0ui5umtjmvfaaafxqfm76fso2kev52kp8fdrf7vej3z3udl5c9wqy8hifwfcp4bqn4awcwbafn8ak+w5fsppuw/ch+xl5dy/68eqgryn79bthtsbo=</latexit> <latexit sha1_base64="vzwrk/uckte7knqmn5jkbjhjnj4=">aaacqhicdzblbxmxfixvtdxkeixcjs2ibfhumznyaitvissgdvkrsfuphkxx7k2wyo8t20gjrpk1/sfdsyuv/4afemllcjefrag48uk759iw7ufwsr/k8mu2tn7t+o2bg7c6t+/cvxe/+2dz0juzezqurhl3zngtkg0ngwykjq0j1fzrez++vsippplz0jtvw8jsrxhsyleugji16u4z6+k8zfdznnyoytxgqmxztsf2cqa5mcexce3fom+zbhiwqy2mtk3wwy9yo+r2yv727k75bde/clw/xko3earlhyy639mpetnntrakvt+pshvqic5ioajtsjkni2kkezrb0qilv8cjgu3blvbjha1q8nvctrasrttelzs/ymomh1znbsw0ipf/fb7go3wujz0faktklhilou9dflnolh1zvf0ui5umtjmvfaaafxqfm76fso2kev52kp8fdrf7vej3z3udl5c9wqy8hifwfcp4bqn4awcwbafn8ak+w5fsppuw/ch+xl5dy/68eqgryn79bthtsbo=</latexit> <latexit sha1_base64="vzwrk/uckte7knqmn5jkbjhjnj4=">aaacqhicdzblbxmxfixvtdxkeixcjs2ibfhumznyaitvissgdvkrsfuphkxx7k2wyo8t20gjrpk1/sfdsyuv/4afemllcjefrag48uk759iw7ufwsr/k8mu2tn7t+o2bg7c6t+/cvxe/+2dz0juzezqurhl3zngtkg0ngwykjq0j1fzrez++vsippplz0jtvw8jsrxhsyleugji16u4z6+k8zfdznnyoytxgqmxztsf2cqa5mcexce3fom+zbhiwqy2mtk3wwy9yo+r2yv727k75bde/clw/xko3earlhyy639mpetnntrakvt+pshvqic5ioajtsjkni2kkezrb0qilv8cjgu3blvbjha1q8nvctrasrttelzs/ymomh1znbsw0ipf/fb7go3wujz0faktklhilou9dflnolh1zvf0ui5umtjmvfaaafxqfm76fso2kev52kp8fdrf7vej3z3udl5c9wqy8hifwfcp4bqn4awcwbafn8ak+w5fsppuw/ch+xl5dy/68eqgryn79bthtsbo=</latexit> <latexit sha1_base64="csxc8jrmbusfn59ucvq4jje/1hq=">aaacqhicdza/t1sxfmx9ofbikqq6dnlqlg4o7z3+fzcqkfi6vkjsa0j4kbo2n6kv+9myhztiep+gb8lwtz34bh0qoa6datj0cigjd797jm3phmakcd7p75k5+rcliy+xlhuvvl6vrjxxn86chlqoha6lthcmhepryccll/hcwatfjj6zwcldfn6n1gldfffjg6wcfiv6goopvrd5ri0nozo9smnpag9fhyq8tukhpuypynouetrwd+mopajkasg2qbzs3m3u0rrbboxtryp9fpsgfqpfo5+oray67tz/0svnhworzyu4d1nkxpcbrbdcyt2gq4cg+ad6eam5ayo2dh3ucr0dz8yrk1doyjbpysymojwbk/bevoc/zppm64eh5h597nv7zrcvgxqsemwmglkoi11ma4y2hpf9gmdde424k8skebxlcrjvrh2xbsr6/neqpg9nw+0i8ued1vhetkgl8pa8i+9jqt6qy/krnjio4esgfcpfyy/knvmz3ce//12ds6zv3pazjx/+al3jswy=</latexit> Entropy Example: Fair dice X = {1,...6} P[x ]= 1 6 x œ X H(x) =2.58 Example: Biased dice P[6]= 5 10 X = {1,...6} P[x ]= 1 10 H(x) =2.16 x œ {1,...,5} There is more uncertainty when tossing a fair dice 30

<latexit sha1_base64="98welh48otnrglx4uwp6aaz+rgg=">aaacuhicdzdnbhmxfebvhj+wleka7timykarzeydaltaissgtauikbrszxrduzeplfhy2b5ezm1t8sbsuoutd8coookilfrcwflrd2xl/pguhxvpehu1hjx89hhn90l77+n+s+edfy9hvtwg05crupllhpzkudhqcvfsptaekpv0wwyffv7ikxkrvpxzztuveqevmaiolktjzlmujccmfh/ne4pcz43p0ibov9r4hs8c++f6245v3xt3umnv6pqk7z/g9yhrpcvpdg5goefjzu/8wvfauuv4idzezal2hufjbc+paee1jy18hlo6wlsjjlp4ksljzsw3dcakjdncl1vzkb6ltxpj7ous3c1myjsaowr263e3osm8qhttqolbldenqxu6twamtfgiozsnyx2c8ceblojowykrg2v7azt2qgfdqfx/gb31sscf3nyh71y9ws68hjdwcbkcwwa+wjkmgcn3uiwf8cv6ef2j/rai1dh1dq9gy1rtowystja=</latexit> <latexit sha1_base64="98welh48otnrglx4uwp6aaz+rgg=">aaacuhicdzdnbhmxfebvhj+wleka7timykarzeydaltaissgtauikbrszxrduzeplfhy2b5ezm1t8sbsuoutd8coookilfrcwflrd2xl/pguhxvpehu1hjx89hhn90l77+n+s+edfy9hvtwg05crupllhpzkudhqcvfsptaekpv0wwyffv7ikxkrvpxzztuveqevmaiolktjzlmujccmfh/ne4pcz43p0ibov9r4hs8c++f6245v3xt3umnv6pqk7z/g9yhrpcvpdg5goefjzu/8wvfauuv4idzezal2hufjbc+paee1jy18hlo6wlsjjlp4ksljzsw3dcakjdncl1vzkb6ltxpj7ous3c1myjsaowr263e3osm8qhttqolbldenqxu6twamtfgiozsnyx2c8ceblojowykrg2v7azt2qgfdqfx/gb31sscf3nyh71y9ws68hjdwcbkcwwa+wjkmgcn3uiwf8cv6ef2j/rai1dh1dq9gy1rtowystja=</latexit> <latexit sha1_base64="98welh48otnrglx4uwp6aaz+rgg=">aaacuhicdzdnbhmxfebvhj+wleka7timykarzeydaltaissgtauikbrszxrduzeplfhy2b5ezm1t8sbsuoutd8coookilfrcwflrd2xl/pguhxvpehu1hjx89hhn90l77+n+s+edfy9hvtwg05crupllhpzkudhqcvfsptaekpv0wwyffv7ikxkrvpxzztuveqevmaiolktjzlmujccmfh/ne4pcz43p0ibov9r4hs8c++f6245v3xt3umnv6pqk7z/g9yhrpcvpdg5goefjzu/8wvfauuv4idzezal2hufjbc+paee1jy18hlo6wlsjjlp4ksljzsw3dcakjdncl1vzkb6ltxpj7ous3c1myjsaowr263e3osm8qhttqolbldenqxu6twamtfgiozsnyx2c8ceblojowykrg2v7azt2qgfdqfx/gb31sscf3nyh71y9ws68hjdwcbkcwwa+wjkmgcn3uiwf8cv6ef2j/rai1dh1dq9gy1rtowystja=</latexit> <latexit sha1_base64="ka+cops08cdynwavbxug1/jdoyc=">aaacuhicdzdnthsxfebvhp6etjsultmmmk1xmrlafbavirhpbgmkbpcyuxrtbokv8djynqqrnu/fm7bjs7d9go6ksufvgnplyuffss35y7ou1qxpbdraefhy1ev2aufn27v36933gydw1ybtkktsmtoglkpr0dajv9kznosslxtkpgcp/vq7gstu9c3nnbusj5uyc44urkpuya6nxyb+eudjg9xnjc/sjs6varyihxz757axhv++n+r20v7w/l66vr8/h6yfzqchj3m06v7klxsvjvwol2jtezzqv3g0tvcsmk5ew9lipzihc0w1ajkfn5cs5mxsuqesujit/lyvbelrwjut7fko0v0uhkypqunmlx53473ci0rxjioe3bx9mrsh22tkyiqlksnzmtihch+ysymofiqsuvg2h7dphhqeooj/dydb/szw8u5vsptyvbs24sn8ggw+wwc+whemgcm13mid/ixuot/rn1b09+jtdh9gyvqde4tftdw=</latexit> <latexit sha1_base64="itbmxkdqjxngqrpgp5hgxzn1xu8=">aaacnhicdzdnshxbfezva4xmnmnezjdn42xctxcb8wcrghcttudbucfuhts1dybfdhwvvdxiuprz+cbu3oozblil2qwfwbjncbpxuovdd6ok7perghsbx9+dufkxcy8xl161lldev3nbfrd6zgslgfwzlkq+ydfqwuvqw24lolgauoqfheetvqd/fe7acfke2qmitoc45cpo0ppo0e5spd2wdj+h6ugjc5u1s+i6tm8qhiypjv65phgddifubuzuxj92w6eqdonmor0p0mz+op07hupwcsotk9cy0yrwnnoolwcf1a20mqsqtxbmpxgrvkqznyypyorprpzyoictuwbzgelqgdmv+znqop02g+zstizm5r/p7wgnc7xulawsedegi/rg9xdnctl+8ojrnbooh/idtfsgpytiinpsdd3wlv/p3w7c5+foo5t4ptjs9lyee4il+ahrsa4jbempvsa+9ihbjvzdddwgv8gp4gfw6/hqxpdnzxuymeduhshkruy=</latexit> <latexit sha1_base64="itbmxkdqjxngqrpgp5hgxzn1xu8=">aaacnhicdzdnshxbfezva4xmnmnezjdn42xctxcb8wcrghcttudbucfuhts1dybfdhwvvdxiuprz+cbu3oozblil2qwfwbjncbpxuovdd6ok7perghsbx9+dufkxcy8xl161lldev3nbfrd6zgslgfwzlkq+ydfqwuvqw24lolgauoqfheetvqd/fe7acfke2qmitoc45cpo0ppo0e5spd2wdj+h6ugjc5u1s+i6tm8qhiypjv65phgddifubuzuxj92w6eqdonmor0p0mz+op07hupwcsotk9cy0yrwnnoolwcf1a20mqsqtxbmpxgrvkqznyypyorprpzyoictuwbzgelqgdmv+znqop02g+zstizm5r/p7wgnc7xulawsedegi/rg9xdnctl+8ojrnbooh/idtfsgpytiinpsdd3wlv/p3w7c5+foo5t4ptjs9lyee4il+ahrsa4jbempvsa+9ihbjvzdddwgv8gp4gfw6/hqxpdnzxuymeduhshkruy=</latexit> <latexit sha1_base64="itbmxkdqjxngqrpgp5hgxzn1xu8=">aaacnhicdzdnshxbfezva4xmnmnezjdn42xctxcb8wcrghcttudbucfuhts1dybfdhwvvdxiuprz+cbu3oozblil2qwfwbjncbpxuovdd6ok7perghsbx9+dufkxcy8xl161lldev3nbfrd6zgslgfwzlkq+ydfqwuvqw24lolgauoqfheetvqd/fe7acfke2qmitoc45cpo0ppo0e5spd2wdj+h6ugjc5u1s+i6tm8qhiypjv65phgddifubuzuxj92w6eqdonmor0p0mz+op07hupwcsotk9cy0yrwnnoolwcf1a20mqsqtxbmpxgrvkqznyypyorprpzyoictuwbzgelqgdmv+znqop02g+zstizm5r/p7wgnc7xulawsedegi/rg9xdnctl+8ojrnbooh/idtfsgpytiinpsdd3wlv/p3w7c5+foo5t4ptjs9lyee4il+ahrsa4jbempvsa+9ihbjvzdddwgv8gp4gfw6/hqxpdnzxuymeduhshkruy=</latexit> <latexit sha1_base64="xf10p1lix2hvjsj2vcgk4ioypvq=">aaacnhicdzdlahsxfiy16c110naallmz6k1wnhk3ppaiymgmm0ac9qu8gzkjh7vco5eqauqnmofim2txbfomhe5kdqhpumvxci7jqyup/5me589kzrsjot/e1rpnl16+qr2ub++8efvof7/b16jufhtu5einm9cyswj7hpkch1ih8czhqty/vvwd76g0e8uxs5cycpgvbmoogben/tiryk6q4hoqtbvqe1jzokqc5fsjk+dw4yoll27sn6jmq9oopnactyib0xiazdvny/8mmqhaciwmzuhrurxjk1pqhtecq3psapra5zddeuqsjkruzlbwngqxrh0wwfgndrn5mrtatv7wbcpkyl6uh5kqcwozfvs5mbztywpzgiyoc0u0yu+7/sj5hsodfp4uwrf0gdtjhzky/bfyoeropsoq7uq57yb4gvqtzuz4/ldrpvovvsp75am5idh5rlrkhjyrhqhkgvwkv+sxd+n98f5613dxt7zvmz2ynt6//0cmrpi=</latexit> <latexit sha1_base64="0ft7jxjmfuzoo3h4sesqtqbipem=">aaacfnicdzc9ahtbfixvok4ik39ylc7nejupgnzlh0qqdii0bgikrd+gxctd0zuyagznmbkfl4vew51b+yxcbbdu/q5+ieykufbedln8nhpvwd2zfty6ol4l9h7tp37ythjqffb8xctxtcpxa6uwhlgfkahmkenlgufud9wjgmldkdnbw2zx5u8+/engcpv/d4wmvoi85zpo0hlruqupwpmwkffdmiipcjakzdwpnelmuacdh3fcxwg147ua3tewvm9su0+mii0l5y4jthbcirvlszsom0grark0pjetce5jjdvqmmk5jyxjmwi79pijjjuw69u2whkltyxmdkyj7se2msm1cjjzfz53s3za8lwvhexmz2sso771duwljix/ppparbptdx+tjqq6ooskmqns0+oq6ut56cd8pwyomi3p3z42up82puef3si7ea8t+axdoiue9ifbardwcvfbexad/apunqn7wd+domwpup0nzp6g5g==</latexit> <latexit sha1_base64="0ft7jxjmfuzoo3h4sesqtqbipem=">aaacfnicdzc9ahtbfixvok4ik39ylc7nejupgnzlh0qqdii0bgikrd+gxctd0zuyagznmbkfl4vew51b+yxcbbdu/q5+ieykufbedln8nhpvwd2zfty6ol4l9h7tp37ythjqffb8xctxtcpxa6uwhlgfkahmkenlgufud9wjgmldkdnbw2zx5u8+/engcpv/d4wmvoi85zpo0hlruqupwpmwkffdmiipcjakzdwpnelmuacdh3fcxwg147ua3tewvm9su0+mii0l5y4jthbcirvlszsom0grark0pjetce5jjdvqmmk5jyxjmwi79pijjjuw69u2whkltyxmdkyj7se2msm1cjjzfz53s3za8lwvhexmz2sso771duwljix/ppparbptdx+tjqq6ooskmqns0+oq6ut56cd8pwyomi3p3z42up82puef3si7ea8t+axdoiue9ifbardwcvfbexad/apunqn7wd+domwpup0nzp6g5g==</latexit> <latexit sha1_base64="0ft7jxjmfuzoo3h4sesqtqbipem=">aaacfnicdzc9ahtbfixvok4ik39ylc7nejupgnzlh0qqdii0bgikrd+gxctd0zuyagznmbkfl4vew51b+yxcbbdu/q5+ieykufbedln8nhpvwd2zfty6ol4l9h7tp37ythjqffb8xctxtcpxa6uwhlgfkahmkenlgufud9wjgmldkdnbw2zx5u8+/engcpv/d4wmvoi85zpo0hlruqupwpmwkffdmiipcjakzdwpnelmuacdh3fcxwg147ua3tewvm9su0+mii0l5y4jthbcirvlszsom0grark0pjetce5jjdvqmmk5jyxjmwi79pijjjuw69u2whkltyxmdkyj7se2msm1cjjzfz53s3za8lwvhexmz2sso771duwljix/ppparbptdx+tjqq6ooskmqns0+oq6ut56cd8pwyomi3p3z42up82puef3si7ea8t+axdoiue9ifbardwcvfbexad/apunqn7wd+domwpup0nzp6g5g==</latexit> <latexit sha1_base64="tklpoqe+hrlpu0nbsl3g7issmuw=">aaacfnicdzc9sgnbfivn41+mf1flm8u0fpldrpgneai2nokciyhseu6o1zhkzmeymyjlkvews9wxsbnbw9/bh3cmsyjiyyqpc+4ducdrnbkbhu9eywp6znauof9awfxaximvrrwmhgiktsq51o0edhkwytmyy7gtnijiof4m/eov/piwtweyvbczwlhal2xxjij1vre83vap/cihbt/iv9iah6nht1wjq/xdg3dn0p8ltwo4uowmddytf0rxkg4eppzymkztc5wnc9cwuy7dujqwqid2oycdcbuo1hheqynq6mwydpicqbpno9smwhyemzli/pgc7m2kmujzt5cyx5/b64m4z6kaweypy0ayb03jggr6cwr3whcabv3ldhetcthyvaseuc1n1egw5or56cd/h1r1as3x+w6lstcuqkg2ycbzijwytxrkhjyrjqekiw/kktx5996z9+k9fo8wvphoopmq9/yju/qgkg==</latexit> <latexit sha1_base64="n6pej54tfpssdi4kdu8b81rwooo=">aaacdhicdzc7sgnbfibpeo3xfi+dttcnnrubeeskiwctrojgmkbcwux4jmpu7awzezesequ7w30jo7h1hxwhh8ix0skkp1n8/p85a+epzmk08f13z2p6znzupreqx1xaxlktrk1fanfxfamqeqhaedgysbqdw0ycbamq8cjbvhsffowtw1saiftcdcsgnprsds0omdbqnhbudovhxyrrh3qljd+t1kr+xq34f8qup1kpvgkjnbufj8srqfscu0mtonwn7estzkqzrhmc5i/7giwhmelhh/issfrh1kpb0ajbzgwxjrx1mi0umggzwrue8oipyym5mtqjiwjdiv3rc3nddtowyr7blnpshjkxanulf0eo7gpe6cdrsmvym7sxein3hiduce1ahoztpt8dfp+hi4pbtny2x6ofjnuchgzbnuxagy6gdg1oqgaubdzaizw5986z8+k8jkenno+ddziq8/yjtj6cfw==</latexit> <latexit sha1_base64="n6pej54tfpssdi4kdu8b81rwooo=">aaacdhicdzc7sgnbfibpeo3xfi+dttcnnrubeeskiwctrojgmkbcwux4jmpu7awzezesequ7w30jo7h1hxwhh8ix0skkp1n8/p85a+epzmk08f13z2p6znzupreqx1xaxlktrk1fanfxfamqeqhaedgysbqdw0ycbamq8cjbvhsffowtw1saiftcdcsgnprsds0omdbqnhbudovhxyrrh3qljd+t1kr+xq34f8qup1kpvgkjnbufj8srqfscu0mtonwn7estzkqzrhmc5i/7giwhmelhh/issfrh1kpb0ajbzgwxjrx1mi0umggzwrue8oipyym5mtqjiwjdiv3rc3nddtowyr7blnpshjkxanulf0eo7gpe6cdrsmvym7sxein3hiduce1ahoztpt8dfp+hi4pbtny2x6ofjnuchgzbnuxagy6gdg1oqgaubdzaizw5986z8+k8jkenno+ddziq8/yjtj6cfw==</latexit> <latexit sha1_base64="n6pej54tfpssdi4kdu8b81rwooo=">aaacdhicdzc7sgnbfibpeo3xfi+dttcnnrubeeskiwctrojgmkbcwux4jmpu7awzezesequ7w30jo7h1hxwhh8ix0skkp1n8/p85a+epzmk08f13z2p6znzupreqx1xaxlktrk1fanfxfamqeqhaedgysbqdw0ycbamq8cjbvhsffowtw1saiftcdcsgnprsds0omdbqnhbudovhxyrrh3qljd+t1kr+xq34f8qup1kpvgkjnbufj8srqfscu0mtonwn7estzkqzrhmc5i/7giwhmelhh/issfrh1kpb0ajbzgwxjrx1mi0umggzwrue8oipyym5mtqjiwjdiv3rc3nddtowyr7blnpshjkxanulf0eo7gpe6cdrsmvym7sxein3hiduce1ahoztpt8dfp+hi4pbtny2x6ofjnuchgzbnuxagy6gdg1oqgaubdzaizw5986z8+k8jkenno+ddziq8/yjtj6cfw==</latexit> <latexit sha1_base64="hy4198yi38ygz0udqfqax426auq=">aaacdhicdzdntgixfiu7+if4h7p0q2sdm5kb/2bhqukgjqkmdpdahhtqbzuztpu2gmiev3dnvl/cnxhro/gopoqvctest7r4cs69te4jresvdt0pk7o0vlk6ll3pbwxube/kd/daio8kay/wimtogbvenazpux1br0jaliighysx33n7hqsipl7rewe+w8oydijb2ljdrml8vlgovgz3tj8vunalvnwpa4vfknvuteu0v7of/+zdcjjiegssyaw6zvdop8fsuxlbnncbkrcyhhgixewkled6yra4ay0n6dhgjbkop5ldlaotzjsasgdbzfjfpc2a81djqp35xa+qfkjjmdiqe5pnmhe8zxpxwgckeds5mjh9yqxzk3iirghsmewkv7bbac7u89tb4x9oveyy4eutyv1sxlqwhabdvejldi7qqigayemecfsintcz9wc9wk/w289oxprv7kourpcvo5qckw==</latexit> Entropy Example: Text documents with terms a,, e P[a ]= 1 10 X = {a,... e} P[d ]= 4 10 P[b ]= 3 10 H(x) =2.05 P[e ]= 1 10 P[c ]= 1 10 What is the connection between entropy and compression? 31

Entropy and Compression Entropy provides us with a lower bound on the average number of bits (i.e., average code length) that any prefix-free encoding of the random variable s outcomes will require Any prefix-free encoding will thus require at least H(X ) bits on average to encode outcomes of X 32

Huffman Codes Huffman coding assigns shorter codes to more frequent symbols and longer codes to less frequent symbols It can be shown that Huffman codes are optimal, under certain assumptions, i.e., the average number of bits required corresponds to the entropy of the random variable 33

Huffman Tree Huffman codes are computed by constructing a Huffman tree each symbol corresponds to a leaf node in the tree edges to the left and right child of an inner node are labeled with 0 and 1, respectively the code of a symbol is obtained as the sequence of labels on the path from the root node to the corresponding leaf node 34

Huffman Tree Huffman tree is constructed in a bottom-up fashion as follows create a leaf node for each symbol x and assign probability P[x ] while there is more than one node left pick the two nodes x and y having the lowest probabilities replace the two nodes x and y by an inner node having probability P[x ] + P[y ] node x as left child with edge label 0 node y as right child with edge label 1 35

Huffman Codes Example: Let us assume that we have observed symbols a,,e with probabilities 1/1 1 0 1/2 1 Symbol: Code: 0 0 1/4 1 1/8 a 0 b 10 c 110 d 1110 e 1111 0 1 a b c d e 36

Huffman Codes Let us compute the expected number of bits per symbol For our example, this corresponds exactly to the entropy of the corresponding random variable, i.e. our code is optimal Huffman codes achieve an optimal average number of bits, whenever all probabilities are of the form 2 -i 37

Integer Encoding We now consider different methods to encode non-negative integers, since most symbols in the indexes of an IR system are integers (e.g., document identifiers or term frequencies) The common objective of these methods is to represent smaller integers using fewer bits or bytes One can also use these methods to encode symbols other than integers by assigning integer identifiers to them, so that more frequent symbols are represented by smaller integers 38

Variable-Byte Encoding Programming languages represent integers using a fixed number of bytes (e.g., 4 for int and 8 for long in Java), so that small numbers tend to have many leading 0s Variable-Byte Encoding (VBE) (also: 7-Bit Encoding) uses a variable number of bytes to represent a non-negative integer n and is based on two key ideas use only as many bytes as required to represent the integer use the first bit in every byte as a stop bit and the remaining seven bits as a payload to represent information stop bit 11010110 payload 39

Variable-Byte Encoding Example: Using 32 bits the integer 13 is represented as 00000000 00000000 00000000 00001101 With VBE it is represented as 10001101 Example: Using 32 bits the integer 131 is represented as 00000000 00000000 00000000 10000011 With VBE it is represented as 00000001 10000011 40

Variable-Byte Encoding Example: Using 32 bits the integer 1337 is represented as 00000000 00000000 00000101 00111001 With VBE it is represented as 00001010 10111001 41

Variable-Byte Encoding In VBE, the stop bit thus indicates whether the current byte is the last one belonging to the current non-negative integer VBE requires bytes to represent the non-negative integer n 42

Gamma Encoding γ Encoding is a bit-level encoding for positive integers The key idea is to split the representation of an integer n into its length which corresponds to the most significant bit an offset which corresponds to the bits after the most significant bit in the binary representation of n 43

Gamma Encoding The length in γ Encoding is stored using unary encoding, which represents the non-negative integer n as a sequence of n 1s followed by a 0 Examples: With unary encoding 0 is represented as 0 1 is represented as 10 7 is represented as 11111110 44

Gamma Encoding Example: With γ Encoding the integer 2 is represented as 10 0 Example: With γ Encoding the integer 13 is represented as 1110 101 Example: With γ Encoding the integer 131 is represented as 11111110 0000011 Example: With γ Encoding the integer 511 is represented as 111111110 11111111 45

Gamma Encoding The number of bits used for the offset always corresponds to the number of 1s in the unary encoding of the length This makes the code prefix free and allows us to interpret a sequence of bits unambiguously: when we have read the length, we know how many of the following bits belong to the same integer It can be shown that the average number of bits for γ Encoding is within a factor of 3 of an optimal encoding 46

Gap Encoding Whenever we have to encode a sequence of non-decreasing integers, we can apply gap encoding as follows represent the first integer explicitly represent all subsequent integers as the difference to their predecessor Example: With gap encoding, the integer sequence 1, 7, 128, 129, 131, 138 would be represented as 1, 6, 121, 1, 2, 7 47

Gap Encoding Note that we reconstruct the original integer sequence on-thefly while reading its gap-encoded representation first integer is used as is subsequent integers are added to the previous integer 1, 6, 121, 1, 2, 7 1, 7, 128, 129, 131, 138 Gap Encoding is only effective when used with another integer encoding method (e.g., VBE or Gamma Encoding) In the above example, when used with VBE, we can represent the numbers 128,, 138 using one instead of two bytes 48

Compressing Posting Lists How can we compress posting lists using these encodings? We will use VBE and Gap Encoding for illustration, and assume that we compress a document-ordered posting list Example: Document identifiers d 1 d 2 d 6 d 9 d 11 corresponds to the integer sequence which is encoded as 1, 1, 4, 5, 2 10000001 10000001 10000100 10000101 10000010 49

Compressing Posting Lists Example: Document identifiers and term frequencies d 2 5 d 4 7 d 7 1 d 9 3 Observe that only the document identifiers form a non-decreasing integer sequence, so that we can apply Gap Encoding only to those With Gap Encoding, we obtain the integer sequence 2, 5, 2, 7, 3, 1, 2, 3 which is represented as 10000010 10000101 10000010 10000111 10000011 50

Compressing Posting Lists Example: Document identifiers and positions d 2 [3, 7] d 7 [11] Observe that the document identifiers from different postings form a non-decreasing integer sequence; also the positions within each posting form a non-decreasing integer sequence With Gap-Encoding for both, we obtain the integer sequence 2, 2, 3, 4, 5, 1, 11 which also the lengths of position arrays and is encoded as 10000010 10000010 10000011 10000100 10000101 51

Compression Effectiveness / Decoding Speed Büttcher et al. [3] report compression effectiveness and decoding speeds on benchmark collections VBE tends to be less effective than Gamma Encoding, but allows for faster decoding speeds 52