Machine Learning for Data Science (CS4786) Lecture 24. Differential Privacy and Re-useable Holdout

Transkrypt

1 Machine Learning for Data Science (CS4786) Lecture 24 Differential Privacy and Re-useable Holdout

2 Defining Privacy

3 Defining Privacy Dataset +

4 Defining Privacy Dataset + Learning Algorithm Distribution of outcome

5 Defining Privacy Dataset + Learning Algorithm Distribution of outcome Dataset + Learning Algorithm Distribution of outcome

6 Defining Privacy Dataset + Learning Algorithm Distribution of outcome Similar Dataset + Learning Algorithm Distribution of outcome

7 Differential Privacy

8 Differential Privacy A deterministic algorithm cannot preserve privacy

9 Differential Privacy A deterministic algorithm cannot preserve privacy Say S = (Data1,,Datan) is the data provided to learning algorithm (be it clustering, supervised learning etc).

10 Differential Privacy A deterministic algorithm cannot preserve privacy Say S = (Data1,,Datan) is the data provided to learning algorithm (be it clustering, supervised learning etc). Say (randomized) learning algorithm A takes this training data and returns solution as A(S)

11 Differential Privacy A deterministic algorithm cannot preserve privacy Say S = (Data1,,Datan) is the data provided to learning algorithm (be it clustering, supervised learning etc). Say (randomized) learning algorithm A takes this training data and returns solution as A(S) Algorithm A is (ε,ẟ)- differentially private if for all samples S and S that only differ by one data point and any set C P (A(S) 2 C) apple e P (A(S 0 ) 2 C)+

12 Differential Privacy A deterministic algorithm cannot preserve privacy Say S = (Data1,,Datan) is the data provided to learning algorithm (be it clustering, supervised learning etc). Say (randomized) learning algorithm A takes this training data and returns solution as A(S) Algorithm A is (ε,ẟ)- differentially private if for all samples S and S that only differ by one data point and any set C P (A(S) 2 C) apple e P (A(S 0 ) 2 C)+ ẟ=0 is called pure differential privacy

14 Differential Privacy Dataset + Dataset + C

15 Differential Privacy Dataset + Dataset + C P (A(S) 2 C) apple e P (A(S 0 ) 2 C)+

16 Differential Privacy Dataset + Dataset + C P (A(S) 2 C) apple e P (A(S 0 ) 2 C)+ 1 (for small ε)

17 Obtaining Differential Privacy

18 Obtaining Differential Privacy Typical mechanism: Add noise to outcome or inside algorithm

19 Obtaining Differential Privacy Typical mechanism: Add noise to outcome or inside algorithm More privacy we want the more noise we add

20 Why it works?

21 Why it works? Take any arbitrary (possibly deterministic) function f(s).

22 <latexit sha1_base64="64q5fqjj9rkhqykoj7qojhjzeci=">aaacbnicbvdlsgmxfm3uv62vuzcibiu0bs0ziuhgkhxjsll7ghyymmmmdu1mhiqjlmlxbvwvny4uces3upnvtb8lbt1wuydz7iw5x4sylcqyvo3u0vlk6lp6pboxubw9y+7u1wuyc0xqogshahpiekydulnumdkmbehcy6th9a/hfuoecend4e4niujw1a2otzfswnlnwzk8gm2ohtykellnjyz+vlqap1c3xghomlmrae0af4k9i1kwq8u1v9qdemecbaozjgxltilljegoihkzzdqxjbhcfdqllu0dxil0kskzi3islq70q6eruhci/t5iejdywd09yzhqyxlvlp7ntwllxzojdajykqbph/jjbluix5nadhuekzbqbgfb9v8h7igbsnljzxqi9vzji6r+vrston17ni2vz3gkwqe4anlggwtqajegamoag0fwdf7bm/fkvbjvxsd0ngxmdvbbhxifp8omlh4=</latexit> <latexit sha1_base64="64q5fqjj9rkhqykoj7qojhjzeci=">aaacbnicbvdlsgmxfm3uv62vuzcibiu0bs0ziuhgkhxjsll7ghyymmmmdu1mhiqjlmlxbvwvny4uces3upnvtb8lbt1wuydz7iw5x4sylcqyvo3u0vlk6lp6pboxubw9y+7u1wuyc0xqogshahpiekydulnumdkmbehcy6th9a/hfuoecend4e4niujw1a2otzfswnlnwzk8gm2ohtykellnjyz+vlqap1c3xghomlmrae0af4k9i1kwq8u1v9qdemecbaozjgxltilljegoihkzzdqxjbhcfdqllu0dxil0kskzi3islq70q6eruhci/t5iejdywd09yzhqyxlvlp7ntwllxzojdajykqbph/jjbluix5nadhuekzbqbgfb9v8h7igbsnljzxqi9vzji6r+vrston17ni2vz3gkwqe4anlggwtqajegamoag0fwdf7bm/fkvbjvxsd0ngxmdvbbhxifp8omlh4=</latexit> <latexit sha1_base64="64q5fqjj9rkhqykoj7qojhjzeci=">aaacbnicbvdlsgmxfm3uv62vuzcibiu0bs0ziuhgkhxjsll7ghyymmmmdu1mhiqjlmlxbvwvny4uces3upnvtb8lbt1wuydz7iw5x4sylcqyvo3u0vlk6lp6pboxubw9y+7u1wuyc0xqogshahpiekydulnumdkmbehcy6th9a/hfuoecend4e4niujw1a2otzfswnlnwzk8gm2ohtykellnjyz+vlqap1c3xghomlmrae0af4k9i1kwq8u1v9qdemecbaozjgxltilljegoihkzzdqxjbhcfdqllu0dxil0kskzi3islq70q6eruhci/t5iejdywd09yzhqyxlvlp7ntwllxzojdajykqbph/jjbluix5nadhuekzbqbgfb9v8h7igbsnljzxqi9vzji6r+vrston17ni2vz3gkwqe4anlggwtqajegamoag0fwdf7bm/fkvbjvxsd0ngxmdvbbhxifp8omlh4=</latexit> <latexit sha1_base64="64q5fqjj9rkhqykoj7qojhjzeci=">aaacbnicbvdlsgmxfm3uv62vuzcibiu0bs0ziuhgkhxjsll7ghyymmmmdu1mhiqjlmlxbvwvny4uces3upnvtb8lbt1wuydz7iw5x4sylcqyvo3u0vlk6lp6pboxubw9y+7u1wuyc0xqogshahpiekydulnumdkmbehcy6th9a/hfuoecend4e4niujw1a2otzfswnlnwzk8gm2ohtykellnjyz+vlqap1c3xghomlmrae0af4k9i1kwq8u1v9qdemecbaozjgxltilljegoihkzzdqxjbhcfdqllu0dxil0kskzi3islq70q6eruhci/t5iejdywd09yzhqyxlvlp7ntwllxzojdajykqbph/jjbluix5nadhuekzbqbgfb9v8h7igbsnljzxqi9vzji6r+vrston17ni2vz3gkwqe4anlggwtqajegamoag0fwdf7bm/fkvbjvxsd0ngxmdvbbhxifp8omlh4=</latexit> Why it works? Take any arbitrary (possibly deterministic) function f(s). B = max f(s) S,S f(s0 ) 0 Say one data point where S and S differ on

23 <latexit sha1_base64="64q5fqjj9rkhqykoj7qojhjzeci=">aaacbnicbvdlsgmxfm3uv62vuzcibiu0bs0ziuhgkhxjsll7ghyymmmmdu1mhiqjlmlxbvwvny4uces3upnvtb8lbt1wuydz7iw5x4sylcqyvo3u0vlk6lp6pboxubw9y+7u1wuyc0xqogshahpiekydulnumdkmbehcy6th9a/hfuoecend4e4niujw1a2otzfswnlnwzk8gm2ohtykellnjyz+vlqap1c3xghomlmrae0af4k9i1kwq8u1v9qdemecbaozjgxltilljegoihkzzdqxjbhcfdqllu0dxil0kskzi3islq70q6eruhci/t5iejdywd09yzhqyxlvlp7ntwllxzojdajykqbph/jjbluix5nadhuekzbqbgfb9v8h7igbsnljzxqi9vzji6r+vrston17ni2vz3gkwqe4anlggwtqajegamoag0fwdf7bm/fkvbjvxsd0ngxmdvbbhxifp8omlh4=</latexit> <latexit sha1_base64="64q5fqjj9rkhqykoj7qojhjzeci=">aaacbnicbvdlsgmxfm3uv62vuzcibiu0bs0ziuhgkhxjsll7ghyymmmmdu1mhiqjlmlxbvwvny4uces3upnvtb8lbt1wuydz7iw5x4sylcqyvo3u0vlk6lp6pboxubw9y+7u1wuyc0xqogshahpiekydulnumdkmbehcy6th9a/hfuoecend4e4niujw1a2otzfswnlnwzk8gm2ohtykellnjyz+vlqap1c3xghomlmrae0af4k9i1kwq8u1v9qdemecbaozjgxltilljegoihkzzdqxjbhcfdqllu0dxil0kskzi3islq70q6eruhci/t5iejdywd09yzhqyxlvlp7ntwllxzojdajykqbph/jjbluix5nadhuekzbqbgfb9v8h7igbsnljzxqi9vzji6r+vrston17ni2vz3gkwqe4anlggwtqajegamoag0fwdf7bm/fkvbjvxsd0ngxmdvbbhxifp8omlh4=</latexit> <latexit sha1_base64="64q5fqjj9rkhqykoj7qojhjzeci=">aaacbnicbvdlsgmxfm3uv62vuzcibiu0bs0ziuhgkhxjsll7ghyymmmmdu1mhiqjlmlxbvwvny4uces3upnvtb8lbt1wuydz7iw5x4sylcqyvo3u0vlk6lp6pboxubw9y+7u1wuyc0xqogshahpiekydulnumdkmbehcy6th9a/hfuoecend4e4niujw1a2otzfswnlnwzk8gm2ohtykellnjyz+vlqap1c3xghomlmrae0af4k9i1kwq8u1v9qdemecbaozjgxltilljegoihkzzdqxjbhcfdqllu0dxil0kskzi3islq70q6eruhci/t5iejdywd09yzhqyxlvlp7ntwllxzojdajykqbph/jjbluix5nadhuekzbqbgfb9v8h7igbsnljzxqi9vzji6r+vrston17ni2vz3gkwqe4anlggwtqajegamoag0fwdf7bm/fkvbjvxsd0ngxmdvbbhxifp8omlh4=</latexit> <latexit sha1_base64="64q5fqjj9rkhqykoj7qojhjzeci=">aaacbnicbvdlsgmxfm3uv62vuzcibiu0bs0ziuhgkhxjsll7ghyymmmmdu1mhiqjlmlxbvwvny4uces3upnvtb8lbt1wuydz7iw5x4sylcqyvo3u0vlk6lp6pboxubw9y+7u1wuyc0xqogshahpiekydulnumdkmbehcy6th9a/hfuoecend4e4niujw1a2otzfswnlnwzk8gm2ohtykellnjyz+vlqap1c3xghomlmrae0af4k9i1kwq8u1v9qdemecbaozjgxltilljegoihkzzdqxjbhcfdqllu0dxil0kskzi3islq70q6eruhci/t5iejdywd09yzhqyxlvlp7ntwllxzojdajykqbph/jjbluix5nadhuekzbqbgfb9v8h7igbsnljzxqi9vzji6r+vrston17ni2vz3gkwqe4anlggwtqajegamoag0fwdf7bm/fkvbjvxsd0ngxmdvbbhxifp8omlh4=</latexit> Why it works? Take any arbitrary (possibly deterministic) function f(s). B = max f(s) S,S f(s0 ) 0 Say one data point where S and S differ on A(S) = f(s) + B Laplace(0,1)/ε

24 <latexit sha1_base64="64q5fqjj9rkhqykoj7qojhjzeci=">aaacbnicbvdlsgmxfm3uv62vuzcibiu0bs0ziuhgkhxjsll7ghyymmmmdu1mhiqjlmlxbvwvny4uces3upnvtb8lbt1wuydz7iw5x4sylcqyvo3u0vlk6lp6pboxubw9y+7u1wuyc0xqogshahpiekydulnumdkmbehcy6th9a/hfuoecend4e4niujw1a2otzfswnlnwzk8gm2ohtykellnjyz+vlqap1c3xghomlmrae0af4k9i1kwq8u1v9qdemecbaozjgxltilljegoihkzzdqxjbhcfdqllu0dxil0kskzi3islq70q6eruhci/t5iejdywd09yzhqyxlvlp7ntwllxzojdajykqbph/jjbluix5nadhuekzbqbgfb9v8h7igbsnljzxqi9vzji6r+vrston17ni2vz3gkwqe4anlggwtqajegamoag0fwdf7bm/fkvbjvxsd0ngxmdvbbhxifp8omlh4=</latexit> <latexit sha1_base64="64q5fqjj9rkhqykoj7qojhjzeci=">aaacbnicbvdlsgmxfm3uv62vuzcibiu0bs0ziuhgkhxjsll7ghyymmmmdu1mhiqjlmlxbvwvny4uces3upnvtb8lbt1wuydz7iw5x4sylcqyvo3u0vlk6lp6pboxubw9y+7u1wuyc0xqogshahpiekydulnumdkmbehcy6th9a/hfuoecend4e4niujw1a2otzfswnlnwzk8gm2ohtykellnjyz+vlqap1c3xghomlmrae0af4k9i1kwq8u1v9qdemecbaozjgxltilljegoihkzzdqxjbhcfdqllu0dxil0kskzi3islq70q6eruhci/t5iejdywd09yzhqyxlvlp7ntwllxzojdajykqbph/jjbluix5nadhuekzbqbgfb9v8h7igbsnljzxqi9vzji6r+vrston17ni2vz3gkwqe4anlggwtqajegamoag0fwdf7bm/fkvbjvxsd0ngxmdvbbhxifp8omlh4=</latexit> <latexit sha1_base64="64q5fqjj9rkhqykoj7qojhjzeci=">aaacbnicbvdlsgmxfm3uv62vuzcibiu0bs0ziuhgkhxjsll7ghyymmmmdu1mhiqjlmlxbvwvny4uces3upnvtb8lbt1wuydz7iw5x4sylcqyvo3u0vlk6lp6pboxubw9y+7u1wuyc0xqogshahpiekydulnumdkmbehcy6th9a/hfuoecend4e4niujw1a2otzfswnlnwzk8gm2ohtykellnjyz+vlqap1c3xghomlmrae0af4k9i1kwq8u1v9qdemecbaozjgxltilljegoihkzzdqxjbhcfdqllu0dxil0kskzi3islq70q6eruhci/t5iejdywd09yzhqyxlvlp7ntwllxzojdajykqbph/jjbluix5nadhuekzbqbgfb9v8h7igbsnljzxqi9vzji6r+vrston17ni2vz3gkwqe4anlggwtqajegamoag0fwdf7bm/fkvbjvxsd0ngxmdvbbhxifp8omlh4=</latexit> <latexit sha1_base64="64q5fqjj9rkhqykoj7qojhjzeci=">aaacbnicbvdlsgmxfm3uv62vuzcibiu0bs0ziuhgkhxjsll7ghyymmmmdu1mhiqjlmlxbvwvny4uces3upnvtb8lbt1wuydz7iw5x4sylcqyvo3u0vlk6lp6pboxubw9y+7u1wuyc0xqogshahpiekydulnumdkmbehcy6th9a/hfuoecend4e4niujw1a2otzfswnlnwzk8gm2ohtykellnjyz+vlqap1c3xghomlmrae0af4k9i1kwq8u1v9qdemecbaozjgxltilljegoihkzzdqxjbhcfdqllu0dxil0kskzi3islq70q6eruhci/t5iejdywd09yzhqyxlvlp7ntwllxzojdajykqbph/jjbluix5nadhuekzbqbgfb9v8h7igbsnljzxqi9vzji6r+vrston17ni2vz3gkwqe4anlggwtqajegamoag0fwdf7bm/fkvbjvxsd0ngxmdvbbhxifp8omlh4=</latexit> Why it works? Take any arbitrary (possibly deterministic) function f(s). B = max f(s) S,S f(s0 ) 0 Say one data point where S and S differ on A(S) = f(s) + B Laplace(0,1)/ε A is (ε,0)-differentially private

25 <latexit sha1_base64="wnll5r0l7s6ctnk7odgkot6x4+a=">aaacbxicdzdlsgmxfiyzxmu9jbrurbci7aykrwylm6iblxxtbdpsmmmmdc1csdjigbpx46u4cagiw9/bnw9jphdq0qobj/8/h5pzo6hgsip0as0sli2vrkbw0usbm1vb9s5uxqwrpkxgaxhipkmue9xnnc21ym1qmui5gjwc4uxin26zvdzwb/qozb2p9h3uckq0kbr2qduvhmzh9u7mzv7ncum5hhvu2hmurwhhjgecuhikdjtlpqiuqzxypjjgvtwu/dhubttymk+pieq1map1jyzscyryon2ofasjhzi+axn0icduj55cmyzhrulbn5dm+rpo1o8tmfgugnmo6fsihqjfxil+5bui7zy6mffdsdofthe5kya6gekksmclo1qmdbaqufkrpaniytemulqjyx4p/b/qhtxgexx1kqmcz+jigx1wcliagykogetqbtvawt14bm/gxxqwnqxx623aumdnzvbaj7levwawczce</latexit> <latexit sha1_base64="wnll5r0l7s6ctnk7odgkot6x4+a=">aaacbxicdzdlsgmxfiyzxmu9jbrurbci7aykrwylm6iblxxtbdpsmmmmdc1csdjigbpx46u4cagiw9/bnw9jphdq0qobj/8/h5pzo6hgsip0as0sli2vrkbw0usbm1vb9s5uxqwrpkxgaxhipkmue9xnnc21ym1qmui5gjwc4uxin26zvdzwb/qozb2p9h3uckq0kbr2qduvhmzh9u7mzv7ncum5hhvu2hmurwhhjgecuhikdjtlpqiuqzxypjjgvtwu/dhubttymk+pieq1map1jyzscyryon2ofasjhzi+axn0icduj55cmyzhrulbn5dm+rpo1o8tmfgugnmo6fsihqjfxil+5bui7zy6mffdsdofthe5kya6gekksmclo1qmdbaqufkrpaniytemulqjyx4p/b/qhtxgexx1kqmcz+jigx1wcliagykogetqbtvawt14bm/gxxqwnqxx623aumdnzvbaj7levwawczce</latexit> <latexit sha1_base64="wnll5r0l7s6ctnk7odgkot6x4+a=">aaacbxicdzdlsgmxfiyzxmu9jbrurbci7aykrwylm6iblxxtbdpsmmmmdc1csdjigbpx46u4cagiw9/bnw9jphdq0qobj/8/h5pzo6hgsip0as0sli2vrkbw0usbm1vb9s5uxqwrpkxgaxhipkmue9xnnc21ym1qmui5gjwc4uxin26zvdzwb/qozb2p9h3uckq0kbr2qduvhmzh9u7mzv7ncum5hhvu2hmurwhhjgecuhikdjtlpqiuqzxypjjgvtwu/dhubttymk+pieq1map1jyzscyryon2ofasjhzi+axn0icduj55cmyzhrulbn5dm+rpo1o8tmfgugnmo6fsihqjfxil+5bui7zy6mffdsdofthe5kya6gekksmclo1qmdbaqufkrpaniytemulqjyx4p/b/qhtxgexx1kqmcz+jigx1wcliagykogetqbtvawt14bm/gxxqwnqxx623aumdnzvbaj7levwawczce</latexit> <latexit sha1_base64="wnll5r0l7s6ctnk7odgkot6x4+a=">aaacbxicdzdlsgmxfiyzxmu9jbrurbci7aykrwylm6iblxxtbdpsmmmmdc1csdjigbpx46u4cagiw9/bnw9jphdq0qobj/8/h5pzo6hgsip0as0sli2vrkbw0usbm1vb9s5uxqwrpkxgaxhipkmue9xnnc21ym1qmui5gjwc4uxin26zvdzwb/qozb2p9h3uckq0kbr2qduvhmzh9u7mzv7ncum5hhvu2hmurwhhjgecuhikdjtlpqiuqzxypjjgvtwu/dhubttymk+pieq1map1jyzscyryon2ofasjhzi+axn0icduj55cmyzhrulbn5dm+rpo1o8tmfgugnmo6fsihqjfxil+5bui7zy6mffdsdofthe5kya6gekksmclo1qmdbaqufkrpaniytemulqjyx4p/b/qhtxgexx1kqmcz+jigx1wcliagykogetqbtvawt14bm/gxxqwnqxx623aumdnzvbaj7levwawczce</latexit> Why it works? P p(x; (A(S) f(s)) =x) P (A(S 0 )=x) = e f(s) p(x; f(s 0 )) e f(s0 ) x /B x /B = e f(s0 ) x f(s) x B apple e f(s0 ) B f(s) apple e

30 The Magic Broker Scam Based on Aaron Roth s slide

31 The Magic Broker Scam Based on Aaron Roth s slide Day 1

32 The Magic Broker Scam Based on Aaron Roth s slide Day 1 Day 2

33 The Magic Broker Scam Based on Aaron Roth s slide Day 1 Day 2 Day 3

34 The Magic Broker Scam Based on Aaron Roth s slide Day 1 Day 2 Day 3 Day 10..

35 The Magic Broker Scam Based on Aaron Roth s slide Day 1 Day 2 Day 3 Day 10.. Day 11: Pay me I will help you invest

36 The Magic Broker Scam Based on Aaron Roth s slide Day 1 Day 2 Day 3 Day 10.. Day 11: Pay me I will help you invest What are the chances that some one guesses randomly and gets correct?

37 The Magic Broker Scam Based on Aaron Roth s slide Day 1 Day 2 Day 3 Day 10.. Day 11: Pay me I will help you invest What are the chances that some one guesses randomly and gets correct? 1 in 1024

38 The Magic Broker Scam Based on Aaron Roth s slide Day 1 Day 2 Day 3 Day 10.. Day 11: Pay me I will help you invest What are the chances that some one guesses randomly and gets correct? 1 in 1024 But the broker will always scam people, why?

39 The Magic Broker Scam Based on Aaron Roth s slide Day 1 Day 2 Day 3 Day 10..

40 The Magic Broker Scam Based on Aaron Roth s slide Day 1 Day 2 Day 3 Day 10.. Day 1: send to a million people (1/2 + and 1/2 -)

41 The Magic Broker Scam Based on Aaron Roth s slide Day 1 Day 2 Day 3 Day 10.. Day 1: send to a million people (1/2 + and 1/2 -) Day 2: only send to 500,000 people we got right (1/2 + and 1/2 -)

42 The Magic Broker Scam Based on Aaron Roth s slide Day 1 Day 2 Day 3 Day 10.. Day 1: send to a million people (1/2 + and 1/2 -) Day 2: only send to 500,000 people we got right (1/2 + and 1/2 -)..

43 The Magic Broker Scam Based on Aaron Roth s slide Day 1 Day 2 Day 3 Day 10.. Day 1: send to a million people (1/2 + and 1/2 -) Day 2: only send to 500,000 people we got right (1/2 + and 1/2 -).. Day 11: We are left with /1024 ~ 1000 people we can scam

44 Reproducibility problem

45 Ideal ML Experiment

46 Data Ideal ML Experiment

47 Ideal ML Experiment Data Training Holdout

48 Ideal ML Experiment Data Training Holdout Reuse many times

49 Ideal ML Experiment Data Training Holdout Reuse many times Form hypothesis

50 Ideal ML Experiment Data Training Holdout Reuse many times Form hypothesis Use holdout set only once and report accuracy

51 Ideal ML Experiment Data Training Holdout Reuse many times Form hypothesis Use holdout set only once and report accuracy We are not allowed to form hypothesis based on data we used to test: age old statistics

52 Ideal ML Experiment Data Training Holdout Reuse many times Form hypothesis Use holdout set only once and report accuracy We are not allowed to form hypothesis based on data we used to test: age old statistics But too tempting to form more informed opinion

53 Ideal ML Experiment Data Training Holdout Reuse many times Form hypothesis We are not allowed to form hypothesis based on data we used to test: age old statistics But too tempting to form more informed opinion We do a train/validation/test set split Use holdout set only once and report accuracy

54 Ideal ML Experiment Data Training Holdout Reuse many times Form hypothesis We are not allowed to form hypothesis based on data we used to test: age old statistics But too tempting to form more informed opinion We do a train/validation/test set split Use holdout set only once and report accuracy But this means we can y reuse datasets over time

55 ML Today

56 ML Today Benchmark dataset from MNIST to IMAGENET

57 ML Today Benchmark dataset from MNIST to IMAGENET Competitions run with feedback from test set to competitors ;)

58 ML Today Benchmark dataset from MNIST to IMAGENET Competitions run with feedback from test set to competitors ;) Effort to collect public dataset to share.

59 ML Today Benchmark dataset from MNIST to IMAGENET Competitions run with feedback from test set to competitors ;) Effort to collect public dataset to share. All great but we need to be careful!

60 ML Today Benchmark dataset from MNIST to IMAGENET Competitions run with feedback from test set to competitors ;) Effort to collect public dataset to share. All great but we need to be careful! Old fix: pre-register experiment, many many examples of people fudging this.

61 An Example Code at: Data: 20,000 data points in dimensions drawn randomly Labels: random +1 or -1 (no correlation with input) Data-scientist runs: Split data into train and test of equal size Select best k features on training data Only use variables also good on holdout Build linear predictor out of these k variables Find best k = 10,20,30,40,.

62 An Example Number of features

63 Can we reuse holdout set?

64 Maybe if we do it right

65 Maybe if we do it right Idea: when we report back accuracies from the dataset, we add noise so as to not to leak too much information

66 Maybe if we do it right Idea: when we report back accuracies from the dataset, we add noise so as to not to leak too much information Eg. Report back accuracies on holdout set only when training and test accuracy are significantly different

67 Threshold-out Algorithm

68 The Example

69 The Guarantee Roughly: Thresholdout guarantees high accuracy, and can handle up to asked is order n^2 overfitting queries

71 Differential Privacy A deterministic algorithm cannot preserve privacy

72 Differential Privacy A deterministic algorithm cannot preserve privacy Say S = (Data1,,Datan) is the data provided to learning algorithm (be it clustering, supervised learning etc).

73 Differential Privacy A deterministic algorithm cannot preserve privacy Say S = (Data1,,Datan) is the data provided to learning algorithm (be it clustering, supervised learning etc). Say (randomized) learning algorithm A takes this training data and returns solution as A(S)

74 Differential Privacy A deterministic algorithm cannot preserve privacy Say S = (Data1,,Datan) is the data provided to learning algorithm (be it clustering, supervised learning etc). Say (randomized) learning algorithm A takes this training data and returns solution as A(S) Algorithm A is (ε,ẟ)- differentially private if for all samples S and S that only differ by one data point and any set C P (A(S) 2 C) apple e P (A(S 0 ) 2 C)+

75 Differential Privacy A deterministic algorithm cannot preserve privacy Say S = (Data1,,Datan) is the data provided to learning algorithm (be it clustering, supervised learning etc). Say (randomized) learning algorithm A takes this training data and returns solution as A(S) Algorithm A is (ε,ẟ)- differentially private if for all samples S and S that only differ by one data point and any set C P (A(S) 2 C) apple e P (A(S 0 ) 2 C)+ ẟ=0 is called pure differential privacy

76 Differential Privacy Post-processing: If A is a differentially private algorithm, and f is any mapping on the outcome space of A, then the algorithm that maps from sample S to f(a(s)) is also differentially private Composability: Algo(S) is given by following procedure For i in 1 to k If we choose any epsilon differentially private algorithm A_i based on outcomes O_1,,O_{i-1} Set O_i = A_i(S) Return O_1,,O_n The above algorithm is O(epsilon sqrt(k)) differentially private