archivist: Managing Data Analysis Results

Transkrypt

1 archivist: Managing Data Analysis Results Marcin Kosiński 1,2, Przemysław Biecek 2 1 IT Research and Development Grupa Wirtualna Polska 2 Faculty of Mathematics, Informatics and Mechanics, University of Warsaw 14 October, 2015

2 About Grupa Wirtualna Polska The Leader Of The Polish Internet IT Research and Development large-scale online learning personalized news article recommendation targeting text mining web user behavior identification 2 / 12 Figure : wp.pl

3 archivist: What does it do? Main features allows to store and archive objects in repositories (stored on a local disk or via GitHub/Dropbox) provides handy tools facilitating objects search and recovery ideally performs as cache supports the philosophy of reproducible research 3 / 12

4 archivist: Why is it useful? Solves reproducible research problems sometimes raw data are large or with limited access computations take a lot of time or require specialized hardware reproducibility requires specific versions of packages 4 / 12

5 archivist: Cache Use Case getmaxdistribution <- function ( D = rnorm, N = 10, R = ) { res <- replicate (R, max (D(N))) summary ( res ) } # or get directly from GitHub library ( archivist ) loadfromgithubrepo ( md5hash =" c", user =" MarcinKosinski ", repo =" Museum ") 5 / 12

6 archivist: Cache Use Case Using the archivist one can prepare a repository which stores calls and results of the cache() function to avoid their re-call in the future. cacherepo <- tempdir () createemptyrepo ( cacherepo ) system. time ( cache ( cacherepo, getmaxdistribution, rnorm, 10) ) user system elapsed system. time ( cache ( cacherepo, getmaxdistribution, rnorm, 10) ) user system elapsed / 12

7 archivist: Retrieving an Object Use Case Fragment of a poster Using archivist to retrieve an object aread ( MarcinKosinski / ISOBM /3b ) -> mutationsplot plot ( mutationsplot ) Figure : ggplot object Figure : downloaded object 7 / 12

8 archivist: Object s Pedigree Use Case createemptyrepo ( "FORUM_BI ", d e f a u l t = TRUE) i n v i s i b l e ( a o p t i o n s ( " s i l e n t ", TRUE) ) d a t a ( i r i s ) i r i s %a% d p l y r : : f i l t e r ( S e p a l. Length < 16) %a% lm ( Petal. Length~ Species, data =.) %a% summary ( ) > obj a h i s t o r y ( o b j ) i r i s [ ff575c261c949d073b2895b05d1097c3 ] > d p l y r : : f i l t e r ( S e p a l. Length < 16) [ 9 f b c d f 3 a c 6 f e 9 c ] > lm ( Petal. Length ~ Species, data =. ) [ 0 a82efeb8250a47718cea9d7f64e5ae7 ] > summary ( ) [ 671 a0b89fccdf02087acb002374a0fcd ] 8 / 12

9 archivist: Objects Exploration Within a Repository models <- asearch (" pbiecek / graphgallery ", patterns = c(" class :lm", " coefname : Sepal. Length ")) lapply ( models, coef ) [ [ 1 ] ] ( I n t e r c e p t ) S e p a l. Length [ [ 2 ] ] ( I n t e r c e p t ) S e p a l. Length S p e c i e s v e r s i c o l o r S p e c i e s v i r g i n i c a / 12

10 archivist: How does it work? library (" archivist ") 10 / 12

11 archivist: Plans & Prototypes Automated repository creation on github, commit, push and a return of a hook to an object. archive (iris, " MarcinKosinski ", " archivist - Museum - RforeveR _ last3 ", USER _ , USER _ PASSWORD, app _key, app _ secret, github _ token ) -> aread _ input # function returns a hook archivist :: aread (" MarcinKosinski / archivist - Museum - RforeveR _ last3 / ff575c261c949d073b2895b05d1097c3 ") archivist :: aread ( aread _ input ) -> x digest :: digest (x) [1] " ff575c261c949d073b2895b05d1097c3 " identical (iris,x) [1] TRUE 11 / 12

12 archivist: Learn More 12 / 12