Dealing with continuous-valued attributes

HTML
DOWNLOAD

Wielkość: px

Rozpocząć pokaz od strony:

Download "Dealing with continuous-valued attributes"

Przybysław Adamski
7 lat temu
Przeglądów:

1 Dealing with continuous-valued attributes

2 An alternative measure: gain ratio

3 Handling incomplete training data

4 Handling attributes with different costs

5 Ensemble of Classifiers

6 Why Ensemble Works? Some unknown distribution Model 1 Model 2 Model 3 Model 4 Model 5 Model 6

7 Ensemble of Classifiers Data model 1 model 2 Ensemble model Combine multiple models into one! model k Basic idea is to learn a set of classifiers (experts) and to allow them to vote. Advantage: improvement in predictive accuracy. Disadvantage: it is difficult to understand an ensemble of classifiers.

8 Generating Base Classifiers Sampling training examples Train k classifiers on k subsets drawn from the training set Using different learning models Use all the training examples, but apply different learning algorithms Sampling features Train k classifiers on k subsets of features drawn from the feature space Learning randomly Introduce randomness into learning procedures

9 Majority vote D Original Training data Step 1: Build Multiple Classifiers C 1 C 2 C t -1 C t Step 2: Combine Classifiers C *

10 Why Majority Voting works? Suppose there are 25 base classifiers Each classifier has error rate, = 0.35 Assume errors made by classifiers are uncorrelated Probability that the ensemble classifier makes a wrong prediction: P( X 13) 25 i i (1 ) i 25 i 0.06

11 Bagging Generate a random sample from training set Repeat this sampling procedure, getting a sequence of K independent training sets A corresponding sequence of classifiers C1,C2,,Ck is constructed for each of these training sets, by using the same classification algorithm To classify an unknown sample X, let each classifier predict. The Bagged Classifier C* then combines the predictions of the individual classifiers to generate the final outcome. (sometimes combination is simple voting)

12 Bagging classifiers Classifier generation Let n be the size of the training set. For each of t iterations: Sample n instances with replacement from the training set. Apply the learning algorithm to the sample. Store the resulting classifier. classification For each of the t classifiers: Bootstrap samples and Predict class of instance using classifier. Return class that classifiers: was predicted most often.

13 Bagging classifiers X 2

14 Boosting The final prediction is a combination of the prediction of several predictors. Differences between Boosting and previous methods? Its iterative. Boosting: Successive classifiers depends upon its predecessors. Previous methods : Individual classifiers were independent. Training Examples may have unequal weights. Look at errors from previous classifier step to decide how to focus on next iteration over data Set weights to focus more on hard examples. (the ones on which we committed mistakes in the previous iterations)

15 AdaBoost (algorithm) W(x) is the distribution of weights over the N training points W(x i )=1 Initially assign uniform weights W 0 (x) = 1/N for all x. At each iteration k : Find best weak classifier C k (x) using weights W k (x) Compute ε k the error rate as ε k = [ W(x i ) I(y i C k (x i )) ] / [ W(x i )] weight α k the classifier C k s weight in the final hypothesis Set α k = log ((1 ε k )/ε k ) For each x i, W k+1 (x i ) = W k (x i ) exp[α k I(y i C k (x i ))] C FINAL (x) =sign [ α i C i (x) ] L(y, f (x)) = exp(-y f (x)) - the exponential loss function

16 AdaBoost - example Original Training set : Equal Weights to all training samples

17 AdaBoost - example ROUND 1

18 AdaBoost - example ROUND 2

19 AdaBoost - example ROUND 3

20 AdaBoost - example

21 N examples Random Forest M features Take he majority vote

22 Random Forest Classifier generation Let n be the size of the training set. For each of t iterations: (1) Sample n instances with replacement from the training set. (2) Learn a decision tree s.t. the variable for any new node is the best variable among m randomly selected variables. (3) Store the resulting decision tree. Classification For each of the t decision trees: Predict class of instance. Return class that was predicted most often.

23 Rodzaje drzew decyzyjnych Podział drzew ze wzglęgu na rodzaj badań: drzewa klasyfikacyjne drzewa regresyjne (np. CART, REPTree) drzewa modelowe (np. M5, SMOTI) Odpowiedź na pytanie typu: Każdy liść zawiera model regresji liniowej (bądź nieliniowej). Czy Jaka otrzymam jest moja kredyt? zdolność kredytowa? Znajduje rozwiązanie zapewniające możliwie maksymalną dokładność parametrycznej reprezentacji funkcji docelowej. nie < Umowa o pracę tak Roczny przychód < >= Zadłużenie < 80 Przykład decyzji w liściu Liście Każdy w liść drzewie drzewa klasyfikacyjnym regresyjnego zawierają wartość decyzję średnią (klasę) zmiennej zależnej (przewidywanej) wszystkich obiektów w nim się znajdujących >= Wiek Przykładowa funkcja docelowa: >= 80 Zdolność kredytowa = 4*roczny przychód 1.5*zadłużenie 0.7*wiek

24 Regression trees Build a regression tree: Divide the predictor space into J distinct not overlapping regions R 1,R 2,R 3,,R J We make the same prediction for all observations in the same region; use the mean of responses for all training observations that are in the region

25 Regression trees

26 Regression trees

27 Regression trees

28 Recursive binary splitting

29 Regression tree - example

30 Overfitting

31 Regression Trees Like decision trees, but: Splitting criterion: minimize intra-subset variation Termination criterion: std. dev. becomes small Pruning criterion: based on numeric error measure Prediction: Leaf predicts average class value of instances Yields piecewise constant functions Easy to interpret More sophisticated version: model trees

32 Model trees Build a regression tree Each leaf linear regression function Smoothing: factor in ancestor s predictions Smoothing formula: Same effect can be achieved by incorporating ancestor models into the leaves Need linear regression function at each node At each node, use only a subset of attributes to build linear regression model Those occurring in subtree p' = np+ kq n+ k (+maybe those occurring in path to the root) Fast: tree usually uses only a small subset of the attributes

33 Building the tree Splitting: standard deviation reduction / squared error reduction Termination of splitting process: Standard deviation < 5% of its value on full training set Too few instances remain (e.g., < 4) Pruning: Heuristic estimate of n+ absolute v error of linear regression models: n- v average_absolute_error Greedily remove terms from LR models to minimize estimated error Proceed bottom up: compare error of LR model at internal node to error of subtree (this happens before smoothing is applied) Heavy pruning: single model may replace whole subtree

dobre rezultaty gdy: modele na ścieżce przewidują różne

34 Metody wygładzania (smoothing) Wymaga wygenerowania modelu liniowego dla każdego węzła wewnętrznego w drzewie Osiąga dobre rezultaty gdy: modele na ścieżce przewidują różne wartości modele konstruowane są dla niewielkiej ilości obiektów uczących

35 Inne aspekty Umożliwienie poszukiwania rozwiązań gdy koszty niedoszacowania i przeszacowania są inne Różne funkcje kosztówkosztów np. LinLin, QuadQuad, LinEx Wartości brakujące Surrogate splits

36 Drzewa modelowe / regresja liniowa

37 Drzewa modelowe / regresja liniowa Zagrożenie gdy nowe dane są spoza zakresu!!!

rozwiązanie podziały minimalizują odchylenie standardowe.

38 Globalna vs lokalna indukcja Przykład sztucznego zbioru danych opisanego funkcją: Lokalnie Globalnie optymalne optymalne (zachłanne) rozwiązanie rozwiązanie podziały minimalizują odchylenie standardowe. Kolejne testy Pierwszy są już wynikiem podział nieoptymalnego węzła jest dla x 1 podziału > -1.2 w korzeniu

39 Algorytmy ewolucyjne i drzewa decyzyjne Algorytmy ewolucyjne: zbiór metod optymalizacji inspirowany naturalnym procesem ewolucji wykorzystują oparte na populacji losowe różnicowanie i selekcję wzajemne przenikanie się różnych technik: algorytmy genetyczne, strategie ewolucyjne, programowanie genetyczne,... efektywne w unikaniu minimów lokalnych Algorytmy ewolucyjne jako narzędzie indukcji drzew: umożliwia równoczesne poszukiwanie struktury drzewa oraz wszystkich testów możliwość wykorzystania znajomości problemu

40 Gdzie dalej? Ewolucyjna indukcja drzew Drzewa rozmyte (fuzzy) Algorytmy równoległe i rozproszone - MPI/OpenMP/GPGPU, Hadoop, Hive

41 Soft Computing / Evolutionary Computation

42 Typical framework of EA algorihm

Selection Selection is a procedure of picking parent chromosome to produce off-spring. Types of selection: Random Selection Parents are selected randomly from the population.

43 Selection Selection is a procedure of picking parent chromosome to produce off-spring. Types of selection: Random Selection Parents are selected randomly from the population. Proportional Selection probabilities for picking each chromosome is calculated as: P(x i ) = f(x i )/Σf(x j ) for all j Rank Based Selection This method uses ranks instead of absolute fitness values. P(x i ) = (1/β)(1 e r(x i ) )

44 Roulette Wheel Selection Let i = 1, where i denotes chromosome index; Calculate P(x i ) using proportional selection; sum = P(x i ); choose r ~ U(0,1); while sum < r do i = i + 1; i.e. next chromosome sum = sum + P(x i ); end return x i as one of the selected parent; repeat until all parents are selected

45 Reproduction Reproduction is a processes of creating new chromosomes out of chromosomes in the population. Parents are put back into population after reproduction. Cross-over and Mutation are two parts in reproduction of an off-spring. Cross-over : It is a process of creating one or more new individuals through the combination of genetic material randomly selected from two or parents.

46 Cross-over Uniform cross-over : where corresponding bit positions are randomly exchanged between two parents. One point : random bit is selected and entire substring after the bit is swapped. Two point : two bits are selected and the substring between the bits is swapped. Uniform Cross-over One point Cross-over Two point Cross-over Parent1 Parent Off-spring1 Off-spring

47 Mutation Mutation procedures depend upon the representation schema of the chromosomes. This is to prevent falling all solutions in population into a local optimum. For a bit-vector representation: random mutation : randomly negates bits in-order mutation : performs random mutation between two randomly selected bit position. Random Mutation In-order Mutation Before mutation After mutation

Podobne dokumenty

Hard-Margin Support Vector Machines

Hard-Margin Support Vector Machines aaacaxicbzdlssnafiyn9vbjlepk3ay2gicupasvu4iblxuaw2hjmuwn7ddjjmxm1bkcg1/fjqsvt76fo9/gazqfvn8y+pjpozw5vx8zkpvtfxmlhcwl5zxyqrm2vrg5zw3vxmsoezi4ogkr6phieky5crvvjhriqvdom9l2xxftevuwcekj3lktmhghgniauiyutvrwxtvme34a77kbvg73gtygpjsrfati1+xc8c84bvraowbf+uwnipyehcvmkjrdx46vlykhkgykm3ujjdhcyzqkxy0chur6ax5cbg+1m4bbjptjcubuz4kuhvjoql93hkin5hxtav5x6yyqopnsyuneey5ni4keqrxbar5wqaxbik00icyo/iveiyqqvjo1u4fgzj/8f9x67bzmxnurjzmijtlybwfgcdjgfdtajwgcf2dwaj7ac3g1ho1n4814n7wwjgjmf/ys8fenfycuzq==