Stability of Tikhonov Regularization Class 07, March 2003 Alex Rakhlin

HTML
DOWNLOAD

Wielkość: px

Rozpocząć pokaz od strony:

Download "Stability of Tikhonov Regularization Class 07, March 2003 Alex Rakhlin"

Maja Mucha
6 lat temu
Przeglądów:

1 Stability of Tikhonov Regularization Class 07, March 2003 Alex Rakhlin

2 Plan Review of Stability Bounds Stability of Tikhonov Regularization Algorithms

3 Uniform Stability Review notation: S = {z 1,..., z l }; S i,z = {z 1,..., z i 1, z, z i+1,..., z l } c(f, z) = V (f(x), y), where z = (x, y). An algorithm A has uniform stability β if (S, z) Z l+1, i, sup u Z c(f S, u) c(f S i,z, u) β. Last class: Uniform stability of β = O ( ) 1 l implies good generalization bounds. This class: Tikhonov Regularization has uniform stability of β = O ( 1 l ). Reminder: The Tikhonov Regularization algorithm: f S = arg min f H 1 l l i=1 V (f(x i ), y i ) + λ f 2 K

4 Generalization Bounds Via Uniform Stability If β = k l for some k, we have the following bounds from the last lecture: P ( I[f S ] I S [f S ] kl ) ( + ɛ lɛ 2 ) 2 exp 2(k + M) 2. Equivalently, with probability 1 δ, I[f S ] I S [f S ] + k l + (2k + M) 2 ln(2/δ). l

5 Lipschitz Loss Functions, I We say that a loss function (over a possibly bounded domain X ) is Lipschitz with Lipschitz constant L if y 1, y 2, y Y, V (y 1, y ) V (y 2, y ) L y 1 y 2. Put differently, if we have two functions f 1 and f 2, under an L-Lipschitz loss function, sup V (f 1 (x), y) V (f 2 (x), y) L f 1 f 2. (x,y) Yet another way to write it: c(f 1, ) c(f 2, ) L f 1 ( ) f 2 ( )

6 Lipschitz Loss Functions, II If a loss function is L-Lipschitz, then closeness of two functions (in L norm) implies that they are close in loss. The converse is false it is possible for the difference in loss of two functions to be small, yet the functions to be far apart (in L ). Example: constant loss. The hinge loss and the ɛ-insensitive loss are both L-Lipschitz with L = 1. The square loss function is L Lipschitz if we can bound the y values and the f(x) values generated. The 0 1 loss function is not L-Lipschitz at all an arbitrarily small change in the function can change the loss by 1: f 1 = 0, f 2 = ɛ, V (f 1 (x), 0) = 0, V (f 2 (x), 0) = 1.

7 Lipschitz Loss Functions for Stability Assuming L-Lipschitz loss, we transformed a problem of bounding sup u Z c(f S, u) c(f S i,z, u) into a problem of bounding f S f S i,z. As the next step, we bound the above L norm by the norm in the RKHS assosiated with a kernel K. For our derivations, we need to make another assumption: there exists a κ satisfying x X, K(x, x) κ.

8 Relationship Between L and L K Using the reproducing property and the Cauchy-Schwartz inequality, we can derive the following: x f(x) = K(x, ), f( ) K K(x, ) K f K = = K(x, ), K(x, ) f K K(x, x) f K κ f K Since above inequality holds for all x, we have f f K. Hence, if we can bound the RKHS norm, we can bound the L norm. Note that the converse is not true. Note that we now transformed the problem to bounding f S f S i,z K.

9 A Key Lemma We will prove the following lemma about Tikhonov regularization: f S f S i,z 2 K L f S f S i,z λl This theorem says that when we replace a point in the training set, the change in the RKHS norm (squared) of the difference between the two functions cannot be too large compared to the change in L. We will first explore the implications of this lemma, and defer its proof until later.

10 Bounding β, I Using our lemma and the relation between L K and L, f S f S i,z 2 K L f S f S i,z λl Lκ f S f S i,z K λl Dividing through by f S f S i,z K, we derive f S f S i,z K κl λl.

11 Bounding β, II Using again the relationship between L K and L, and the L Lipschitz condition, sup V (f S ( ), ) V (f S z,i( ), ) L f S f S z,i Lκ f S f S z,i K L2 κ 2 λl = β

12 Divergences Suppose we have a convex, differentiable function F, and we know F (f 1 ) for some f 1. We can guess F (f 2 ) by considering a linear approximation to F at f 1 : ˆF (f 2 ) = F (f 1 ) + f 2 f 1, F (f 1 ). The Bregman divergence is the error in this linearized approximation: d F (f 2, f 1 ) = F (f 2 ) F (f 1 ) f 2 f 1, F (f 1 ).

13 Divergences Illustrated d F (f 2, f 1 ) (f 2, F (f 2 )) (f 1, F (f 1 ))

14 Divergences Cont d We will need the following key facts about divergences: d F (f 2, f 1 ) 0 If f 1 minimizes F, then the gradient is zero, and d F (f 2, f 1 ) = F (f 2 ) F (f 1 ). If F = A + B, where A and B are also convex and differentiable, then d F (f 2, f 1 ) = d A (f 2, f 1 ) + d B (f 2, f 1 ) (the derivatives add).

15 The Tikhonov Functionals We shall consider the Tikhonov functional T S (f) = 1 l l i=1 V (f(x i ), y i ) + λ f 2 K, as well as the component functionals and V S (f) = 1 l l i=1 V (f(x i ), y i ) N(f) = f 2 K. Hence, T S (f) = V S (f) + λn(f). If the loss function is convex (in the first variable), then all three functionals are convex.

16 A Picture of Tikhonov Regularization Ts(f) Ts (f) R Vs(f) Vs (f) N(f) F fs fs

17 Proving the Lemma, I Let f S be the minimizer of T S, and let f S i,z be the minimizer of T S i,z, the perturbed data set with (x i, y i ) replaced by a new point z = (x, y). Then λ(d N (f S i,z, f S ) + d N (f S, f S i,z)) d TS (f S i,z, f S ) + d TS i,z (f S, f S i,z) = 1 l (c(f S i,z, z i ) c(f S, z i ) + c(f S, z) c(f S i,z, z)) 2L f S f S i,z. l We conclude that d N (f S i,z, f S ) + d N (f S, f S i,z) 2L f S f S i,z λl

18 Proving the Lemma, II But what is d N (f S i,z, f S )? We will express our functions as the sum of orthogonal eigenfunctions in the RKHS: f S (x) = f S i,z(x) = n=1 n=1 c n φ n (x) c nφ n (x) Once we express a function in this form, we recall that f 2 K = c 2 n n=1 λ n

19 Proving the Lemma, III Using this notation, we reexpress the divergence in terms of the c i and c i : d N (f S i,z, f S ) = f S i,z 2 K f S 2 K f S i,z f S, f S 2 K = = = We conclude that n=1 n=1 n=1 c 2 n λ n n=1 c 2 n λ n c 2 n + c 2 n 2c nc n λ n (c n c n ) 2 λ n = f S i,z f S 2 K i=1 (c n c n )( 2c n ) λ n d N (f S i,z, f S ) + d N (f S, f S i,z) = 2 f S i,z f S 2 K

20 Proving the Lemma, IV Combining these results proves our Lemma: f S i,z f S 2 K = d N(f S i,z, f S ) + d N (f S, f S i,z) 2 2L f S f S i,z λl

21 Bounding the Loss, I We have shown that Tikhonov regularization with an L- Lipschitz loss is β-stable with β = L2 κ 2 λl. If we want to actually apply the theorems and get the generalization bound, we need to bound the loss. Let C 0 be the maximum value of the loss when we predict a value of zero. If we have bounds on X and Y, we can find C 0.

22 Bounding the Loss, II Noting that the all 0 function 0 is always in the RKHS, we see that Therefore, λ f S 2 K T (f S) T ( 0) = 1 l C 0. l i=1 V ( 0(x i ), y i ) f S 2 K C 0 λ = f S κ f S K κ C0 λ Since the loss is L-Lipschitz, a bound on f S boundedness of the loss function. implies

23 A Note on λ We have shown that Tikhonov regularization is uniformly stable with β = L2 κ 2 λl. If we keep λ fixed( as we ) increase l, the generalization bound will tighten as O 1. However, keeping λ fixed is equiva- l lent to keeping our hypothesis space fixed. As we get more data, we want λ to get smaller. If λ gets smaller too fast, the bounds become trivial.

24 Tikhonov vs. Ivanov It is worth noting that Ivanov regularization ˆf H,S = arg min f H s.t. 1 l f 2 K τ l i=1 V (f(x i ), y i ) is not uniformly stable with β = O ( 1 n ), essentially because the constraint bounding the RKHS norm may not be tight. This is an important distinction between Tikhonov and Ivanov regularization.

Podobne dokumenty

Helena Boguta, klasa 8W, rok szkolny 2018/2019

Helena Boguta, klasa 8W, rok szkolny 2018/2019 Poniższy zbiór zadań został wykonany w ramach projektu Mazowiecki program stypendialny dla uczniów szczególnie uzdolnionych - najlepsza inwestycja w człowieka w roku szkolnym 2018/2019. Składają się na