Data-driven Coordination of Distributed Energy Resources

Data-driven Coordination of Distributed Energy Resources Alejandro D. Domínguez-García Coordinated Science Laboratory Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign PSERC Webinar Series November 5, 2019

Outline 1 Introduction 2 DER Coordination for Active Power Provision 3 LTC Coordination for Voltage Regulation 4 Concluding Remarks

Impacts of Renewable-Based Power Generation Resources Deep penetration of renewable-based generation imposes additional requirements on ancillary services including: Frequency regulation (in bulk power systems) Reactive power support (in distribution systems) Frequency regulation in bulk power systems is typically achieved by controlling large synchronous generators Resources in distribution systems are not utilized for this task Reactive power support in distribution systems is provided by devices such as load tap changers (LTCs) and fixed/switched capacitors These devices are not designed to manage high variability in voltage fluctuations induced by renewable-based generation Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 1 / 32

The Solution An increasing number of DERs are being integrated into distribution systems DERs could potentially be utilized to provide ancillary services if properly coordinated by, e.g., an aggregator PV systems Electric Vehicles Fuel Cells Residential Storage Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 2 / 32

Need for Data-Driven Coordination G tie line aggregator G bulk power system power distribution system DER aggregators needs to develop appropriate coordination schemes so DERs can collectively provide services that meet certain requirements Model-based schemes may be infeasible due to the lack of accurate models Data-driven schemes that only rely on measurements provide a promising alternative for developing efficient coordination schemes Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 3 / 32

Presentation Overview Objective To develop data-driven coordination frameworks for assets (DERs, LTCs) in distribution systems Part I. Active power provision problem total active power exchanged between the distribution and bulk systems needs to equal to some amount requested by the bulk system operator Part II. Voltage regulation problem the voltage magnitude at each bus needs be maintained to stay close to some reference value Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 4 / 32

0<latexit sha1_base64="rspgdo38dcurlsat/ftnosrchua=">aaab6hicbvbns8naej3ur1q/qh69lbbbu0mqomeif48t2fpoq9lsj+3azsbsboqs+gu8efdeqz/jm//gbzudtj4yelw3w8y8ibfcg9f9dgpr6xubw8xt0s7u3v5b+fcorenumwyxwmsqe1cngktsgw4edhkfnaoepgtj25n/8irk81jem0mcfkshkoecuwolptsvv9yqowdzjv5okpcj0s9/9qyxsyouhgmqdddze+nnvbnobe5lvvrjqtmydrfrqaqraj+bhzolz1yzkdbwtqqhc/x3reyjrsdrydsjakz62zuj/3nd1itxfszlkhqubleotauxmzl9tqzcitniygllittbcrtrrzmx2zrscn7yy6ukxat6f9va87jsv8njkmijnmi5ehafdbidbrsaacizvmkb8+i8oo/ox6k14oqzx/ahzucpememua==</latexit> 1<latexit sha1_base64="ttigpqprnje4hss++pum3etxya8=">aaab6hicbvbns8naej3ur1q/qh69lbbbu0mqomeif48t2fpoq9lsj+3azsbsboqs+gu8efdeqz/jm//gbzudtj4yelw3w8y8ibfcg9f9dgpr6xubw8xt0s7u3v5b+fcorenumwyxwmsqe1cngktsgw4edhkfnaoepgtj25n/8irk81jem0mcfkshkoecuwolptcvv9yqowdzjv5okpcj0s9/9qyxsyouhgmqdddze+nnvbnobe5lvvrjqtmydrfrqaqraj+bhzolz1yzkdbwtqqhc/x3reyjrsdrydsjakz62zuj/3nd1itxfszlkhqubleotauxmzl9tqzcitniygllittbcrtrrzmx2zrscn7yy6ukxat6f9va87jsv8njkmijnmi5ehafdbidbrsaacizvmkb8+i8oo/ox6k14oqzx/ahzucpe+umuq==</latexit> 3<latexit sha1_base64="lnugvcgcwmrkaqmqfwjgrne8jzm=">aaab6hicbvdltgjbeozff+il9ehlijhxrhbbri9elx4hkuccgzi79mli7oxmztaeel7aiwen8eonefnvhgapclbssawqo91dqsk4nq777eq2nre2d/k7hb39g8oj4vfjs8epythksyhvj6aabzfynnwi7cqkarqibafju7nffkkleswfzcrbp6jdyupoqlfso9ovltyyuwbzj15gspch3i9+9qyxsyouhgmqdddze+npqtkcczwveqnghlixhwlxukkj1p50ceimxfhlqmjy2zkglntfe1maat2jatszutpsq95c/m/rpia88adcjqlbyzalwlqqe5p512taftijjpzqpri9lbarvzqzm03bhuctvrxowpwyvy1xglel2m0wrx7o4bwuwynrqme91kejdbce4rxenefnxxl3ppatoseboyu/cd5/ah7zjls=</latexit> 2<latexit sha1_base64="jk/1fpohxujb3eq/tofnvjxofrw=">aaab6hicbvbns8naej3ur1q/qh69lbbbu0mqomeif48t2fpoq9lsj+3azsbsboqs+gu8efdeqz/jm//gbzudtj4yelw3w8y8ibfcg9f9dgpr6xubw8xt0s7u3v5b+fcorenumwyxwmsqe1cngktsgw4edhkfnaoepgtj25n/8irk81jem0mcfkshkoecuwolzq1frrhvdw6ysrycvcbho1/+6g1ilkyodrnu667njsbpqdkcczyweqnghlixhwlxukkj1h42p3rkzqwyiggsbeld5urviyxgwk+iwhzg1iz0sjct//o6qqmv/yzljduo2wjrmapiyjl7mgy4qmbexblkfle3ejaiijjjsynzelzll1dju1b1lqq15mwlfpphuyqtoivz8oak6nahdwgba4rneiu359f5cd6dj0vrwclnjuepnm8ffw+mug==</latexit> <latexit sha1_base64="ehc+d5cywammzlse/g19iyaznzq=">aaab+3icbvdlssnafl3xwesr1qwbybfclarwdfmo6mkfsax7gcauyxtsdp1mwsxelkw/4safim79exf+jzm0c209mjfdofdy7xw/zlqq2/42vlbx1jc2c1vf7z3dvx3zonswusiwaegiralri0ky5aslqgkkgwucqp+rjj++sv3oixgsrvxbtwlihwjiauaxulrqm6w7+rnrfm+zyus1xftm2a7ygaxl4uskddmaffplhuq4cqlxmcepe44dk2+khkkykvnrtssjer6jielpylfipdfnbp9zj1ozweek9opkyttfe1musjkjfd0zijwsi14q/uf1ehvcelpk40qrjuelgorzkrlsikwbfqqrnteeyuh1rryeiygw0nglitilx14m7wrfoatu72vlxnuerwgo4bhowyelamannkefgj7ggv7hzzgzl8a78tfvxthymup4a+pzb05bkgg=</latexit> Optimal DER Coordination Problem (ODCP) bulk grid tie line y p g 4 5 f p 1 f p 5 p d 5 <latexit sha1_base64="m6z2tylkluktjhb6esfgrt/vw+o=">aaab7hicbvbns8naej34wetx1aoxxsj4kklv9fjqg8ckpi20sww2m3bpzjfsboqs+hu8efdeqz/im//gbzudtj4yelw3w8y8movmg9f9dlzw19y3nktb5e2d3b39ysfhs8tmeeotyaxqhfhtzgt1dtocdljfcrjy2g5hn1o//usvzli8mhfkgwqpbiszwczkftq/fiz6lapbc2day8qrsbuknpuvr14kszzqyqjhwnc9nzvbjpvhhnnjuzdpmmiywgpatvtghoognx07qadwivasls1h0ez9pzhjrotxetrobjuhxvsm4n9enzpxdzazkwagcjjffgccgymmn6oikuomh1uciwl2vksgwgfibd5lg4k3+piyadvr3nmtfn9rbdwwcztgge7gddy4ggbcqrn8imdggv7hzrhoi/pufmxbv5xi5gj+wpn8axnhjng=</latexit> p g 1 <latexit sha1_base64="azmen4pyl58tdinfvraycjrhg04=">aaab7hicbvbns8naej3ur1q/qh69lbbbu0mqomechjxwsb/qxrlzttqlm03y3qgl9dd48aciv3+qn/+n2zyhbx0w8hhvhpl5qsk4nq777rtw1jc2t4rbpz3dvf2d8ufrs8epythksyhvj6aabzfynnwi7cqkarqibafjm5nffkkleswfzcrbp6jdyupoqlfsm+l7j8n+uejw3tnikvfyuoecjx75qzeiwrqhnexqrbuemxg/o8pwjnba6quae8rgdihdsywnupvz/ngpobpkgisxsiunmau/jziaat2jatszutpsy95m/m/rpia89jmuk9sgzitfysqiicnsczlgcpkre0sou9zestiiksqmzadkq/cwx14lrvrvu6jw7i8r9ds8jikcwcmcgwdxuic7aeatghb4hld4c6tz4rw7h4vwgppphmmfoj8/eduodw==</latexit> 4 p d 4 p d 1 f p 3 <latexit sha1_base64="wc7t1+cdvsgrk59wh6wvx/8cs7q=">aaab7hicbvbns8naej34wetx1aoxxsj4kkkr6lggb48vtftoy9lsj+3szsbsbors+hu8efdeqz/im//gbzudtj4yelw3w8y8mbvcg9f9dtbwnza3tgs7xd29/ypd0tfxuyezyuizrcsqhvkngkv0dtcc26lcgoccw+hozua3nlbpnsghm04xiola8ogzaqzkr73ay9orld2kowdzjv5oypcj0st9dfsjy2kuhgmqdcdzuxnmqdkcczwwu5ngllirhwdhuklj1mfkfuyunfult6je2zkgznxfexmaaz2oq9szuzpuy95m/m/rzca6dizcpplbyralokwqk5dz56tpftijxpzqpri9lbahvzqzm0/rhuatv7xkmtwkv6tu7y/l9ds8jgkcwhlcgadxuic7aiapddg8wyu8odj5cd6dj0xrmpppnmafoj8/ezuoea==</latexit> f p 4 <latexit sha1_base64="crb9/fk0anrkxjjlxq0fkmuzz4u=">aaab7hicbvbns8naej3ur1q/qh69lbbbu0lqqy8fpxisyguhjwwznbrln5uwuxfk6g/w4kerr/4gb/4bt20o2vpg4pheddpzgkrwbvz32ymsrw9sbhw3szu7e/sh5cojto5txbdfyhgrtka1ci6xzbgr2eku0igq+bcmr2f+wxmqzwn5byyj+hedsh5yro2vwmg//pj0yxw36s5bvomxkwrkapblx71bzniipwgcat313mt4gvwgm4htui/vmfa2pkpswipphnrp5sdoyzlvbismls1pyfz9pzhrsotjfnjoijqrxvzm4n9enzxhlz9xmaqgjvssclnbtexmn5mbv8immfhcmel2vsjgvffmbd4lg4k3/piqadeq3kw1dlevng7yoipwaqdwdh5cqgnuoqktymdhgv7hzzhoi/pufcxac04+cwx/4hz+ahshjnk=</latexit> p d 3 <latexit sha1_base64="irulathtdaytvdfxlh/nreiqkje=">aaab7hicbvbns8naej34wetx1aoxxsj4kkkr6lggb48vtftoy9lsnu3s3u3y3qgl9dd48aciv3+qn/+n2zyhbx0w8hhvhpl5ycqznq777aytb2xubzd2yrt7+wehlapjtk4yrahpep6obog15uxs3zddatdvfiuq0044vpn5nseqnevkg5mknbb4kfnmcdzw8tnb4zeavkpuzz0drrkvifuo0bpuvvprqjjbpseca93z3nqeovageu6n5x6mayrjga9pz1kjbdvbpj92is6teqe4ubakqxp190sohdytedpogc1il3sz8t+vl5n4osiztdndjvksijootijmn6oikuomn1iciwl2vkrgwgfibd5lg4k3/piqaddrxqnwv7+snm+loepwcmdwar5cqrpuoau+egdwdk/w5kjnxxl3phata04xcwj/4hz+aha7jny=</latexit> f p 2 p g 3 p g 2 <latexit sha1_base64="kycwbbhvutl/nawjalrgyir3k7g=">aaab7hicbvbns8naej3ur1q/qh69lbbbu0mqomechjxwmg2hjwwz3brln5uwoxfk6w/w4kerr/4gb/4bt20o2vpg4pheddpzwlqkg6777rtw1jc2t4rbpz3dvf2d8ufr0yszztxniux0o6sgs6g4jwilb6ea0ziuvbwobmz+64lrixl1gooubzedkbejrtfkftqrpq565ypbdecgq8tlsqvynhrlr24/yvnmftjjjel4borbhgoutpjpqzsznli2ogpesvtrmjtgmj92ss6s0idrom0pjhp198sexsam49b2xhshztmbif95nqyj62aivjohv2yxkmokwytmpid9otldobaemi3sryqnqaymbt4lg4k3/piqadaq3kw1dn9zqd/mcrthbe7hhdy4gjrcqqn8ycdggv7hzvhoi/pufcxac04+cwx/4hz+ahnbjng=</latexit> p d 2 <latexit sha1_base64="gmdhp9fl/e9c16ixar0usca2vme=">aaab7hicbvbns8naej2tx7v+vt16wsycp5juqy8fpxisynpcg8tms2mxbjzhdyou0n/gxymixv1b3vw3btsctpxbwoo9gwbmbang2jjonyqtrw9sbpw3kzu7e/sh1cojtk4yrzlhe5gobka0e1wyz3ajwddvjmsbyj1gfdpzo09maz7ibznjmr+toeqrp8ryyushjcdwuk05dwcovercgtsgqgtq/eqhcc1ijg0vroue66tgz4kynao2rfqzzvjcx2tiepzkejpt5/njp/jmkigoemvlgjxxf0/kjnz6ege2myzmpje9mfif18tmdo3nxkazyziufkwzwcbbs89xybwjrkwsivrxeyumi6iintafig3bxx55lbqbdfei3ri/rdvvizjkcaknca4uxeet7qafhldg8ayv8iykekhv6gprwklfzdh8afr8axu1jnu=</latexit> N =5 L =5 n =4 distribution system <latexit sha1_base64="5oxhhblagdtarg/dpbskmcuxc24=">aaacbxicbvdjsgnbeo1xjxeb9slooteinsikhvqy8ojfiwawselo6dqktxowumvematbs/yuxdwo4tv/8cd4n3awgyy+khi8v0vvptesqqpjffsli0vlk6uptft6xubwtr2zw9zhrdiuechdvxwzbikckkfacdviafndcrw3ezhyk3egtaidw+xf0pbzoxce4ayn1lqp6wj3ql2kzxyp4cyjmeqervd7ttvjzj0x6dzjtummsp80gh5dprsb9me9fflyhwc5zfrx8k6ejyqpffxcp12pnusmd1kbaoygzafdsmzf9omxuvruc5wpaoly/t2rmf/rnu+atp9hr896i/e/rxajd95irbdfcagflpjistgko0hosyjgkhugmk6euzxydlomowkubulizb48t8r5bo40m78xavytcvlkgbyre5ijz6ralkmrlagnj2rixsirnbcertfrfdk6ye1n9sgfwb8/hs+dwa==</latexit> Legend Synchronous generator Inverter-interfaced source Load Bus Determine the DER active power injection vector, p g, that minimizes total cost of operation while satisfying: C1. The power exchanged with the bulk system, y, tracks some pre-specified value, y C2. The active power injection from each DER does not exceed its corresponding capacity limits, i.e., p g p g p g C3. The power flow on each line does not exceed its maximum capacity, i.e., f f f Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 5 / 32

Input-Output System Model y can be written as a function of p g, p d, q d as follows: y = h(p g, p d, q d ) h captures the impacts from both the physical laws as well as the effect of any reactive power control scheme Assumption 1 H1. The rate of change in y w.r.t. p g is bounded for bounded changes in the DER active power injections H2. The total active power provided to the bulk power system will increase when more active power is injected in the power distribution system Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 6 / 32

ODCP Formulation The ODCP can be formulated as: minimize p g [p g,p g ] c(p g ) subject to h(p g, p d, q d ) = y, f M 1 (Cp g p d ) f p g p g, p g p d, q d y f M C DER active power injections DER upper and lower capacity limits Load active, reactive power demands Requested power to be exchanged with bulk grid Line flow limits Reduced node-to-edge incidence matrix Matrix mapping DER indices to buses Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 7 / 32

Data-Driven DER Coordination Framework Data defining the ODCP problem: Cost function, c( ) DER capacity limits, p g, p g Network topology and DER location, M, C Load active and reactive power demand, p d, q d Line flow limits, f Input-output model, h(,, ) Assumed unknown Real-time measurements available: DER active power injections, p g [k], k = 1, 2,... Active power exchanged with the bulk system, y[k], k = 1, 2,... Framework Building Blocks An input-output (IO) model estimator that uses available real-time measurement data A controller that uses uses the identified IO model to solve the ODCP Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 8 / 32

Two-Timescale Coordination Framework controller solves ODCP time estimator updates sensitivities time Estimation Process iterations in estimation process p d and q d remain approximately constant between two time instants; therefore, changes in y[k] that occur across time steps in the estimation process depend only on changes in p g [k]; thus, y[k] = h(p g [k], p d, q d ), k = 0, 1,... Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 9 / 32

Input-Output Model as a Linear Time-Varying System For notational simplicity, define u[k] = p g [k], u = p g, u = p g, and π = [(p d ), (q d ) ] ; then, the IO model can be written as: y[k] = h(u[k], π), k = 0, 1,..., For k > 1, the above equation can be transformed into the following equivalent linear time-varying model: y[k] = y[k 1] + φ[k] (u[k] u[k 1]) where φ[k] = [φ i [k]] = h u ũ[k], with ũ[k] = a k u[k] + (1 a k )u[k] φ[k] is referred to as the sensitivity vector at time step k The entries of φ[k] are bounded for all k by Assumption 1 Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 10 / 32

Estimator Estimation Step At time step k, the objective of the estimator is to obtain an estimate of φ[k], denoted by ˆφ[k], using available measurements of u and y Problem P1: subject to ˆφ[k] = arg min J e ( ˆφ) = 1 (y[k 1] ŷ[k 1])2 ˆφ Q=[b 1,b 1 2 ] n ŷ[k 1] = y[k 2] + ˆφ (u[k 1] u[k 2]) Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 11 / 32

Estimator Control Step The objective of the controller during the estimation process is to ensure that the output tracks the target [Different from the ODCP] Problem P2: subject to u[k] = arg min u U=[u,u] J c (u) = 1 2 (y ŷ[k]) 2 ŷ[k] = y[k 1] + ˆφ[k] (u u[k 1]) Note that ˆφ[k] is used to predict the value of y[k] for a given u Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 12 / 32

Estimation Process Workflow control step u[k 1] y[k 1] }{{ ˆφ {}}{ [k] u[k] y[k] } ˆφ[k + 1] estimation step At the beginning of iteration k, y[k 1] is used in Problem P1 to update the sensitivity vector estimate, ˆφ[k] The updated sensitivity vector estimate, ˆφ[k], is then used in Problem P2 to determine the control, u[k] Then, the DERs are instructed to change their active power injection set-points based on u[k] Problems P1 and P2 are not solved to completion for each k Instead, we iterate the projected gradient descent algorithm that would solve them for one step at each iteration k Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 13 / 32

Simulation Setup 33 32 31 27 0 30 29 48 28 25 26 23 24 21 22 18 19 20 11 14 10 2 9 1 7 8 12 3 4 5 6 120 47 13 44 45 115 34 42 15 40 37 16 49 117 59 50 35 36 52 17 43 46 41 58 51 38 65 66 57 53 54 96 95 39 116 55 94 64 63 93 62 60 56 92 91 90 101 89 61 97 88 67 87 72 76 111 121 110 112 113 114 109 108 107 106 104 105 103 102 118 119 98 86 68 73 99 69 74 77 78 DER 80 81 82 122 100 70 75 79 71 84 85 83 Figure 1: The IEEE 123-bus distribution test feeder. Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 14 / 32

Tracking Performance During Estimation tracking error (kw) 0 200 400 600-3000 kw -2900 kw -2800 kw -2700 kw -2600 kw -2500 kw 0.0 2.0 4.0 6.0 8.0 10.0 time (s) Figure 2: Tracking error for β k = 0.02 under various tracking targets. 0 tracking error (kw) 25 50 75 100 0.005 0.01 0.02 0.03 0.04 0.05 0.0 2.0 4.0 6.0 8.0 10.0 time (s) Figure 3: Tracking error for y = 3000 kw and various constant control step sizes. Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 15 / 32

Estimation Accuracy Mean absolute error (MAE) of estimation errors: MAE = n i=1 where n is the number of DERs ˆφi [k] φ i [k], n MAE (p.u./p.u.) 0.06 0.04 0.02 0.005 0.01 0.02 0.03 0.04 0.05 0.00 0.0 2.0 4.0 6.0 8.0 10.0 time (s) Figure 4: Estimation error under various control step sizes. Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 16 / 32

<latexit sha1_base64="o95+zhquihfz3qgd3xj3pvnivia=">aaae6nicjvrlb9naehabacw8wjhygagdsfsrnagtpsbvlaohhopih1kcrpv1jtl2vta765ti9u/gwggeupklupfvmhxstgklyiu7szpffpn0uong2nje77n5uvngzvsltyt37t67/2bx6egejlnf2s6nrawoukqzwsxbndwidpaorqkuypvd4y1r3x8ypxksm2aushze+pl3ocugvz2lkhn0wz/lzbb0yvtptlsceoiicfgpoqsyzcc0jiiiwywgspgrnoozosgzv9bilzoyazjism8vu7puejvnvfhi1q0oeduhbqntrmagfyrc48izwcvepbaladjhzgdbzs29au7pxokuxgp+ubaeyoxudbh121e8imoemocmhrewpm+gkm+kqdndzkfpcwv/wggsw/9woljo2mzsqdhk+balaybdv7maazobe6jwa5ielo3tmu1nbu1fpo7fkmikxhbvir/gernjx1tup3+beu1oas6oq+agbxlmhc2pmpnjj1otxtif9mfg4stfptobbfzcpj82wjbreyvbdznnmq6i7ommpvwibyjvkhm1jrxal7dwx11bmja2occ/5m001let4af5hvg6jyylpuy0x6u6d3dgpe38nb9jkwgwhdcbwmq9mj8om77uz0otrv4xarawamc3xckld9z4smdx2ua1b64k/krydiznp7p4kwhjmka49fqqrvu+l5h2rpth1bigqwyjoce4ki0ujymybmffp5pdftuh4gdxkqyk7uwpjeraj6iuiinibvqyzsqvs7vs03vvzrhmusmkhqfqpqjmdpa7h5djmhsxqofqxtfx2ykfg4x/dryj/uwsrwp79zrfqnxf15c330zasea8dp46lxzfwxc2nxfojrpr0fk/9ln0tfstlmpfyt/lp8bq+bmjzynn5pr//ghbvogb</latexit> Background Voltage regulation transformers also referred to as Load Tap Changers (LTCs) are widely utilized in power distribution systems to regulate voltage magnitudes along a feeder Model for a load tap changer on a line connecting buses i and j: i + V i tl :1 + V j r l +ix l = V i t 2 l j + V j V i V j 0 V j t` r` x` Primary side voltage magnitude Secondary side voltage magnitude Line receiving end voltage magnitude Tap ratio Transformer + line resistance Transformer + line reactance The tap ratio, t l, typically takes 33 discrete values ranging from 0.9 to 1.1, by an increment of 5/8%, i.e., t l T = {0.9, 0.90625,, 1.09375, 1.1} Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 17 / 32

Problem Motivation Current LTC control schemes are myopic: Based on local voltage measurements Do not account for future uncertainty effects on current control actions These schemes are no longer suitable because of increased variability and uncertainty in uncontrolled power injections arising from: Residential PV installations Electric vehicles In addition, the use of controlled DERs for providing frequency regulation to the bulk grid has an impact on voltage regulation Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 18 / 32

Volt/VAR Control Architecture Fast Time-Scale Control time t 0 t 1 t 2 Slow Time-Scale Control Slow Time-Scale: LTCs are periodically dispatched so as to reduce mechanical wear Fast Time-Scale Control: Power-electronic-interfaced DERs with reactive power provision capability Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 19 / 32

<latexit sha1_base64="ehc+d5cywammzlse/g19iyaznzq=">aaab+3icbvdlssnafl3xwesr1qwbybfclarwdfmo6mkfsax7gcauyxtsdp1mwsxelkw/4safim79exf+jzm0c209mjfdofdy7xw/zlqq2/42vlbx1jc2c1vf7z3dvx3zonswusiwaegiralri0ky5aslqgkkgwucqp+rjj++sv3oixgsrvxbtwlihwjiauaxulrqm6w7+rnrfm+zyus1xftm2a7ygaxl4uskddmaffplhuq4cqlxmcepe44dk2+khkkykvnrtssjer6jielpylfipdfnbp9zj1ozweek9opkyttfe1musjkjfd0zijwsi14q/uf1ehvcelpk40qrjuelgorzkrlsikwbfqqrnteeyuh1rryeiygw0nglitilx14m7wrfoatu72vlxnuerwgo4bhowyelamannkefgj7ggv7hzzgzl8a78tfvxthymup4a+pzb05bkgg=</latexit> Optimal LTC Dispatch Problem tie line V 0 V 1 V 2 V5 V4 V 3 N =5 L =5 n =4 Legend Synchronous generator Inverter-interfaced source Load Bus bulk grid distribution system <latexit sha1_base64="5oxhhblagdtarg/dpbskmcuxc24=">aaacbxicbvdjsgnbeo1xjxeb9slooteinsikhvqy8ojfiwawselo6dqktxowumvematbs/yuxdwo4tv/8cd4n3awgyy+khi8v0vvptesqqpjffsli0vlk6uptft6xubwtr2zw9zhrdiuechdvxwzbikckkfacdviafndcrw3ezhyk3egtaidw+xf0pbzoxce4ayn1lqp6wj3ql2kzxyp4cyjmeqervd7ttvjzj0x6dzjtummsp80gh5dprsb9me9fflyhwc5zfrx8k6ejyqpffxcp12pnusmd1kbaoygzafdsmzf9omxuvruc5wpaoly/t2rmf/rnu+atp9hr896i/e/rxajd95irbdfcagflpjistgko0hosyjgkhugmk6euzxydlomowkubulizb48t8r5bo40m78xavytcvlkgbyre5ijz6ralkmrlagnj2rixsirnbcertfrfdk6ye1n9sgfwb8/hs+dwa==</latexit> Load tap changer (LTC) Objective Find a policy for determining the LTC tap positions based on measurements of current tap ratios bus voltage magnitudes so as to minimize bus voltages deviations from some reference value as power injections change as time evolves Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 20 / 32

Power Distribution System Model The relation between square voltage magnitudes and active/reactive power injections and LTC tap ratios at instant k can be written as V [k] = g(p[k], q[k], t[k]) V [k] p[k], q[k] t[k] Vector of voltage magnitudes at instant k Vector of active, reactive, power injections at instant k Vector of tap ratios at instant k Network topology and transmission line parameters are encapsulated into g Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 21 / 32

Assumptions T1. Changes in active and reactive power injections at instant k, p[k] := p[k + 1] p[k] and q[k] := q[k + 1] q[k], are random T2. Active and reactive power injections k, p[k] and q[k], are not measured and their joint probability distribution is unknown T3. Network topology is known, i.e., M is known T4. Line parameters are unknown, i.e., r and x are unknown Because of Assumption T1 is natural to formulate the problem as a Markov Decision Process (MDP) Because of Assumptions T2 T4, we will have to resort to reinforcement learning (RL) techniques to solve this MDP Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 22 / 32

LTC Coordination Problem as an MDP State space, S: the state, s, is composed of tap ratio and squared voltage magnitude vector, i.e., s = (t, v), t T Lt, v R N ; thus, S T Lt R N Action space, A: the action, a, is the change in LTC tap ratio between two consecutive time instants, i.e., a = t, t T Lt =: A, where T = {0, ±0.00625,, ±0.19375, ±0.2} is the set set of feasible tap ratio changes Reward function, R: the reward when the system transitions from state s = (t, v) into state s = (t, v ) after taking action a = t is R(s, a, s ) = 1 N v v, i.e., we are penalizing voltage deviation from some reference value v Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 23 / 32

LTC Coordination Problem as an MDP State transitions: governed by random changes in active and reactive power injection vectors, p := p p and q := q q: Objective s = h(s, a, p, q) Network topology and transmission line paramters are encapsulated into h Probability of transitioning from s to s under action a, P(s s, a), could be computed if h and the joint pdf of p and q were know Find a policy π : (t, v) t, t T Lt, v R N, t T Lt 1 N k=0 γ k E [ v[k + 1] v ] t[0] = t 0, v[0] = v 0 is maximized (equivalent to minimizing voltage deviations) so that Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 24 / 32

Solving the LTC Coordination Problem Let R(s, a) denote the expected reward for the pair (s, a) The MDP is solved when we find Q (s, a), s S, a A, and π (s), s S, satisfying Q (s, a) = R(s, a) + γ s S P(s s, a) max a A Q (s, a ) π (s) = arg max Q (s, a) a Two issues in our setting: I1. We do not know P(, ) I2. Even if we knew P(, ), we could not solve for Q (s, a) efficiently because of the curse of dimensionality in the state and action spaces To circumvent Issue I1, we apply a model-free RL algorithm that utilizes transition samples obtained via a virtual transition generator To circumvent Issue I2, we use function approximation and a learning scheme for sequential estimation of the action-value function Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 25 / 32

LTC Coordination Framework Building Blocks action Environment Power Distribution System state reward Learning Agent Exploratory Actor History action Virtual Transition Generator Action-Value Function Estimator (Critic) Acting Agent Greedy Actor action-value function estimate Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 26 / 32

Action-Value Function Value Estimator Let ˆQ(, ) denote an approximation of the optimal action-value function Q(, ) Methods for obtaining ˆQ(, ) include: Parametric functions Neural networks Here we consider a linear parametrization of ˆQ(, ): ˆQ(s, a) = w φ(s, a), where w R f is the parameter vector and φ : S A R f is some basis function Let D = {(s, a, r, s ) : s, s S, a A} denote a set (batch) of transition samples obtained via observation or simulation We use least-square policy iteration (LSPI) algorithm [Lagoudakis and Parr, 2003] to find w that best fits the transition samples in D Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 27 / 32

Challenges The LSPI algorithm requires adequate transition samples that spread over S A This is challenging in power systems since the system operational reliability might be jeopardized when exploring randomly We address this by developing by generating samples via a virtual transition generator that leverages historical system operational data The LSPI also suffers from the curse of dimensionality when the action space is large the case in the LTC coordination problem We address this by using a sequential scheme that breaks the learning problem into smaller problems and uses the LSPI algorithm on those Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 28 / 32

IEEE 123-bus Test Feeder 33 32 31 27 0 30 29 48 28 25 26 23 24 22 21 18 19 20 11 14 10 2 9 1 7 8 12 3 4 5 6 120 51 50 49 47 46 44 45 43 42 41 40 115 35 65 66 39 38 36 37 57 58 59 53 54 52 117 13 96 34 95 15 17 16 116 55 94 64 63 93 62 60 56 92 91 121 108 105 118 90 101 89 110 109 107 106 119 61 97 88 67 87 102 72 76 111 98 86 103 68 73 99 69 74 77 78 112 113 114 80 81 82 104 122 100 70 75 79 84 83 71 85 Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 29 / 32

IEEE 123-bus Test Feeder Results tap position 4 2 0-2 -4-6 -8 1 2 3 4 Batch RL 0 2 4 6 8 10 12 14 16 18 20 22 24 time (hour) reward ( 10 3 ) 5 10 batch RL exhaustive search conventional scheme 0 2 4 6 8 10 12 14 16 18 20 22 24 time (hour) Figure 6: Immediate rewards. 4 2 Exhaustive search 0 tap position 0-2 -4-6 -8 1 2 3 4 0 2 4 6 8 10 12 14 16 18 20 22 24 time (hour) Figure 5: Tap positions. reward ( 10 3 ) 5 10 batch RL exhaustive search conventional scheme 1 2 3 4 5 6 7 8 time (day) Figure 7: Rewards over 7 days. Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 30 / 32

Conclusions We developed a data-driven coordination framework for coordinating assets in distribution systems to provide ancillary services: The proposed framework: It assumes no prior information on the distribution system model, except knowledge of network topology It mainly relies on measurements It is adaptive and robust to changes in operating conditions Refer to the following papers for more details: 1. H. Xu, A. Domínguez-García, and P. Sauer, Data-driven Coordination of Distributed Energy Resources for Active Power Provision, IEEE Transactions on Power System, 2019. 2. H. Xu, A. Domínguez-García, and P. W. Sauer, Optimal tap setting of voltage regulation transformers using batch reinforcement learning, IEEE Transactions on Power System, 2019. Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 31 / 32

Questions? Alejandro D. Domínguez-García e-mail: aledan@illinois.edu url: https://aledan.ece.illinois.edu Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 32 / 32

References I [Lagoudakis and Parr, 2003] Lagoudakis, M. G. and Parr, R. (2003). Least-squares policy iteration. Journal of Machine Learning Research, 4:1107 1149.

A Primer on Markov Decision Processes A Markov Decision Process (MDP) is defined as a 5-tuple (S, A, P, R, γ) where S is a finite set of states A is a finite set of actions P is a Markovian transition model R : S A S R is a reward function γ [0, 1) is a discount factor Let s[k] and a[k] denote random variables (r.v. s) respectively describing the value the state and action take at time instant k Let r[k] denote a r.v. describing the reward received after taking action a[k] in state s[k] and transitioning to state s[k + 1]; then r[k] = R ( s[k], a[k], s[k + 1] ) A deterministic policy π is a mapping from S to A, i.e., a = π(s), s S, a A Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 33 / 32

A Primer on Markov Decision Processes Objective Given some initial state, s 0, we want to find a deterministic policy, π, that maximizes the expected value of the cumulative discounted reward, i.e., π = arg max π k=0 γ k E [r[k] ] s[0] = s 0 Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 34 / 32

A Primer on Markov Decision Processes The action-value function under policy π is defined as Q π (s, a) = k=0 γ k E [r[k] ] s[k] = s, a[k] = a; π, s S, a A The optimal action-value function, Q (s, a), s S, a A is the maximum action-value function over all policies, i.e., Q (s, a) = max π Qπ (s, a) Q (s, a), s S, a A, satisfies the following Bellman equation: Q (s, a) = E [r s, a] + γ s S P(s s, a) max a A Q (s, a ) The optimal policy, π (s), s S, is obtained as follows: π (s) = arg max Q (s, a) a The MDP is solved if we find Q (, ) and the corresponding π ( ) Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 35 / 32

LSPI Algorithm Iteration Let w i denote the estimate of w at the beginning of iteration i For each transition sample (s, a, r, s ) D, compute a = arg max wi φ(s, α) α A Update the estimate of w as follows: w i+1 = B 1 b where B = b = (s,a,r,s ) D (s,a,r,s ) D ( ) φ(s, a) φ(s, a) γφ(s, a ) } {{ } rank-1 matrix φ(s, a)r Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 36 / 32

Timeline Fast Time-Scale Control time t 0 t 1 t 2 Slow Time-Scale Control Policy updated every K T units of time, e.g., 2 hours Updated policy used for tap setting for K time instants Domínguez-García (ECE ILLINOIS) Data-driven Coordination aledan@illinois.edu 37 / 32