Network Automation czyli jak można automatyzować w sieciach. Adam Obszyński CCIE #8557, CISSP aobszynski@infoblox.com

Transkrypt

1 Network Automation czyli jak można automatyzować w sieciach. Adam Obszyński CCIE #8557, CISSP aobszynski@infoblox.com

2 Typowy interfejs systemu NMS? Windows 95 2

3 NMS + zaangażowany administrator 3

4 Dlaczego tak się dzieje? 4

5 Bo tak :-) Source 5

6 To przynajmniej działa :-) Windows 95 6

7 Ale, rozmiar ma znaczenie! 7

8 Ale, rozmiar sieci ma znaczenie! 8

9 I co teraz?! 3*Z czyli ZZZ demoty.pl 9

10 I co teraz?! Zaplanuj Zautomatyzuj Zapomnij demoty.pl 10

11 Przypadki z życia wzięte. Network Changes Manual CLI, Perl scripts, and basic config back-ups Time intensive and requires senior engineer Network Discovery Spreadsheets, periodic scans, multiple tools Out-of-date and incomplete data Compliance/Standardization Periodic audit focus with special task force Adds security risk for policy violations Access Provisioning Massive spike in requests and delivery expectations Longer SLAs - manual processes and needed expertise 11

12 Przypadki z życia wzięte. Network Changes Manual CLI, Perl scripts, and basic config back-ups Time intensive and requires senior engineer 12

13 Ogromne centrum wystawiennicze w słonecznej Hiszpanii. Czasami do portów zmienia konfig Manualny proces: inżynier -> tester -> help desk -> dział sieci -> LAN Dużo pomyłek Ogromna pracochłonność Brak kontroli nad schematem konfiguracji demoty.pl 13

14 Ogromne centrum wystawiennicze w słonecznej Hiszpanii. Czasami do portów zmienia konfig Zautomatyzowany proces: inżynier -> tester -> portal+api / job -> LAN Dużo mniej pomyłek Pełen self-service Dodatkowo policy & standard control demoty.pl 14

15 Ogromne centrum wystawiennicze. Z lotu ptaka. demoty.pl 15

16 Przypadki z życia wzięte. Network Discovery Spreadsheets, periodic scans, multiple tools Out-of-date and incomplete data 16

17 Dokumentacja sieci w koncernie energetycznym (FR). Problem aktualności danych (sprzęt, soft) Dokumentacja sieci wiecznie w planach Planowanie powolne bo zawsze zaczyna się od analizy tego co jest w sieci ;-) Zmiany nie zawsze były optymalne Ręczne procesy demoty.pl 17

18 Dokumentacja sieci w koncernie energetycznym (FR) po staremu #1 Sprawdź dokumentację i kiedy była aktualizowana? #2 Czy rzeczywistość to czy fikcja? #3 Zróbmy spotkanie zespołu, może ktoś coś zmieniał? #4 Wykonajmy zmianę może będzie dobrze. demoty.pl 18

19 Dokumentacja sieci w koncernie energetycznym (FR) - dzisiaj #1 Wykorzystanie aktualnych danych (Inwentarz, Topologia, Konfigi) #2 Narzędzie do analizy i dyskusji na spotkaniach z aktualnym widokiem #3 Wprowadzenie zadań/skryptów i wykonanie ich w sieci z ew. rollback. #4 Eksport danych + Eksport Topologii do Visio demoty.pl 19

20 Dokumentacja sieci w koncernie energetycznym (FR) - dzisiaj 20

24 Przypadki z życia wzięte. Compliance/Standardization Periodic audit focus with special task force Adds security risk for policy violations 24

25 Duży BANK (USA): standaryzacja + zgodność urządzeń sieciowych :-) Prawie milion interfejsów sieciowych. #1 Robótki ręczne przestały się skalować #2 Co raz nowsze wymagania prawne #3 Czas (skrypt A jeszcze trwa, skrypt B już by chciał wystartować ) #4 Brak centralnego światowego repozytorium konfiguracji #5 Praca interaktywna (CLI^2) demoty.pl 25

26 Duży BANK (USA): standaryzacja + zgodność urządzeń sieciowych :-) Prawie milion interfejsów sieciowych. #1 Globalne repozytorium konfigów #2 Backup, SLA itd.. #3 API do integracji ( tylko jedno ;-) ) #4 Globalne sprawdzanie policy + raportowanie #5 Provision from baseline #6 Praca dużo mniej terminalowa (less CLI) demoty.pl 26

27 Duży BANK (USA): Przykłady Rule -> Policy -> Deploy == ZZZ demoty.pl 27

28 Duży BANK (USA): Przykłady Rule -> Policy -> Deploy == ZZZ demoty.pl 28

29 Duży BANK (USA): Przykłady demoty.pl 29

30 Duży BANK (USA): Przykłady Wersja CLI use NetMRI::API::Client; our $_client = new NetMRI::API::Client( UserName Password URL => "$http_username, => "$http_password, => "$api_url ); our $_dis = $_client->get_broker("dissession"); our $_cli = $_client- >get_broker("cliconnection"); our $_issue = $_client->get_broker("issueadhoc"); our $_session_id = 0; my $dev1 = $device_id; my $cli_command_s = "show version"; my $cli_command_d = "show version"; my $cli-match; my $d_if; my $s_if; my $d_ifname; my $s_ifname; my $d_device; my $s_mtu; my $d_mtu; open_session(); open_connection($device_id); my $broker = $_client ->get_broker("device"); my $bint = $_client->get_broker("interface"); $dev1 = $broker->find($dev1); my $output1 ; print "\n\ncurrent neighbors of $dev1->{devicename} $dev1- >{DeviceIPDotted} ($dev1->{deviceid}):\n ; = sort { $a->{ifindex} <=> $b->{ifindex} $a->{neighbordeviceid} <=> $b->{neighbordeviceid} $a->{neighborifindex} <=> $b- >{NeighborIfIndex} } $dev1->get_neighbors(); foreach my $n (@ns) { my $nd; eval {$nd = $broker->find_by_id($n->{neighbordeviceid});}; if ($@ =~ /^H404/) { print "Could not find device $n- >{NeighborDeviceID}\n";next;}; printf "\non %7s %6d %15s %16s \%s\n", ($n->{ifindex}? ("if". $n->{ifindex}) : "unknown"), $nd->{deviceid}, $nd->{devicename}, $nd->{deviceipdotted}, ($n->{neighborifindex}? ("if". $n- >{NeighborIfIndex}) : "unknown"); = $bint->find_by_id($n->{interfaceid}); die "\nsource Interface not found.\n\n" if!@sif; foreach my $s (@sif) {printf "Source IF - %10s %s\n",$s- >{ifname}, $s->{ifdescr}; $s_if = $s->{ifname};$s_ifname = $s->{ifdescr};} = $bint->find_by_id($n->{neighborinterfaceid}); die "\nsource Interface not found.\n\n" if!@dif; foreach my $d (@dif) {printf "Dest IF - %10s %s\n",$d- >{ifname}, $d->{ifdescr}; $d_if = $d->{ifname};$d_ifname = $d->{ifdescr}; $d_device = $d->{deviceid};} if ($dev1->{devicevendor} eq "Cisco") {$cli_command_s = "show interface $s_if include MTU ; $cli_match = "MTU ([0-9]+) bytes.*";} else {$cli_command_s = "show interface $s_if match MTU";$cli_match = "Protocol inet, MTU: ([0-9]+).*";} print "command to push is $cli_command_s\n"; $output1 = send_command($device_id, $cli_command_s); print "\tsource Device/interface $dev1->{devicename}/$s_if output: $output1\n"; $d_mtu = 0;$s_mtu = 0; if ($output1 =~ m/$cli_match/){$s_mtu = $1;}; print "\nok, now finding far end device for $d_device\n ; my $destdev = $broker->find_by_id($d_device); print "\tgot Device $destdev->{devicename}\n ; print "\tnetwork device Indication is $destdev->{networkdeviceind}\n \tmanaged is $destdev->{devicemanagedind}\n\tccs Collection is $destdev->{deviceccscollection}\n\tconfig Polling is $destdev- >{DeviceConfigPolling}\n ; if (($destdev->{deviceccscollection} eq "on") and ($destdev- >{DeviceManagedInd} eq "true") and ($destdev->{deviceccscollection} eq "on") and ($destdev->{networkdeviceind} eq "true") ) { if ($destdev->{devicevendor} eq "Cisco") {$cli_command_d = "show interface $d_if include MTU ; $cli_match = "MTU ([0-9]+) bytes.*";} else {$cli_command_d = "show interface $d_if match MTU"; $cli_match = "Protocol inet, MTU: ([0-9]+).*";} print "\tcommand to push is $cli_command_d\n ; open_connection($d_device); $output1 = send_command($d_device, $cli_command_d); print "\tdest Device/interface $nd->{devicename}/$d_if output: $output1\n ; close_connection($d_device); if ($output1 =~ m/$cli_match/) {$d_mtu = $1;} if ($d_mtu!= $s_mtu) { print "\tmtus do not match\n ; my $issue_id = generate_issue( "MTUmismatch, "Warning", { "IP Address" => $dev1->{deviceipdotted}, "Host" => $dev1->{devicename}, "Interface" => $s_ifname, "MTU" => $s_mtu, => $nd->{devicename}, "Remote Device" demoty.pl => $d_ifname, => $d_mtu }); "Remote Interface" "Remote MTU" } else {print "\tmtus match $s_mtu - $d_mtu\n";}}} sub open_session {our $_dis_response = $_dis->open(job_id=> $job_id); $_session_id = $_dis_response->{dis_session}->{sessionid}; END {close_session();}} sub close_session {our $_dis_response = $_dis->close(id=> $_session_id,);} sub open_connection { my $devid = shift; print "++++ Opening session to device $devid\n"; our $_cli_response = $_cli->open(id => $_session_id, device_id => $devid); print "DEBUG: _cli_reponse: $_cli_response\n"; END {close_connection($devid);}} sub close_connection { my $devid = shift; our $_cli_response = $_cli->close(id => $_session_id, device_id => $devid);} sub send_command { my $devid = shift; my ($command, $debug) if($debug eq "") {$debug = 0;} print "DEBUG: Device ID is: $devid\n"; $_cli_response = $_cli->send_command(id => $_session_id, device_id => $devid,command => $command, debug => $debug); return($_cli_response->{command_response});} sub generate_issue { my ($issue_type_id, $severity, $params) my %baseparams = (DeviceID => $device_id, BatchID => $batch_id, IssueTypeID => $issue_type_id, Severity => $severity); my %allparams = (%baseparams, %{$params}); our $_issue_response = $_issue- >generate_issue(%allparams);return($_issue_response->{issueid});} 30

31 Duży BANK (USA): Przykłady Wersja bez CLI Info z NetMRI # BEGIN-SCRIPT-BLOCK # Script-Filter: true # Script-Login: false # END-SCRIPT-BLOCK use NetMRI_Easy; my $easy = new NetMRI_Easy; foreach my $IF (@IFs){ foreach my $NB (@NBs){ if (defined $NB->NeighborInterfaceID) { my $NIF=$easy->broker->interface->show(InterfaceID=>$NB->NeighborInterfaceID)->{interface}; if ($IF->ifMtu==$NIF->ifMtu){ print "Interface ".$IF->ifDescrRaw." has same MTU on neighbor\n"; }else{ print "Interface ".$IF->ifDescrRaw." (MTU:".$IF->ifMtu.") "." has different MTU (".$NIF->ifMtu.") on neighbor\n"; my $NDevice=$easy->broker->device->show(DeviceID=>$NIF->DeviceID)->{device}; print "Neighbor Device:".$NDevice->DeviceName." Interface:".$NIF->ifDescrRaw."\n\n"; };};};}; 31

32 Przypadki z życia wzięte. Access Provisioning Massive spike in requests and delivery expectations Longer SLAs - manual processes and needed expertise 32

33 Bank (NL) + zespoły Managed Services Inny zespół od rdzenia/dystrybucji a inny od dostępu Ciągłę błędy na styku dystrybucja <-> dostęp (trunking, MTU). Duża ilość niespodzianek w głębokim ukryciu Ogromna pracochłonność Brak proaktywności demoty.pl 33

34 Bank (NL) + zespoły Managed Services Kilka procesów (job) Automatyka zgodności VLAN, Trunk, MTU (wykorzystanie tabel/list). Błyskawiczne wykrywanie błędów i pomyłek Stała weryfikacja poprawności. Wykrywanie błędów przed użytkownikami!!! demoty.pl 34

35 Bank (NL) + zespoły Managed Services Sprawdź VLAN # BEGIN-SCRIPT-BLOCK # Script-Filter: true # Script-Login: false # END-SCRIPT-BLOCK use NetMRI_Easy; my $easy = new NetMRI_Easy; foreach my $VLAN (@VLANs){ $VLN{$VLAN->VlanIndex}{Index}=$VLAN->VlanIndex; $VLN{$VLAN->VlanIndex}{Name}=$VLAN->VlanName; }; foreach my $IF (@IFs){ IntefaceID=>$IF->InterfaceID); foreach my $VlanId (@IFVlans){ if ($VLN{$VlanId->VlanID}{Index} and $VlanId->InterfaceID==$IF->InterfaceID) {print "Interface: ", $IF->ifDescrRaw," VLAN:",$VLN{$VlanId->VlanID}{Index}," ",$VLN{$VlanId->VlanID}{Name}, "\n";}; }; 35

36 Przypadki z życia wzięte. Network Changes Manual CLI, Perl scripts, and basic config back-ups Time intensive and requires senior engineer Network Discovery Spreadsheets, periodic scans, multiple tools Out-of-date and incomplete data Compliance/Standardization Periodic audit focus with special task force Adds security risk for policy violations Access Provisioning Massive spike in requests and delivery expectations Longer SLAs - manual processes and needed expertise 36

37 Large Federal Agency NetMRI serves as a constant monitor, and found specific issues in the first few hours of deployment for the agency, such as: Configuration errors before going live Over-temperature conditions Redundant power-supply disconnects Redundant link outages Unstable or marginal WAN links and VPN connections Spanning tree instability Device crashes in remote offices 37

38 Infoblox Network Automation Automated Network Discovery Change & Configuration Management Discover Automate Compliance & Policy Standardization Maintain Control Firewall ACL & Rule Automation 38

39 Large broadband ISP Challenge New naming convention Change ports Manual 9000 manhours / several weeks NetMRI By creating a series of scripts in the NetMRI GUI, the network engineering team was able to automate the changes to interface names on 60,000 switch ports. Script generation was extremely simple, requiring no programming skills Solved also similar issue: Password rotation 39

40 Example 1 Change: All switches Set snmp values Commands on Cisco: config t snmp- server community infoblox RO snmp- server community netmri RW end write mem 40

41 The Manual Way Script that deals with Login to switches Apply the commands Build in logging and error handling Maintain a list of switches to run it on Verify manually Easy to make errors Easy to miss errors Time consuming Expert user 41

42 With Infoblox Network Automation Product does the difficult bit Automation Logic script on NetMRI: Script-Filter: $vendor eq "Cisco" and $sysdescr like /IOS/ Action: Config SNMP Action-Commands: config t 42 snmp-server community infoblox RO snmp-server community netmri RW end wr 2013 Infoblox mem Inc. All Rights Reserved.

43 Example 2: Like 1 But 2 Different Vendors Without Infoblox Network Automation: Effort doubles With Infoblox Network Automation: Focus on the change itself 43

44 Example 3 - Not only adding the snmp values you want - But also removing the others - Challenge - You first need to see what is configured - Automation Logic parts - Filter -> Cisco - Command to get the snmp config - Parse the output of that command and remove 44

45 Create your own DHCP & LAN Case #1 DHCP network/range == LAN helper address? Case #2 LAN ip helper address == DHCP networks + range? 45

46 Closing the Security Lifecycle Gap InfoSec Team Set department policy Know all connected L2/L3 devices & end hosts Automate remediation of non-compliant devices Simplify compliance audits Ensure security policies being followed Reduce risk of security vulnerabilities Integrate with SIEMs and other 3 rd parties Secure and enforce access to network infrastructure Network Ops Team Deploy & implement Enforce & monitor Security Ops Team

47 Discovery & Change Monitoring Network Ops Team Infoblox Network Automation Send for approval Alert on violations 1 2 Discover, fingerprint, and identify all switches, routers, firewalls, etc., from 50 vendors Backup the configurations for all L2, L3 devices 3 Automatically check if devices have been changed 4 Approve change for auditing or remediate directly 5 Deploy policies to continuously monitor network for compliance

48 Visibility & Compliance Auditing Security Ops Team Send to SIEM, Network Monitoring Infoblox Network Automation Leverage network topology maps to monitor for unmanaged devices Set user roles to track who changed what and when Identify network hardware security gaps (EOL, PSIRT, etc.) 4 Turn unused switch ports off to reduce security profile 5 6 Track end hosts to determine how network being accessed Generate reports on assets, inventories to reduce risk

49 Bridging the Gap Reduced time to audit Simplified & customizable security policies Out of the box compliance reports Set department policy InfoSec Team Improve agility with automated network change provisioning Inventory all network infrastructure Role based access & user auditing Deploy & implement Communication and Network reporting Automation Reduce risk profile Continuous real-time monitoring SIEM and 3 rd party integrations Enforce & monitor Network Ops Team Single version of truth Security Ops Team

50 To działa najlepiej :-) Windows 95 50

51 FIN