Bacula. Czy warto? Tomasz Górny tgorn(at)onet.pl Zimowisko 2010

Transkrypt

1 Bacula Czy warto? Tomasz Górny tgorn(at)onet.pl Zimowisko 2010

2 Bacula - The Open Source Network Backup Solution It comes by night and sucks the vital essence from your computers.

3

4 Rozproszona architektura 4

5 Zalety Open Source Brak pojedynczego punktu awarii Różne funkcje mogą być rozmieszczone na osobnych komputerach Ciągły rozwój

6 Wady Architektura rozproszona Konfiguracja Brak dobrego interfejsu graficznego (webacula?)

7 Przykładowa konfiguracja

8

9 Instalacja yum install bacula-dir ======== Matched: bacula ======= bacula-client.i386 : Bacula backup client bacula-common.i386 : Common Bacula utilities bacula-console.i386 : Bacula management console bacula-console-gnome.i386 : Bacula console for the Gnome desktop environment bacula-console-wxwidgets.i386 : Bacula console using the wx widgets toolkit bacula-director-common.i386 : Common Bacula Director files bacula-director-mysql.i386 : Bacula Director with MySQL database support bacula-director-postgresql.i386 : Bacula Director with PostgresSQL database support bacula-director-sqlite.i386 : Bacula Director with sqlite database support bacula-docs.i386 : Bacula documentation bacula-storage-common.i386 : Common Bacula storage daemon files bacula-storage-mysql.i386 : MySQL Bacula storage daemon files bacula-storage-postgresql.i386 : Common Bacula storage daemon files bacula-storage-sqlite.i386 : SQLite Bacula storage daemon files bacula-traymonitor.i386 : Bacula monitor for the Gnome and KDE system tray webacula.noarch : Web interface of a Bacula backup system

10 Instalacja ==================================================================================== Package Arch Version Repository Size ==================================================================================== Installing: bacula-director-postgresql i el5 epel 832 k Installing for dependencies: bacula-common i el5 epel 354 k bacula-director-common i el5 epel 169 k fedora-usermgmt noarch el5 epel 6.6 k fedora-usermgmt-core noarch el5 epel 7.6 k fedora-usermgmt-default-fedora-setup noarch el5 epel 6.6 k fedora-usermgmt-shadow-utils noarch el5 epel 7.4 k postgresql i el5_4.1 updates 2.8 M postgresql-server i el5_4.1 updates 3.8 M Transaction Summary ==================================================================================== Install 9 Package(s) Update 0 Package(s) Remove 0 Package(s) Total download size: 8.0 M Is this ok [y/n]:

11 Dostęp do katalogu Bacula pozwala na dowolny dostęp do bazy danych backupu. Można używać predefiniowanych opcji lub wprowadzić własne zapytanie SQL

12 SQL commands to check DB To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved 3: Enter list of comma separated JobIds to select 4: Enter SQL list command 5: Select the most recent backup for a client 6: Select backup for a client before a specified time 7: Enter a list of files to restore 8: Enter a list of files to restore before a specified time 9: Find the JobIds of the most recent backup for a client 10: Find the JobIds for a backup for a client before a specified time 11: Enter a list of directories to restore for found JobIds 12: Cancel Select item: (1-12): 4 Enter SQL list command: select jobid, job, name, schedtime from Job where poolid= jobid job name schedtime Backup_gda:sim_archive_ _ _18 Backup gda:sim_archive :49: Backup_gda:sim_archive_ _ _03 Backup gda:sim_archive :38: Backup_gda:sim_archive_ _ _04 Backup gda:sim_archive :41: Backup_gda:sim_archive_ _ _05 Backup gda:sim_archive :45: Backup_gda:sim_archive_ _ _06 Backup gda:sim_archive :57: Backup_gda:sim_archive_ _ _15 Backup gda:sim_archive :13: Backup_gda:sim_archive_ _ _18 Backup gda:sim_archive :16: Catalog database

13 Bacula Catalog database

14 Cykl backupu Cykl odtworzenia plików bazuje na : Full Differential Incremental

15 Planowanie Backupu - Scheduling Full Full When the Level is set to Full all files in the FileSet whether or not they have changed will be backed up. Differential When the Level is set to Differential all files specified in the FileSet that have changed since the last successful Full backup of the same Job will be backed up. Incremental When the Level is set to Incremental all files specified in the FileSet that have changed will be backed up. If the Director cannot find a previous valid Full backup then the job will be upgraded into a Full backup.

16 Przykładowy scheduling Full, Differential, Inc Sob Nie Pon Wto Sro Czw Pia Sob Nie Pon Wto Sro Czw Pia Sob Nie Pon Wto Sro Czw Pia Sob Nie Pon Wto Sro Czw Pia Sob Nie Pon Wto Sro Czw Pia Full Diff Diff Diff Diff Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc

17 Cykl zadania - diagram czasowy Jest to ważne gdyż przeciągające się zadanie może zablokować dostęp do biblioteki. Np. jeżeli full-backup będzie trwał 26 godzin to nie ma potrzeby robienia backupu incrementalnego. Zakolejkowanie zadań może poważnie wpłynać na cykl backupowy Okna backupów powinno się ograniczyć np gdyż dostęp do biblioteki może być potrzebny do innych zadań (archive/restore)

18 Schemat diagramu czasowego dla zadania

19 Odtwarzanie z backupu

20 *restore Odtwarzanie plików z backupu (before date). The restored files will the most current backup BEFORE the date you specify below. Enter date as YYYY-MM-DD HH:MM:SS : :00:00 Defined Clients: 1: backup01.aaa.org. 12: appsrv07_fd Select the Client (1-12): 11 The defined FileSet resources are: 1: :gda. 4: :archive_aaa-aa Select FileSet resource (1-4): jobid level jobfiles jobbytes starttime volumename F 3,229,699 36,420,514, :10: L3 823 D 156,910 4,546,321, :56: L3 839 I , :33: L3 849 I 50, ,751, :10: L3 859 I 1, ,887, :10: L3 869 I 34, ,969, :10: L3 886 I 12, ,340, :10: L You have selected the following JobIds: 720,823,839,849,859,869,886 Building directory tree for JobId(s) 720,823,839,849,859,869, ,566,366 files inserted into the tree. `

21 *restore Odtwarzanie plików z backupu (most recent) To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved 3: Enter list of comma separated JobIds to select 4: Enter SQL list command 5: Select the most recent backup for a client 6: Select backup for a client before a specified time 7: Enter a list of files to restore 8: Enter a list of files to restore before a specified time 9: Find the JobIds of the most recent backup for a client 10: Find the JobIds for a backup for a client before a specified time 11: Enter a list of directories to restore for found JobIds 12: Cancel Select item: (1-12): 5 Defined Clients: 11: appsrv06-fd Select the Client (1-12): 11 The defined FileSet resources are: 1: :gda_xxxx Select FileSet resource (1-4): jobid level jobfiles jobbytes starttime volumename F 3,229,699 36,420,514, :10: L3 899 D 199,773 4,409,543, :56: L3 915 I :11: L3 925 I 82,352 1,231,279, :11: L You have selected the following JobIds: 720,899,915,925 Building directory tree for JobId(s) 720,899,915, ,569,689 files inserted into the tree.

22 Wyszukiwanie plików *query Available queries: 1: List up to 20 places where a File is saved regardless of the directory 2: List where the most recent copies of a file are saved 3: List last 20 Full Backups for a Client 4: List all backups for a Client after a specified time 5: List all backups for a Client 6: List Volume Attributes for a selected Volume 7: List Volumes used by selected JobId 8: List Volumes to Restore All Files 9: List Pool Attributes for a selected Pool 10: List total files/bytes by Job 11: List total files/bytes by Volume 12: List Files for a selected JobId 13: List Jobs stored on a selected MediaId 14: List Jobs stored for a given Volume name 15: List Volumes Bacula thinks are in changer 16: List Volumes likely to need replacement from age or errors Choose a query (1-16):

23 Cykl wykonania zadania. Job report 19-Nov 00:11 gdansk_backup01-gd_dir JobId 877: Start Backup JobId 877, Job=Backup_AAA:aaa_storage _ _12 19-Nov 00:12 gdansk_backup01_dir JobId 877: Using Device "Tape3" 19-Nov 00:12 gdansk_backup01_sd JobId 877: 3301 Issuing autochanger "loaded? drive 0" command. 19-Nov 00:12 gdansk_backup01_sd JobId 877: 3302 Autochanger "loaded? drive 0", result: nothing loaded. 19-Nov 00:12 gdansk_backup01_sd JobId 877: 3304 Issuing autochanger "load slot 13, drive 0" command. 19-Nov 00:13 gdansk_backup01_sd JobId 877: 3305 Autochanger "load slot 13, drive 0", status is OK. 19-Nov 00:13 gdansk_backup01_sd JobId 877: Volume "000013L3" previously written, moving to end of data. 19-Nov 00:14 gdansk_backup01_sd JobId 877: Ready to append to end of Volume "000013L3" at file= Nov 00:14 gdansk_backup01_sd JobId 877: Spooling data Nov 00:55 gdansk_backup01_sd JobId 877: Job write elapsed time = 00:41:17, Transfer rate = M bytes/second 19-Nov 00:55 gdansk_backup01_sd JobId 877: Committing spooled data to Volume "000013L3". Despooling 21,130,955,499 bytes Nov 01:04 gdansk_backup01_sd JobId 877: Despooling elapsed time = 00:08:38, Transfer rate = M bytes/second 19-Nov 01:05 gdansk_backup01_sd JobId 877: Alert: smartctl version 5.38 [i686-redhat-linux-gnu] Copyright (C) Bruce Allen 19-Nov 01:05 gdansk_backup01_sd JobId 877: Alert: Home page is 19-Nov 01:05 gdansk_backup01_sd JobId 877: Alert: 19-Nov 01:05 gdansk_backup01_sd JobId 877: Alert: TapeAlert: OK 19-Nov 01:05 gdansk_backup01-gd_sd JobId 877: Alert: 19-Nov 01:05 gdansk_backup01_sd JobId 877: Alert: Error Counter logging not supported 19-Nov 01:05 gd_backup01_sd JobId 877: Alert: 19-Nov 01:05 gdansk_backup01-gd_sd JobId 877: Alert: 8304ffa1: : : : Nov 01:05 gdansk_backup01-gd_sd JobId 877: Alert: 19-Nov 01:05 gdansk_backup01-gd_sd JobId 877: Sending spooled attrs to the Director. Despooling 3,327,526 bytes Nov 01:05 gdansk_backup01-gd_dir JobId 877: Bacula gdansk_backup01-dir (30Apr09): 19-Nov :05:36 Build OS: i686-redhat-linux-gnu redhat JobId: 877 Job: Backup_AAA:aaa_storage _ _12

24 Cykl wykonania zadania. Job report. Backup Level: Incremental, since= :12:35 Client: "appsrv07-fd" (30Apr09) i686-redhat-linux-gnu,redhat, FileSet: ":gda_storage" :10:01 Pool: "Pool_storage_every_day" (From Job resource) Catalog: "MyCatalog" (From Client resource) Storage: "PowerVault124T" (From Job resource) Scheduled time: 19-Nov :11:00 Start time: 19-Nov :12:48 End time: 19-Nov :05:36 Elapsed time: 52 mins 48 secs Priority: 10 FD Files Written: 8,602 SD Files Written: 8,602 FD Bytes Written: 21,107,427,981 (21.10 GB) SD Bytes Written: 21,109,101,169 (21.10 GB) Rate: KB/s Software Compression: 33.9 % VSS: no Encryption: no Accurate: no Volume name(s): L3 Volume Session Id: 73 Volume Session Time: Last Volume Bytes: 365,742,978,048 (365.7 GB) Non-fatal FD errors: 0 SD Errors: 1 FD termination status: OK SD termination status: OK Termination: Backup OK

25 Zarządzanie biblioteką Biblioteka robot z drivem (SCSI device) dostęp przez http z poziomu serwera

26 Dostęp do biblioteki przez http

27 Tape library status - console [root@backup01 ~]# /usr/sbin/mtx -f /dev/sg8 status Storage Changer /dev/sg8:1 Drives, 16 Slots ( 0 Import/Export ) Data Transfer Element 0:Full (Storage Element 7 Loaded):VolumeTag = L3 Storage Element 1:Full :VolumeTag=000075L3 Storage Element 2:Empty Storage Element 3:Full :VolumeTag=000003L3 Storage Element 4:Full :VolumeTag=000004L3 Storage Element 5:Full :VolumeTag=000047L3 Storage Element 6:Full :VolumeTag=000006L3 Storage Element 7:Empty Storage Element 8:Full :VolumeTag=000008L3 Storage Element 9:Full :VolumeTag=000009L3 Storage Element 10:Full :VolumeTag=000010L3 Storage Element 11:Full :VolumeTag=000011L3 Storage Element 12:Empty Storage Element 13:Full :VolumeTag=000013L3 Storage Element 14:Full :VolumeTag=000014L3 Storage Element 15:Empty Storage Element 16:Full

28 Restore cycle example *restore First you select one or more JobIds that contain files to be restored. You will be presented several methods of specifying the JobIds. Then you will be allowed to select which files from those JobIds are to be restored. To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved 3: Enter list of comma separated JobIds to select 4: Enter SQL list command 5: Select the most recent backup for a client 6: Select backup for a client before a specified time 7: Enter a list of files to restore 8: Enter a list of files to restore before a specified time 9: Find the JobIds of the most recent backup for a client 10: Find the JobIds for a backup for a client before a specified time 11: Enter a list of directories to restore for found JobIds 12: Cancel Select item: (1-12): 5 Defined Clients: 1: backup01-gd.chipidea.com 2: terminalsrv07_fd 3: terminalsrv05_fd 4: terminalsrv06_fd 5: terminalsrv01_fd 6: terminalsrv02_fd 7: terminalsrv03_fd 8: terminalsrv04_fd 9: gda34_fd 10: appsrv05_fd 11: appsrv06_fd 12: appsrv07_fd

29 Restore cycle example Select the Client (1-12): 10 The defined FileSet resources are: 1: :gda33_aaa 2: :gda33_bbb Select FileSet resource (1-2): jobid level jobfiles jobbytes starttime volumename F 1,101,766 20,811,002, :10: L3 721 F 1,101,766 20,811,002, :10: L3 824 D ,923, :56: L3 840 I :27: L3 850 I 106 1,219, :10: L3 860 I 6 3, :10: L3 870 I 7,284 16,220, :10: L You have selected the following JobIds: 721,824,840,850,860,870 Building directory tree for JobId(s) 721,824,840,850,860, You are now entering file selection mode where you add (mark) and remove (unmark) files to be restored. No files are initially added, unless you used the "all" keyword on the command line. Enter "done" to leave this mode. cwd is: / $ dir drwxr-xr-x 17 root root :14:49 /usrappsrv05 $ done No files selected to be restored.

30 Purged vs Pruned Pruned Purged

31 Pruning As Bacula writes files to tape, it keeps a list of files, jobs, and volumes in a database called the catalog. Among other things, the database helps Bacula to decide which files to back up in an incremental or differential backup, and helps you locate files on past backups when you want to restore something. Bacula's process for removing entries from the catalog is called Pruning. The default is Automatic Pruning, which means that once an entry reaches a certain age (e.g. 30 days old) it is removed from the catalog. Once a job has been pruned, you can still restore it from the backup tape, but one additional step is required: scanning the volume with bscan.

32 Purge Assuming that you want to keep the Volume name, but you simply want to write new data on the tape, the steps to take are: Use the update volume command in the Console to ensure that the Recycle field is set to 1 Use the purge jobs volume command in the Console to mark the Volume as Purged. Check by using list volumes.

33 Media live time

34 What if pruned? Purged vs Pruned Once a job has been pruned, you can still restore it from the backup tape, but one additional step is required: scanning the volume with bscan.

35 Konfiguracja

36 bconsole. = cancel

37 Zmiana konfiguracji *reload You have messages. *mess 19-Nov 08:21 gdansk_backup01_dir: ERROR in parse_conf.c:477 Config error: Could not find config Resource gda36 referenced on line 282 : Client = gda36 : line 282, col 16 of file /etc/bacula/bacula_jobs.conf Client = gda36

38 Dobrym pomysłem jest konfigurowanie Baculi przy pomocy plików includowanych. Director { # define myself Name = gdansk_backup01_dir DIRport = 9101 # where we listen for UA connections QueryFile = "/backup1/bacula/lib/bacula/query.sql" WorkingDirectory = "/backup1/bacula/lib/bacula" PidDirectory = "/var/run" Maximum Concurrent Jobs = 30 Password = "test1" # Console password Messages = @/etc/bacula/bacula_pools.conf Catalog { Name = MyCatalog dbaddress = ; dbport = 5432 dbname = "bacula"; dbuser = "bacula"; dbpassword = "" }. bacula-dir.conf

39 @bacula_schedule.conf Schedule { Name = "WeeklyCycle" Run = Full 1st sun at 23:05 Run = Differential 2nd-5th sun at 23:05 Run = Incremental mon-sat at 23:05 } # This schedule does the catalog. It starts after the WeeklyCycle Schedule { Name = "WeeklyCycleAfterBackup" Run = Full sun-sat at 23:10 } Schedule { Name = gdansk_daily" Run = Full 1st sun at 00:10 Run = Differential 2nd-5th fri at 22:55 Run = Incremental sun-thu at 22:10 } Schedule { Name = "gdansk_storage_daily_b" Run = Full 1st Fri at 23:15 Run = Differential 2nd, 3rd, 4th, 5th fri at 23:25 Run = Incremental sun-fri at 00:11 } Schedule { Name = "gdansk_storage_daily_a" Run = Full 2nd Fri at 23:15 Run = Differential 1st, 3rd, 4th, 5th Fri at 23:25 Run = Incremental sun-fri at 00:11 } Schedule { Name = "gdansk_storage_daily_ab" Run = Full 3rd Fri at 23:15 Run = Differential 1st, 2nd, 4th, 5th Fri at 23:25 Run = Incremental sun-fri at 00:11 }

40 Backup vs Archive Backup Auto Purge Auto Prune Retention 3 months Archive No prune No purge. Retention 100 years

41 Medium backupu dyskietka płyta cd/dvd dyski taśmy

42 Backup medium np. taśmy

43 Add new tapes to Bacula Dodać volumen do katalogu Baculi (Najlepiej jest dodać do scratch całą listę volumenow a następnie w miare możliwości operować na dostępnych fizycznych odpowiednikach) Włożyć nośnik Zapisać nazwę na nośniku (label)

44 Before job Create snapshot Backup scripts After job Destroy snapshot

45 Dostęp do Baculi Tryb tekstowy bconsole BAT (Bacula Admin Tool)

46 webacula

47 Monitoring Bacula może zapisywać operacje w logach. Bacula może wysyłać maila. Monitoring Bacula with Nagios bacula-dir.conf : Run After Job = "/usr/local/sbin/bacula2nagios \"%n\" 0 \"%e %l %v\"" Run After Failed Job = "/usr/local/sbin/bacula2nagios \"%n\" 1 \"%e %l %v\""

48 Adding new job Configure Pool Configure FileSet Backup Restore Configure Job_def Configure Job Configure Schedule (most important)

49 Archive scripts Before Collect data Create info file arch.txt After Update list of arch.txt files with job info

50 Implementacja backupu - konsekwencje Wybranie metody backupu determinuje używanie danego rozwiązania tak długo, jak długo przechowywane dane na taśmie będą potrzebne (restore).

51 Podsumowanie Bacula jest równorzędnym programem do robienia backupu w porówaniu z rozwiązaniami komercyjnymi Koniguracja sprawia dużo radości przeciętnemu consolowemu gnomowi Zachęcam do eksperymentów z Bacula

52 Temat wystąpienia : Bacula : Czy warto? Bacula : Czy warto używać tego rozwiązania w trybie produkcyjnym? Każdy musi odpowiedzieć sobie na to pytanie sam. ;-)

53 Dziękuję za uwagę.