Bacula Czy warto? Tomasz Górny tgorn(at)onet.pl Zimowisko 2010
www.bacula.org Bacula - The Open Source Network Backup Solution www.bacula.org It comes by night and sucks the vital essence from your computers.
http://www.baculasystems.com
Rozproszona architektura 4
Zalety Open Source Brak pojedynczego punktu awarii Różne funkcje mogą być rozmieszczone na osobnych komputerach Ciągły rozwój
Wady Architektura rozproszona Konfiguracja Brak dobrego interfejsu graficznego (webacula?)
Przykładowa konfiguracja
http://wiki.bacula.org/doku.php?id=comparisons
Instalacja yum install bacula-dir ======== Matched: bacula ======= bacula-client.i386 : Bacula backup client bacula-common.i386 : Common Bacula utilities bacula-console.i386 : Bacula management console bacula-console-gnome.i386 : Bacula console for the Gnome desktop environment bacula-console-wxwidgets.i386 : Bacula console using the wx widgets toolkit bacula-director-common.i386 : Common Bacula Director files bacula-director-mysql.i386 : Bacula Director with MySQL database support bacula-director-postgresql.i386 : Bacula Director with PostgresSQL database support bacula-director-sqlite.i386 : Bacula Director with sqlite database support bacula-docs.i386 : Bacula documentation bacula-storage-common.i386 : Common Bacula storage daemon files bacula-storage-mysql.i386 : MySQL Bacula storage daemon files bacula-storage-postgresql.i386 : Common Bacula storage daemon files bacula-storage-sqlite.i386 : SQLite Bacula storage daemon files bacula-traymonitor.i386 : Bacula monitor for the Gnome and KDE system tray webacula.noarch : Web interface of a Bacula backup system
Instalacja ==================================================================================== Package Arch Version Repository Size ==================================================================================== Installing: bacula-director-postgresql i386 2.0.3-10.el5 epel 832 k Installing for dependencies: bacula-common i386 2.0.3-10.el5 epel 354 k bacula-director-common i386 2.0.3-10.el5 epel 169 k fedora-usermgmt noarch 0.8.91-1.el5 epel 6.6 k fedora-usermgmt-core noarch 0.8.91-1.el5 epel 7.6 k fedora-usermgmt-default-fedora-setup noarch 0.8.91-1.el5 epel 6.6 k fedora-usermgmt-shadow-utils noarch 0.8.91-1.el5 epel 7.4 k postgresql i386 8.1.18-2.el5_4.1 updates 2.8 M postgresql-server i386 8.1.18-2.el5_4.1 updates 3.8 M Transaction Summary ==================================================================================== Install 9 Package(s) Update 0 Package(s) Remove 0 Package(s) Total download size: 8.0 M Is this ok [y/n]:
Dostęp do katalogu Bacula pozwala na dowolny dostęp do bazy danych backupu. Można używać predefiniowanych opcji lub wprowadzić własne zapytanie SQL
SQL commands to check DB To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved 3: Enter list of comma separated JobIds to select 4: Enter SQL list command 5: Select the most recent backup for a client 6: Select backup for a client before a specified time 7: Enter a list of files to restore 8: Enter a list of files to restore before a specified time 9: Find the JobIds of the most recent backup for a client 10: Find the JobIds for a backup for a client before a specified time 11: Enter a list of directories to restore for found JobIds 12: Cancel Select item: (1-12): 4 Enter SQL list command: select jobid, job, name, schedtime from Job where poolid=7 +-------+----------------------------------------------------+-----------------------------+---------------------+ jobid job name schedtime +-------+----------------------------------------------------+-----------------------------+---------------------+ 879 Backup_gda:sim_archive_2009-11-19_11.50.03_18 Backup gda:sim_archive 2009-11-19 11:49:36 882 Backup_gda:sim_archive_2009-11-19_13.38.18_03 Backup gda:sim_archive 2009-11-19 13:38:09 883 Backup_gda:sim_archive_2009-11-19_13.41.12_04 Backup gda:sim_archive 2009-11-19 13:41:09 884 Backup_gda:sim_archive_2009-11-19_13.45.29_05 Backup gda:sim_archive 2009-11-19 13:45:21 885 Backup_gda:sim_archive_2009-11-19_13.57.37_06 Backup gda:sim_archive 2009-11-19 13:57:33 896 Backup_gda:sim_archive_2009-11-20_15.13.58_15 Backup gda:sim_archive 2009-11-20 15:13:48 898 Backup_gda:sim_archive_.2009-11-20_20.17.07_18 Backup gda:sim_archive 2009-11-20 20:16:52 +-------+----------------------------------------------------+-----------------------------+---------------------+ Catalog database
Bacula Catalog database
Cykl backupu Cykl odtworzenia plików bazuje na : Full Differential Incremental
Planowanie Backupu - Scheduling Full Full When the Level is set to Full all files in the FileSet whether or not they have changed will be backed up. Differential When the Level is set to Differential all files specified in the FileSet that have changed since the last successful Full backup of the same Job will be backed up. Incremental When the Level is set to Incremental all files specified in the FileSet that have changed will be backed up. If the Director cannot find a previous valid Full backup then the job will be upgraded into a Full backup.
Przykładowy scheduling Full, Differential, Inc 1 2 3 4 5 Sob Nie Pon Wto Sro Czw Pia Sob Nie Pon Wto Sro Czw Pia Sob Nie Pon Wto Sro Czw Pia Sob Nie Pon Wto Sro Czw Pia Sob Nie Pon Wto Sro Czw Pia Full Diff Diff Diff Diff Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc Inc
Cykl zadania - diagram czasowy Jest to ważne gdyż przeciągające się zadanie może zablokować dostęp do biblioteki. Np. jeżeli full-backup będzie trwał 26 godzin to nie ma potrzeby robienia backupu incrementalnego. Zakolejkowanie zadań może poważnie wpłynać na cykl backupowy Okna backupów powinno się ograniczyć np. 2100 0900 gdyż dostęp do biblioteki może być potrzebny do innych zadań (archive/restore)
Schemat diagramu czasowego dla zadania
Odtwarzanie z backupu
*restore Odtwarzanie plików z backupu (before date). The restored files will the most current backup BEFORE the date you specify below. Enter date as YYYY-MM-DD HH:MM:SS :2009-11-20 20:00:00 Defined Clients: 1: backup01.aaa.org. 12: appsrv07_fd Select the Client (1-12): 11 The defined FileSet resources are: 1: :gda. 4: :archive_aaa-aa Select FileSet resource (1-4): 1 +-------+-------+-----------+----------------+---------------------+------------+ jobid level jobfiles jobbytes starttime volumename +-------+-------+-----------+----------------+---------------------+------------+ 720 F 3,229,699 36,420,514,476 2009-11-01 00:10:01 000003L3 823 D 156,910 4,546,321,512 2009-11-13 22:56:50 000003L3 839 I 24 303,042 2009-11-16 02:33:41 000003L3 849 I 50,221 425,751,673 2009-11-16 22:10:01 000003L3 859 I 1,729 191,887,497 2009-11-17 22:10:01 000003L3 869 I 34,327 370,969,471 2009-11-18 22:10:00 000003L3 886 I 12,375 671,340,655 2009-11-19 22:10:56 000003L3 +-------+-------+-----------+----------------+---------------------+------------+ You have selected the following JobIds: 720,823,839,849,859,869,886 Building directory tree for JobId(s) 720,823,839,849,859,869,886... +++++++++++++++++++++++++++++++++++++++ 2,566,366 files inserted into the tree. `
*restore Odtwarzanie plików z backupu (most recent) To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved 3: Enter list of comma separated JobIds to select 4: Enter SQL list command 5: Select the most recent backup for a client 6: Select backup for a client before a specified time 7: Enter a list of files to restore 8: Enter a list of files to restore before a specified time 9: Find the JobIds of the most recent backup for a client 10: Find the JobIds for a backup for a client before a specified time 11: Enter a list of directories to restore for found JobIds 12: Cancel Select item: (1-12): 5 Defined Clients: 11: appsrv06-fd Select the Client (1-12): 11 The defined FileSet resources are: 1: :gda_xxxx Select FileSet resource (1-4): 1 +-------+-------+-----------+----------------+---------------------+------------+ jobid level jobfiles jobbytes starttime volumename +-------+-------+-----------+----------------+---------------------+------------+ 720 F 3,229,699 36,420,514,476 2009-11-01 00:10:01 000003L3 899 D 199,773 4,409,543,176 2009-11-21 22:56:05 000003L3 915 I 5 0 2009-11-22 22:11:07 000003L3 925 I 82,352 1,231,279,063 2009-11-23 22:11:31 000003L3 +-------+-------+-----------+----------------+---------------------+------------+ You have selected the following JobIds: 720,899,915,925 Building directory tree for JobId(s) 720,899,915,925... +++++++++++++++++++++++++++++++++++++++ 2,569,689 files inserted into the tree.
Wyszukiwanie plików *query Available queries: 1: List up to 20 places where a File is saved regardless of the directory 2: List where the most recent copies of a file are saved 3: List last 20 Full Backups for a Client 4: List all backups for a Client after a specified time 5: List all backups for a Client 6: List Volume Attributes for a selected Volume 7: List Volumes used by selected JobId 8: List Volumes to Restore All Files 9: List Pool Attributes for a selected Pool 10: List total files/bytes by Job 11: List total files/bytes by Volume 12: List Files for a selected JobId 13: List Jobs stored on a selected MediaId 14: List Jobs stored for a given Volume name 15: List Volumes Bacula thinks are in changer 16: List Volumes likely to need replacement from age or errors Choose a query (1-16):
Cykl wykonania zadania. Job report 19-Nov 00:11 gdansk_backup01-gd_dir JobId 877: Start Backup JobId 877, Job=Backup_AAA:aaa_storage.2009-11- 19_00.11.00_12 19-Nov 00:12 gdansk_backup01_dir JobId 877: Using Device "Tape3" 19-Nov 00:12 gdansk_backup01_sd JobId 877: 3301 Issuing autochanger "loaded? drive 0" command. 19-Nov 00:12 gdansk_backup01_sd JobId 877: 3302 Autochanger "loaded? drive 0", result: nothing loaded. 19-Nov 00:12 gdansk_backup01_sd JobId 877: 3304 Issuing autochanger "load slot 13, drive 0" command. 19-Nov 00:13 gdansk_backup01_sd JobId 877: 3305 Autochanger "load slot 13, drive 0", status is OK. 19-Nov 00:13 gdansk_backup01_sd JobId 877: Volume "000013L3" previously written, moving to end of data. 19-Nov 00:14 gdansk_backup01_sd JobId 877: Ready to append to end of Volume "000013L3" at file=347. 19-Nov 00:14 gdansk_backup01_sd JobId 877: Spooling data... 19-Nov 00:55 gdansk_backup01_sd JobId 877: Job write elapsed time = 00:41:17, Transfer rate = 8.522 M bytes/second 19-Nov 00:55 gdansk_backup01_sd JobId 877: Committing spooled data to Volume "000013L3". Despooling 21,130,955,499 bytes... 19-Nov 01:04 gdansk_backup01_sd JobId 877: Despooling elapsed time = 00:08:38, Transfer rate = 40.79 M bytes/second 19-Nov 01:05 gdansk_backup01_sd JobId 877: Alert: smartctl version 5.38 [i686-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen 19-Nov 01:05 gdansk_backup01_sd JobId 877: Alert: Home page is http://smartmontools.sourceforge.net/ 19-Nov 01:05 gdansk_backup01_sd JobId 877: Alert: 19-Nov 01:05 gdansk_backup01_sd JobId 877: Alert: TapeAlert: OK 19-Nov 01:05 gdansk_backup01-gd_sd JobId 877: Alert: 19-Nov 01:05 gdansk_backup01_sd JobId 877: Alert: Error Counter logging not supported 19-Nov 01:05 gd_backup01_sd JobId 877: Alert: 19-Nov 01:05 gdansk_backup01-gd_sd JobId 877: Alert: 8304ffa1:02120406:00000000:00000000:00000000 19-Nov 01:05 gdansk_backup01-gd_sd JobId 877: Alert: 19-Nov 01:05 gdansk_backup01-gd_sd JobId 877: Sending spooled attrs to the Director. Despooling 3,327,526 bytes... 19-Nov 01:05 gdansk_backup01-gd_dir JobId 877: Bacula gdansk_backup01-dir 3.0.1 (30Apr09): 19-Nov-2009 01:05:36 Build OS: i686-redhat-linux-gnu redhat JobId: 877 Job: Backup_AAA:aaa_storage.2009-11-19_00.11.00_12
Cykl wykonania zadania. Job report. Backup Level: Incremental, since=2009-11-18 00:12:35 Client: "appsrv07-fd" 3.0.1 (30Apr09) i686-redhat-linux-gnu,redhat, FileSet: ":gda_storage" 2009-08-14 22:10:01 Pool: "Pool_storage_every_day" (From Job resource) Catalog: "MyCatalog" (From Client resource) Storage: "PowerVault124T" (From Job resource) Scheduled time: 19-Nov-2009 00:11:00 Start time: 19-Nov-2009 00:12:48 End time: 19-Nov-2009 01:05:36 Elapsed time: 52 mins 48 secs Priority: 10 FD Files Written: 8,602 SD Files Written: 8,602 FD Bytes Written: 21,107,427,981 (21.10 GB) SD Bytes Written: 21,109,101,169 (21.10 GB) Rate: 6662.7 KB/s Software Compression: 33.9 % VSS: no Encryption: no Accurate: no Volume name(s): 000013L3 Volume Session Id: 73 Volume Session Time: 1257895716 Last Volume Bytes: 365,742,978,048 (365.7 GB) Non-fatal FD errors: 0 SD Errors: 1 FD termination status: OK SD termination status: OK Termination: Backup OK
Zarządzanie biblioteką Biblioteka robot z drivem (SCSI device) dostęp przez http z poziomu serwera
Dostęp do biblioteki przez http
Tape library status - console [root@backup01 ~]# /usr/sbin/mtx -f /dev/sg8 status Storage Changer /dev/sg8:1 Drives, 16 Slots ( 0 Import/Export ) Data Transfer Element 0:Full (Storage Element 7 Loaded):VolumeTag = 000032L3 Storage Element 1:Full :VolumeTag=000075L3 Storage Element 2:Empty Storage Element 3:Full :VolumeTag=000003L3 Storage Element 4:Full :VolumeTag=000004L3 Storage Element 5:Full :VolumeTag=000047L3 Storage Element 6:Full :VolumeTag=000006L3 Storage Element 7:Empty Storage Element 8:Full :VolumeTag=000008L3 Storage Element 9:Full :VolumeTag=000009L3 Storage Element 10:Full :VolumeTag=000010L3 Storage Element 11:Full :VolumeTag=000011L3 Storage Element 12:Empty Storage Element 13:Full :VolumeTag=000013L3 Storage Element 14:Full :VolumeTag=000014L3 Storage Element 15:Empty Storage Element 16:Full
Restore cycle example *restore First you select one or more JobIds that contain files to be restored. You will be presented several methods of specifying the JobIds. Then you will be allowed to select which files from those JobIds are to be restored. To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved 3: Enter list of comma separated JobIds to select 4: Enter SQL list command 5: Select the most recent backup for a client 6: Select backup for a client before a specified time 7: Enter a list of files to restore 8: Enter a list of files to restore before a specified time 9: Find the JobIds of the most recent backup for a client 10: Find the JobIds for a backup for a client before a specified time 11: Enter a list of directories to restore for found JobIds 12: Cancel Select item: (1-12): 5 Defined Clients: 1: backup01-gd.chipidea.com 2: terminalsrv07_fd 3: terminalsrv05_fd 4: terminalsrv06_fd 5: terminalsrv01_fd 6: terminalsrv02_fd 7: terminalsrv03_fd 8: terminalsrv04_fd 9: gda34_fd 10: appsrv05_fd 11: appsrv06_fd 12: appsrv07_fd
Restore cycle example Select the Client (1-12): 10 The defined FileSet resources are: 1: :gda33_aaa 2: :gda33_bbb Select FileSet resource (1-2): 1 +-------+-------+-----------+----------------+---------------------+------------+ jobid level jobfiles jobbytes starttime volumename +-------+-------+-----------+----------------+---------------------+------------+ 721 F 1,101,766 20,811,002,153 2009-11-01 00:10:01 000003L3 721 F 1,101,766 20,811,002,153 2009-11-01 00:10:01 000009L3 824 D 340 50,923,070 2009-11-13 22:56:50 000003L3 840 I 1 0 2009-11-15 22:27:32 000003L3 850 I 106 1,219,205 2009-11-16 22:10:01 000003L3 860 I 6 3,771 2009-11-17 22:10:01 000003L3 870 I 7,284 16,220,153 2009-11-18 22:10:00 000003L3 +-------+-------+-----------+----------------+---------------------+------------+ You have selected the following JobIds: 721,824,840,850,860,870 Building directory tree for JobId(s) 721,824,840,850,860,870... ++++++++++++++++++++++++++++++++++++++ You are now entering file selection mode where you add (mark) and remove (unmark) files to be restored. No files are initially added, unless you used the "all" keyword on the command line. Enter "done" to leave this mode. cwd is: / $ dir drwxr-xr-x 17 root root 4096 2009-08-19 08:14:49 /usrappsrv05 $ done No files selected to be restored.
Purged vs Pruned Pruned Purged
Pruning As Bacula writes files to tape, it keeps a list of files, jobs, and volumes in a database called the catalog. Among other things, the database helps Bacula to decide which files to back up in an incremental or differential backup, and helps you locate files on past backups when you want to restore something. Bacula's process for removing entries from the catalog is called Pruning. The default is Automatic Pruning, which means that once an entry reaches a certain age (e.g. 30 days old) it is removed from the catalog. Once a job has been pruned, you can still restore it from the backup tape, but one additional step is required: scanning the volume with bscan.
Purge Assuming that you want to keep the Volume name, but you simply want to write new data on the tape, the steps to take are: Use the update volume command in the Console to ensure that the Recycle field is set to 1 Use the purge jobs volume command in the Console to mark the Volume as Purged. Check by using list volumes.
Media live time
What if pruned? Purged vs Pruned Once a job has been pruned, you can still restore it from the backup tape, but one additional step is required: scanning the volume with bscan.
Konfiguracja http://www.bacula.org/3.1.x-manuals/en/main/main/index.html
bconsole. = cancel
Zmiana konfiguracji *reload You have messages. *mess 19-Nov 08:21 gdansk_backup01_dir: ERROR in parse_conf.c:477 Config error: Could not find config Resource gda36 referenced on line 282 : Client = gda36 : line 282, col 16 of file /etc/bacula/bacula_jobs.conf Client = gda36
Dobrym pomysłem jest konfigurowanie Baculi przy pomocy plików includowanych. Director { # define myself Name = gdansk_backup01_dir DIRport = 9101 # where we listen for UA connections QueryFile = "/backup1/bacula/lib/bacula/query.sql" WorkingDirectory = "/backup1/bacula/lib/bacula" PidDirectory = "/var/run" Maximum Concurrent Jobs = 30 Password = "test1" # Console password Messages = Daemon } @/etc/bacula/bacula_schedule.conf @/etc/bacula/bacula_clients.conf @/etc/bacula/bacula_filesets.conf @/etc/bacula/bacula_storages.conf @/etc/bacula/bacula_pools.conf @/etc/bacula/bacula_jobs.conf @/etc/bacula/bacula_jobsdef.conf Catalog { Name = MyCatalog dbaddress = 10.124.8.22; dbport = 5432 dbname = "bacula"; dbuser = "bacula"; dbpassword = "" }. bacula-dir.conf
@bacula_schedule.conf Schedule { Name = "WeeklyCycle" Run = Full 1st sun at 23:05 Run = Differential 2nd-5th sun at 23:05 Run = Incremental mon-sat at 23:05 } # This schedule does the catalog. It starts after the WeeklyCycle Schedule { Name = "WeeklyCycleAfterBackup" Run = Full sun-sat at 23:10 } Schedule { Name = gdansk_daily" Run = Full 1st sun at 00:10 Run = Differential 2nd-5th fri at 22:55 Run = Incremental sun-thu at 22:10 } Schedule { Name = "gdansk_storage_daily_b" Run = Full 1st Fri at 23:15 Run = Differential 2nd, 3rd, 4th, 5th fri at 23:25 Run = Incremental sun-fri at 00:11 } Schedule { Name = "gdansk_storage_daily_a" Run = Full 2nd Fri at 23:15 Run = Differential 1st, 3rd, 4th, 5th Fri at 23:25 Run = Incremental sun-fri at 00:11 } Schedule { Name = "gdansk_storage_daily_ab" Run = Full 3rd Fri at 23:15 Run = Differential 1st, 2nd, 4th, 5th Fri at 23:25 Run = Incremental sun-fri at 00:11 }
Backup vs Archive Backup Auto Purge Auto Prune Retention 3 months Archive No prune No purge. Retention 100 years
Medium backupu dyskietka płyta cd/dvd dyski taśmy
Backup medium np. taśmy
Add new tapes to Bacula Dodać volumen do katalogu Baculi (Najlepiej jest dodać do scratch całą listę volumenow a następnie w miare możliwości operować na dostępnych fizycznych odpowiednikach) Włożyć nośnik Zapisać nazwę na nośniku (label)
Before job Create snapshot Backup scripts After job Destroy snapshot
Dostęp do Baculi Tryb tekstowy bconsole BAT (Bacula Admin Tool)
webacula
Monitoring Bacula może zapisywać operacje w logach. Bacula może wysyłać maila. Monitoring Bacula with Nagios http://wiki.bacula.org/doku.php?id=nagios bacula-dir.conf : Run After Job = "/usr/local/sbin/bacula2nagios \"%n\" 0 \"%e %l %v\"" Run After Failed Job = "/usr/local/sbin/bacula2nagios \"%n\" 1 \"%e %l %v\""
Adding new job Configure Pool Configure FileSet Backup Restore Configure Job_def Configure Job Configure Schedule (most important)
Archive scripts Before Collect data Create info file arch.txt After Update list of arch.txt files with job info
Implementacja backupu - konsekwencje Wybranie metody backupu determinuje używanie danego rozwiązania tak długo, jak długo przechowywane dane na taśmie będą potrzebne (restore).
Podsumowanie Bacula jest równorzędnym programem do robienia backupu w porówaniu z rozwiązaniami komercyjnymi Koniguracja sprawia dużo radości przeciętnemu consolowemu gnomowi Zachęcam do eksperymentów z Bacula
Temat wystąpienia : Bacula : Czy warto? Bacula : Czy warto używać tego rozwiązania w trybie produkcyjnym? Każdy musi odpowiedzieć sobie na to pytanie sam. ;-)
Dziękuję za uwagę.