A Method Supporting Monitoring And Repair Processes of Information Systems

Transkrypt

1 GDANSK UNIVERSITY OF TECHNOLOGY Faculty of Electronics, Telecommunications and Informatics Marek Kamiński A Method Supporting Monitoring And Repair Processes of Information Systems PhD Dissertation Supervisor: prof. dr hab. inż. Janusz Górski Faculty of Electronics, Telecommunications and Informatics Gdansk University of Technology Gdansk, 2011

2 Mojemu Bogu, Panu, Stworzycielowi i Zbawicielowi Jezusowi Chrystusowi Mojej Ukochanej Żonie Agnieszce Moim Drogim Rodzicom Marianowi i Zenobii Za wszystko co Wam zawdzięczam, za waszą miłośd i za waszą obecnośd Kocham Was, Jesteście Wszystkim co mam Dedykuję tę rozprawę dla Was To My God, Lord, Creator and Saviour Jesus Christ To My Beloved Wife Agnieszka To My Dear Parents Marian and Zenia For all I owe You, for Your love and Your presence I Love You, You Are Everything I have I dedicate this dissertation to You ii

3 Abstract This thesis proposes a new method of on-the-fly monitoring and repair of IT systems, called ARM. The method integrates monitoring processes (taking into account the already existing and used monitoring systems) with repair processes. A conceptual model of monitoring and repair was proposed and formally specified together with formal specifications of the repair mechanism and selected repair procedures. The formal specifications were expressed in Z notation. These specifications were subjected to analyses verifying their adequacy and consistency (the Z/EVES tool was applied for consistency checks). The specifications comprise an abstract definition of ARM which can be then used to drive implementation of the method in a chosen target environment. They were used to drive a prototype implementation of ARM in the industrial environment of Lufthansa Systems Poland company. The prototype was then used in a series of experiments and case studies targeting at assessment of effectiveness and efficiency of ARM. Effectiveness was demonstrated by implementing a set of genetic repair procedures and integrating the existing and widely used monitoring system NAGIOS. Efficiency was assessed by evaluating the gain resulting from using ARM in comparison with the situation were the repairs were conducted in a traditional way. The results from the experiments and case studies fully confirmed that ARM brings a considerable added value to the processes of monitoring and repair of industrial IT systems. Streszczenie W pracy zaproponowano nową metodę (ARM) zarządzania monitorowaniem i naprawą systemów IT, bez przerywania ciągłości ich funkcjonowania. Metoda ta integruje procesy monitorowania (uwzględniając już istniejące i stosowane rozwiązania) z procesami naprawy. Stworzono formalny model koncepcyjny opisujący dziedzinę monitorowania i naprawy oraz formalne specyfikacje mechanizmu napraw oraz wybranych procedur naprawczych. Do specyfikacji formalnych wykorzystano metodę Z. Specyfikacje te zostały poddane analizie pod kątem oceny ich adekwatności i spójności (wykorzystano tu narzędzie Z/EVES). Specyfikacje te stanowią abstrakcyjną definicję metody ARM, która może być wykorzystana do implementacji metody w wybranym środowisku docelowym. Wykorzystano je do wytworzenia prototypu wspierającego ARM, który został zrealizowany i poddany badaniom eksperymentalnym w firmie Lufthansa Systems Poland. Badania te miały na celu ocenę skuteczności i wydajności metody ARM. Skuteczność została potwierdzona przez implementację reprezentatywnego zestawu procedur naprawczych oraz integrację z przemysłowym systemem monitorowania NAGIOS. Wydajność oceniono poprzez ocenę poprawy czasu realizacji napraw przy wykorzystaniu metody ARM w stosunku do sytuacji, gdy nie była ona stosowana. Przeprowadzone badania w pełni potwierdziły przydatność metody ARM w warunkach przemysłowych. iii

4 Table of Contents CHAPTER 1. INTRODUCTION ROLE AND IMPORTANCE OF MONITORING AND REPAIR CURRENT STATE OF ART RATIONALE THESIS PROPOSITION SCOPE OF RESEARCH THESIS STRUCTURE... 4 CHAPTER 2. MONITORING AND REPAIR PROCESSES A MONITORING PROCESS Common view on the monitored reality Monitoring mechanisms A monitoring process model A REPAIR PROCESS A repair process model OVERVIEW OF PROBLEMS AND THEIR SOLUTIONS Problem 1: Wrong entries in scheduler table Problem 2: Backup failures preventing DBMS from working Problem 3: Failures during optimizing physical structures of production database Problem 4: Unacceptable delay in replication of time-critical data GENERIC SOLUTIONS TO PROBLEMS Solution to problem 1: Changing scheduler table Solution to problem 2: Bringing DBMS to working status Solution to problem 3: Optimizing unoptimized database structures Solution to problem 4: Forcing emergency replication of time-critical data SUMMARY CHAPTER 3. ARM A METHOD OF AUTOMATED REPAIR MANAGEMENT INTRODUCTION COMPONENTS OF THE METHOD RMM Repair Management Model RMF Repair Management Framework RMS Repair Management System ARM: STEPS OF THE METHOD FORMAL DEFINITION OF RMM Submodel of monitoring Submodel of repair Problem detectors DEFINITION OF RMF Repair library Interfacing steps to the calling repair procedure DEFINITION OF RMS Integrating the ideas SUMMARY CHAPTER 4. ARCHITECTURE OF RMS BASIC ASSUMPTIONS PREPARATION ACTIVITIES ARCHITECTURAL MODEL OF SYSTEM FUNCTIONING OF SYSTEM ARCHITECTURE OF REPAIR API TEMPLATE OF A TYPICAL REPAIR PROCEDURE SUMMARY CHAPTER 5. ASSESSMENT OF ARM THE APPROACH INTRODUCTION Scope of assessment Criteria of assessment CONSISTENCY CRITERIA iv

5 5.3 EFFECTIVENESS CRITERIA EFFICIENCY CRITERIA Sources of data Encountered problems Comparison rules Test environment SUMMARY CHAPTER 6. EXPERIMENTS AND CASE STUDIES INTRODUCTION CASE 1: CHANGING SCHEDULER TABLE Detailed description Unique features of the case Summary and results of experiments CASE 2: BRINGING DBMS TO WORKING STATUS Detailed description Unique features of the case Summary and results of experiments CASE 3: OPTIMIZING UNOPTIMIZED DATABASE STRUCTURES Detailed description Unique features of the case Summary and results of experiments CASE 4: FORCING EMERGENCY REPLICATION OF TIME-CRITICAL DATA Detailed description Unique features of the case Summary and results of experiments DISCUSSION OF OBTAINED EFFICIENCY GAIN SUMMARY CHAPTER 7. CONCLUSIONS INTRODUCTION RESULTS OF ASSESSMENT ORIGINAL CONTRIBUTION Contributions at the conceptual level Contributions at the realisation level BENEFITS AND LIMITATIONS FURTHER WORK AREAS REFERENCES APPENDIX A. EXTENDED ABSTRACT IN POLISH A.1 WPROWADZENIE A.2 PROCESY MONITOROWANIA I NAPRAWY A.3 TYPOWE PROBLEMY I ALGORYTMY ICH NAPRAWY A.4 ARM METODA ZAUTOMATYZOWANEGO ZARZĄDZANIA NAPRAWAMI A.5 ARCHITEKTURA SYSTEMU RMS A.6 OCENA METODY ARM A.7 EKSPERYMENTY I STUDIA PRZYPADKÓW A.8 PODSUMOWANIE APPENDIX B. THE Z NOTATION APPENDIX C. PROTOTYPE IMPLEMENTATION IN INDUSTRIAL ENVIRONMENT C.1 INTRODUCTION C.1.1 Unix philosophy C.1.2 Underlying protocol C.1.3 Implementation language C.2 PREDICATES C.2.1 Time restrictions elimination by delegation C.2.2 Introduced notation C.2.3 Developed mechanism C.3 MODEL OF RMS DATABASE v

6 C.3.1 ERD Entity Relationship Diagram C.3.2 Description C.4 COMPONENTS OF PROTOTYPE C.4.1 Database C.4.2 Main scripts C.4.3 Repository of repair procedures C.4.4 Repository of repair actions C.4.5 Repair API C.4.6 Configuration of system C.4.7 Log of system C.5 SUMMARY APPENDIX D. PERL SOURCE CODE EXEMPLARY REPAIR PROCEDURE APPENDIX E. SYMBOLS AND ABBREVIATIONS vi

7 List of Figures Figure 2.1 Solution to problem 1: Changing scheduler table Figure 2.2 Solution to problem 2: Bringing DBMS to working status Figure 2.3 Solution to problem 3: Optimizing unoptimized database structures Figure 2.4 Solution to problem 4: Forcing emergency replication of time-critical data Figure 3.1 Scope of ARM method Figure 3.2 Components of ARM method Figure 3.3 Exemplary containers tree Figure 4.1 Division Procedure-Action-Step Figure 4.2 Exemplary connections tree Figure 4.3 Architecture of RMS Figure 4.4 Architecture of Repair API Figure 6.1 Data flow related to case 1: Changing scheduler table Figure 6.2 Data flow related to case 2: Bringing DBMS to working status Figure 6.3 Data flow related to case 3: Optimizing unoptimized database structures Figure 6.4 Data flow related to case 4: Forcing emergency replication of time-critical data Rysunek A.1 Kontekst metody ARM Rysunek A.2 Komponenty metody ARM Figure C.1 Entity Relationship Diagram of RMS database Figure C.2 Implementation of RMS vii

8 List of Tables Table 4.1 Template of a typical repair procedure Table 6.1 Case 1: Summary and results of experiments Table 6.2 Case 2: Summary and results of experiments Table 6.3 Case 3: Summary and results of experiments Table 6.4 Case 4: Summary and results of experiments Table 6.5 Summary of quantitative results Table 7.1 Results of assessment of ARM Tabela A.1 Zestawienie zysków na wydajności Tabela A.2 Rezultat oceny metody ARM Table B.1 Elements of Z notation viii

9 Acknowledgements I would like to thank Professor Janusz Górski, my academic supervisor, for his invaluable support and advice. I would like to thank the people who supported the creation of this thesis and contributed to development and assessment of the software tool and to realization of the experiments: Olaf Jendryssek, Piotr Butkiewicz, Paweł Kudzia, and Helpdesk Team of the Lufthansa Systems Poland Company. Marek Kamiński ix

10 Chapter 1. Introduction 1.1 Role and importance of monitoring and repair The current world is characterized by a very fast and progressive informatization of almost all domains of our lives. Number of IT systems and new functionalities they provide increase dynamically. Almost every system requires a kind of supervision and control. It results from the fact that hundred percent faultless and reliable systems do not exist. Complex IT systems are built of components that interact with one another and utilize various kinds of resources. Problems in their functioning often stem from running the systems continuously and performing changes and updates on the fly. It includes underestimating essential parameters of the components or resources, errors introduced during maintenance and other factors. In many cases, the types of reoccurring problems are well recognized and procedures of solving such problems are known. Because detecting a problem is the starting point to undertake an attempt of solving it, the IT systems and their elements are subjected to monitoring. Monitoring involves taking periodical measurements of the state of a system or its elements and applying some correctness criteria to distinguish a faulty state. The task of monitoring can be conducted manually, if the number of monitored objects is small. When this number is high, the manual monitoring becomes impossible without bearing too high cost. This is why various monitoring systems and mechanisms, automating the task of monitoring, are being used. Many such systems were already proposed (see e.g. [Radič, Ńkiljan 2004] for an overview). In general, such systems allow to [Albrand 2005]: assess if the monitored system works according to its specification, achieving its business goals, assess if the QoS requirements (reliability, availability, performance, etc.) are met, make the processes of checking a state of the system parameters and comparing them to predefined patterns mode efficient, detect potential anomalies in functioning of the system or even predict the future ones, provide the up-to-date (real-time) information about the state of the monitored system. Solving a detected problem is called a repair. It involves corrective steps aimed to remove the symptoms and causes of a detected problem. These steps can be undertaken in either a manual or an automated manner. The need for monitoring the systems and the high costs of their repairs resulted in establishing the market of IT monitoring and repair services. These services are in the scope of so called helpdesk. They include remote monitoring and technical support and assistance in maintenance and running the IT system, 24 hours a day and 365 days a year. 1.2 Current state of art Numerous monitoring ideas, methods and systems have been developed over the last years. Great deal of them focus on monitoring the selective functional aspects of networks, hosts and applications. In addition, general purpose monitoring systems (see e.g. [Sakellariou, Zanikolas 2005]) that allow to monitor almost any parameter of an IT system have been proposed. In such systems the scope of monitoring can be controlled by parametrisation and/or extendability (plug-ins). The problem of automatic repairs is more complex and is still the area of active research. Repairs are more difficult to automate and often involve manual and time-consuming interventions of administrators of the monitored systems. Nevertheless, repairs of some selected classes of problems have been already successfully automated. These include: application exception handling (e.g. java exceptions), limiting time of operations (e.g. RPC timeouts), purposeful redundancy (e.g. hard disks mirroring or back-up elements), 2

11 retrying failed actions finite number of times. If some components of systems are adequately equipped with the above mechanisms, they become more resistant to problems. Another mechanism in use is event handlers. The handler involves execution of a remote command (commands). An event is an undesired monitoring result and the associated handler provides an adequate repair action [Barth 2008; Josephsen 2007; Turnbull 2006; ZabbixSIA 2008]. Some solutions [Bauer 2009; Strejcek 2009] focus on automation of administrative tasks and do not integrate with monitoring. Other approaches [Gerlan, Schmerl 2002; Gerlan et al. 2003; Gerlan et al. 2002] focus on describing the architecture to facilitate automated repair. Efforts were made to integrate NAGIOS Monitoring System [Barth 2008; Josephsen 2007; Turnbull 2006] with CFengine system (in particular, regarding network problems [Retkowski 2006]). Some other works show directions the monitoring and healing (repairing) can go in a longer perspective [Pervilä 2007]. 1.3 Rationale This thesis takes a pragmatic view and is underpinned by observations of industrial reality. They led to the conclusion that in many cases, interventions being conducted manually by administrators of the monitored systems are repeatable. As they are repeatable, the added value resulting from their automation increases. As they are triggered by unacceptable monitoring results, it makes sense to integrate them with the monitoring tasks. The current industrial situation taken as the starting point of this thesis can be characterized as follows: the systems needing monitoring and repair are already designed and running, they do not have healing-oriented architecture, their administrators may only use a narrow set of acceptable and proven, tools and solutions, the monitoring task has already been automated and implemented to a satisfactory level, it is hard and error prone to implement complex repairs using only a set of simple commands performed mainly manually. 1.4 Thesis proposition The main goal of this thesis is to propose a method allowing automated identification and solving of chosen classes of problems occurring in industrial IT systems. The thesis proposition is as follows: The proposed method provides effective and efficient support for the monitoring and repair processes. 1.5 Scope of research The scope of research includes: development of a general model providing for representation of monitored objects and their states, selection of representative classes of problems occurring in industrial IT systems, development of a language framework for specification of repair processes, using this framework to specify repairs solving chosen problems, development of a method of automated execution of repairs, development of an architecture integrating monitoring and repair processes, demonstrating the feasibility of the method and architecture by a prototype implementation, validation of the proposed solutions. 3

12 1.6 Thesis structure The chapters of the thesis are organized in the following way: Chapter Two provides an overview of the monitoring process. It lists and classifies the monitoring mechanisms used nowadays. Then it introduces a generic model of the monitoring process. It discusses the problems and solutions related to the repair of IT systems. In particular, it identifies repeatable elements of repairs as candidates for automation. Finally, it presents four classes of problems that are representative to the industrial practice and presents related solving strategies. Chapter Three introduces the ARM method a method aimed to automate executions of repeatable repairs. The method consists of three components, each of which is introduced in a separate section. The language used to describe the method is a formal notation Z [Potter et al. 1991]. Chapter Four proposes an architecture of RMS system which forms the base for implementing the ARM method. It explains the structure, functioning and some key elements of the architecture. Chapter Five focuses on validation of the ARM method and RMS system. It introduces objectives of the assessment, the scope of measurements and the assessment criteria. Chapter Six presents case studies corresponding to the problems described in chapter two. The case studies were conducted in an industrial environment and served as the source of data for the validation process defined in chapter five. Chapter Seven contains conclusions drawn from the research. This thesis is supplemented with five appendixes: Appendix A contains an extended abstract of this dissertation, written in Polish. Appendix B contains a short introduction to Z notation, necessary to understand ARM models. Appendix C describes a prototype implementation of RMS in an industrial environment. The chapter explains and justifies basic design decisions and the technical means used in implementation. Appendix D contains a source code of an exemplary repair procedure. Appendix E Symbols and abbreviations. 4

13 Chapter 2. Monitoring and repair processes 2.1 A monitoring process Monitoring is a general term and it can be defined as: systematic and purposeful observation [Bartle 2006] Monitoring can refer to different objects (e.g. monitoring of Members of Parliament [Kamiński 2005]). However, in this thesis the scope of monitoring is restricted to IT systems and various aspects of their functioning. The IT systems monitoring is primarily intended to identify what has gone, or is about to go, wrong. It can be defined as a combination of four activities [Sellens 2000]: data acquisition collecting or generating the data, data storage logging the data, data processing analysis, comparison, evaluation, data presentation reporting and exception alerting Common view on the monitored reality In practice, monitoring is often implemented by a helpdesk (call center). This call center is intended to provide users with assistance in solving technical issues [Blokdijk 2008a]. The world of IT systems monitored by a helpdesk can be generally characterized as follows: company providing the helpdesk service has many customers, the customers own IT systems that provide services needed by their business processes, the customers and their systems are geographically (and widely) distributed, the systems consist of various kinds of hardware and software resources, they are accessible via external and/or internal networks Monitoring mechanisms Numerous monitoring mechanisms and numerous monitoring systems are presently in use [Cottrell 2010]. There are monitoring systems that monitor separated, and often independent IT systems and their monitoring function is centralized [Radič, Ńkiljan 2004] we will call them traditional monitoring systems. Those systems collect (active monitoring) or receive (passive monitoring) the results in a designated central place, for further processing and analysis. Subsequent actions (e.g. actions related to presenting the processed information or informing proper persons) are undertaken based on the outcomes of the analyses. The limits of values of the monitored parameters are set by system operators (or default values are used instead). The monitoring system periodically generates reports providing the information on the state of monitored objects. The reports are presented in a comprehensible form on a central interface of the monitoring system. It is accessible to the authorised persons (for instance, via a web page). If a problem with a monitored object is being detected, the administrator of this object can see it (on the relevant web page) or is notified (for instance, via a SMS) and is obliged to undertake an action solving the problem [Barth 2008]. Some systems are capable to analyse the historical data to detect trends of monitored parameters. They can utilize sophisticated methods of artificial intelligence to predict future states of the systems [Savage 1999]. Detecting that a certain parameter is likely to exceed its critical value, the administrator may undertake a preventive action before the problem actually instantiates. Examples of traditional monitoring systems are: Nagios, Zenoss, Zabbix, Pandora, Big Brother, Big Sister, HP OpenView, NeDi [Aslett 2006; Jones 2004; Cacheda et al. 2007]. 5

14 In addition to traditional centralized systems a broad class of distributed monitoring systems exists. Those systems mainly aim to monitor grid or cluster structures [Ceccanti, Panzieri 2004; Hongbin et al. 2005]. In many cases they have a grid based architecture. They are equipped with specialized software to allow the users to share various kinds of resources (e.g. computational resources or disk resources). This provides for balancing usage of resources, for coordinating parallel work of all nodes, and for saving the results of processing in a shared storage. The advantage of a grid is its ability to consume computational power of temporarily unused machines (e.g. computational power of workstations left idle at night) [Mosorow 2007]. Well-known example of a computational grid system is SETI@home project. Distributed monitoring systems often provide additional functions, not directly related to monitoring. For instance, they may provide information on availability of grid resources. They register new grid nodes, along with services they provide, and they answer the queries related to the services [Byrom et al. 2004]. Distributed monitoring usually involves hierarchical filtering of the information. The nodes transfer gathered information upwards the hierarchy with possible filtering and aggregation [Groothuyse 2004; Cosnard et al. 1995]. The monitoring logic of such systems is also distributed. Examples of distributed monitoring systems include: Ganglia, Hawkeye, MonALISA, NWS (Network Weather Service), RGMA (Relational Grid Monitoring Architecture), GridRM, Merkury [Sakellariou, Zanikolas 2005] A monitoring process model After analysing common features of monitoring systems we propose the following definition of the monitoring task. Monitoring is a process of examining subsequent states of a group of objects against defined criteria and informing about detected anomalies. This definition abstracts from the physical nature of the monitored objects. It may be a remote machine (computer or network device) running an operating system, a service, or it may be any resource of the machine. Each monitored object assumes mutually exclusive states, and each state assumes a value from a specific domain. Some of these values indicate that the state is correct, while the other values tell us that the monitored object has encountered a problem that must be solved. We can assign priorities to the undesired states to distinguish between levels of their relevance. This classification of states is usually based on so called threshold values. Some of the undesired states can then be classified as CRITICAL, whereas others are interpreted as WARNINGS. To distinguish between the monitored objects we need some naming system. We represent such objects as variables assuming that the value of a variable represents the current state of the associated object. We also assume that the mechanism which binds variables with the real objects subjected to monitoring is available. With these assumptions, monitoring becomes a process of continuous checking (observing) values of multiple variables against the specific set of monitoring criteria. The monitored world is often structured hierarchically, and the monitored objects are the leaves (e.g. hosts storing replication hubs, storing replication channels, storing replication lines [Kamiński 2007c]) of the hierarchy. It is convenient, therefore, to use qualified names for the variables representing such objects, like for file names of a hard disk drive, the Internet names of the DNS protocol, or the MIB tree of the SNMP protocol. For example: eclipse.oracle.testdb.fs.usage represents usage of the main filesystem of machine eclipse, belonging to DBMS oracle, and identified by SID TESTDB, eclipse.cpu.temperature represents current temperature of main processor of the same machine, eclipse.cpu.usage represents usage of the central processor. Values of the variables representing monitored objects can change in time. A qualified name of a variable augmented by a timestamp indicates precisely a single measurement, informing where and when it was taken. Frequency of changes depends on the frequency of retrieving and processing data from the monitored objects and on the stability of their states. 6

15 Because sometimes the measurements (especially these indicating problems) carry extra information, it is convenient to allow optional details that can be passed together with the values of the monitored variables. Monitoring involves checking the monitoring criteria. These criteria can be formalized as predicates. Simple examples of such criteria are shown below: eclipse.oracle.${anydatabase}.fs.usage > 95% this predicate becomes true if any oracle database of machine eclipse is running out of space of its main filesystem, (${hostx}.cpu.temperature > 70ºC) AND (${hostx}.cpu.usage = 100%) this predicate becomes true if the main processor of any machine is overheating and it is 100% busy at the moment. In the above examples, the notation: ${...} indicates a part of the name of a variable that may be substituted by any valid string. Detecting that a given predicate becomes true tells us that the monitored objects associated with the corresponding variables have a problem, and the time and location where the problem occurs are precisely indicated. 2.2 A repair process Problems of IT systems reappear and therefore it is convenient to represent the corresponding repair actions as repair procedures that can be initiated if needed. Further in this dissertation, the repair procedures are referred to as procedures. A procedure describes a solution. A repair is an activity of using a procedure. To define repair actions as a procedure, we need to express them formally, precisely, and in a general manner. All unnecessary details should be abstracted away. Other details, like those pointing to the location of the problem and to its wider context (if needed), could be passed to the procedure as its input parameters A repair process model We propose the following model for describing repair procedures that drive repair processes Standard control flow instructions Standard flow control instructions [Daviduck]: loops iterations of a fragment of an algorithm, conditions decision points forks in an algorithm indicating the path it conditionally goes, sequences instructions executed in a consecutive manner, not changing a path of the flow control. In addition, as sequences, we distinguish: interactions with the outer world points where an algorithm sends output and receives input, calls of internal routines executions of procedures or functions being a part of an algorithm, calls of external routines executions of procedures or functions defined outside an algorithm External routines steps of repair The last group of the instructions distinguished above represents the smallest units of repair activities. They are the actual corrective steps. We assume they aim at fixing or changing those components of a monitored system that affected its state in an adverse way. They are routines that are external to main body of the repair procedure. They are distinguished because they are usually executed on machines owning broken components, being a subject of a particular repair. 7

16 Let us consider a simple example of a system including two filesystems, residing on different machines (A and B). To solve a particular problem we may need to enlarge the capacity of each filesystem. That is, we need two actions, an action enlarging the filesystem on the machine A, and an action doing the same on the machine B. The actions of the repair are, therefore, naturally distributed. The remaining part of the logic of the repair procedure can be considered as location-independent. This observation forms the cornerstone of the method presented in this dissertation: the repair procedures may be started and run centrally, connecting to the remote machines only when necessary. External routine is a well-known concept of programming languages. Conceptually we can consider it as a black box performing the required action in the required place. Further in this dissertation, the external routines are referred to as steps. 2.3 Overview of problems and their solutions In this section we present a range of typical problems encountered in industrial IT systems. Each problem is described in a general way and represents a broader class of problems that occur in practice Problem 1: Wrong entries in scheduler table Many operating systems offer scheduler programs which execute tasks regularly at certain times (e.g. at night or during weekends). Such tasks are commonly used to automate system maintenance and administration but they can also be used for other purposes, such as connecting to the Internet and downloading s. A well-known and a popular scheduler program, existing in many UNIX distributions, is called CRON [Peek, Powers 2002]. Definitions of the tasks and exact times of their planned executions are stored in the entries of so called scheduler table. In case of CRON it is called CRONTAB. Examples of unwanted events are: A particular scheduler table or its variations are deployed in enormous number of machines. It turns out that some of the planned tasks are totally wrong or have serious defects (it could be the result of previously introduced changes or other factors) Problem 2: Backup failures preventing DBMS from working Businesses and other organizations become increasingly reliant upon OLTP systems [DRJ]. Online and batch processing of databases requires full integrity and recoverability. It is ensured by using special files often called redo log files (Oracle DBMS [Whalen 2006]) or journal files (Ingres DBMS [Flower 2006]). The associated mechanism follows the idea that as transactions are processed, the DBMS writes log (or journal) records to the file recording the changes in a consistent format. In case of e.g. a hardware failure, a forward recovery utility is used to recover from the starting backup copy of the entire database. This means that at least two kinds of backups exist in current OLTP systems: A full backup a copy of the entire database at one point in time. It contains all data and relevant journal files. The full backup requires a large amount of storage space, but the database can be restored from this kind of backup relatively quickly. An incremental backup a backup that is still active, even when the users or batch processes are working. It records each change of data since the last full backup. The incremental backup requires a small amount of storage space, but more time is needed to restore the database from this kind of backup. After a failure of the media storing the current datafiles, full recovery of the database requires the last full backup of the database and the related incremental backup. Files belonging to full backups and to journal files usually reside on dedicated filesystems. As such filesystems have limited capacity, the stored files are moved to other storages (removed from the 8

17 filesystems) when they are no longer needed. Usually, old backups are removed, after newer ones are created. A problem occurs when the storage resource runs out of its free space. A common scenario leading to this problem is as follows: A procedure performing routine full backup of a production database referred to as checkpoint terminates abnormally due to random factors. As a consequence, the journal files supposed to reside in the journal-dedicated filesystem for a short period of time are not removed. This increases the probability of overflowing that filesystem. The journal-dedicated filesystem runs out of its free space indeed, preventing DBMS from creating new journal records. The DBMS stops (transactions suspend), preventing the users and batch processes from working. In addition, some other unwanted events enlarging only the problem may occur: Abnormal termination of checkpoint prevents it from removing corrupted files from the checkpoint-dedicated filesystem, which increases the probability of overflowing that filesystem. The DBMS archiver process creating new journal records terminates. The symptoms detected by a monitoring system, and indicating this kind of problem, include: 1) The journal-dedicated filesystem is 100% full. 2) The archiver process is down Problem 3: Failures during optimizing physical structures of production database Due to continuous changes, the OLTP database systems need a regular maintenance to provide for a good response time and other performance parameters. This goal is usually achieved in the following way: The scheduler table has the reference to the task optimizing physical structure of the production database of a customer. The task is executed regularly at times suitable for the customer. It produces a log file with information which database structures have been optimized successfully. The following factors indicate the condition of the OLTP database, concerning its performance: Database storage structures underlying tables stored in the database [García-Molina et al. 2000]. Indexes created for these tables. Collections of statistics, controlling for instance the behaviour of Query Optimizer Facilities. The process of optimizing the performance of the whole database focuses on improving those three factors for each database table [CA; Kaelin 2004; PostgreSQL 2008]. The problem occurs when the above mentioned log file contains errors Problem 4: Unacceptable delay in replication of time-critical data The value of some data is time dependent. Examples include data related to stock markets, flight planning, flight safety, medical treatment, and so on. In contemporary OLTP systems, time-critical data usually follow the following path: Time-critical data is inserted to a centralized or a distributed database. It is transported over the network to other databases using one of many replication techniques [Khadzynov, Maksymiuk 2006; McKinstry; HVR]. The consumer of the data detects their availability and starts using the data. Unwanted situation may happen when the replica fails. It is not certain that repairing the replica can be conducted fast enough to keep its downtime unnoticeable to the customer. Thus, it is necessary to 9

18 make sure that at least the time-critical data is available at the remote site to avoid an unacceptable delay. 2.4 Generic solutions to problems The following subsections present solutions to the four described problems. Common flowchart notation has been used to provide for precision and to reduce possible misunderstandings, omissions and ambiguities. It also makes the ground for converting human-executable tasks into machineexecutable code Solution to problem 1: Changing scheduler table The solution to this problem is very simple. The algorithm is shown in Figure 2.1: START Dump the original crontab file Apply changes Dump a new crontab file STOP Figure 2.1 Solution to problem 1: Changing scheduler table Solution to problem 2: Bringing DBMS to working status Typical solution to this problem involves at most eight steps (rectangles with a thin border line) and one internal routine (rectangle with a thick border line). The internal routine determines a strategy of solving the problem. If the algorithm cannot solve the problem, then it rolls performed changes back and stops. The algorithm is shown in Figure 2.2 (rhombi represent points where it waits for receiving input from the outer world): START Was it successful? No Make list of inactive journal files Find free disk space Move inactive journal files to temporary locations Yes Startup database archiver process Find solution Apply it Was it successful? Yes No What shall I do? Finish What shall I do? No Was it found? Remove broken checkpoints Rollback moving of inactive journal files Finish Yes Perform checkpoint Retry STOP Finish What shall I do? What shall I do? Finish Ask user for another solution Was it successful? No Ask user for the last resort solution Retrieve data from user Yes Remove temporary files STOP Figure 2.2 Solution to problem 2: Bringing DBMS to working status Solution to problem 3: Optimizing unoptimized database structures Typical solution to this problem involves one step (REEXECUTE THIS task) which is executed repeatedly with different parameters. Solving the problem usually requires conducting some auxiliary tasks, like suspending or unsuspending related replications, to prevent the problem from reoccurring. Those 10

19 tasks, despite being helpful, do not relate to the problem solution directly. The algorithm describing this solution is shown in Figure 2.3: START Create (for each unoptimized table) ordered list of tasks to reexecute Show unempty lists and their contents STOP Finish What shall I do? Maintain potentially interfering processes Find and show related replications Ask user to point chosen tables Reexecute all tasks for some of the tables Ask user to point some replications Ask user to point some replications Suspend them Unsuspend them Is there any TABLE without reexecuted tasks? No Are all lists empty? No Yes Yes Are any replications (suspended in this repair) still suspended? No No Has THIS table on its list any next TASK to reexecute? Yes Reexecute THIS task Yes Unsuspend them Remove task from the list Yes Was reexecution successful? STOP No Figure 2.3 Solution to problem 3: Optimizing unoptimized database structures Solution to problem 4: Forcing emergency replication of time-critical data An example solution involves watching the status of a replication. After time-critical data is inserted to a database being the replication source, the supervising process (commonly called watchdog ) starts. Its role is to intervene when the time-critical data is not available at the target databases. The algorithm describing this solution is shown in Figure 2.4: 11

20 START Prepare list of replication lines (SOURCE and TARGETS) and a current replication status Export the time-critical data from replication SOURCE to files Wait configured amount of time Prepare list of replication lines (SOURCE and TARGETS) and a current replication status Wait entered amount of time Remove from the list TARGETS having the data already Import the time-critical data to target #N Is the list empty? Yes No Import the time-critical data to target #2 Ask user how long Wait What shall I do? Finish STOP Import the time-critical data to target #1 Ask user to point targets Insert the data to targets immediately Figure 2.4 Solution to problem 4: Forcing emergency replication of time-critical data 2.5 Summary Problems detected during monitoring can be modelled by predicates defined over the set of the variables representing the monitored objects. This provides a base for the generic model of the monitoring process which abstracts from the physical nature of the monitored objects. The model provides for specification of the logical nature of the problem, precise identification of the object within the monitored (hierarchical) space and localization of the problem in time. In most cases repairs solving the problems are deterministic algorithms and therefore can be specified by conventional means, e.g. flowcharts. Expressing the solutions in a formal manner is a starting point to automate the process of repair. The algorithms involve human-machine interactions when a machine is unable to make a proper decision. The logic of generic elements of repairs (steps) may be delegated to a lower abstraction level. 12