How to minimize upgrade downtime during SAP ERP 6.0 upgrade projects

Managing SAP ERP 6.0 Upgrade Projects
Chapter 5: Executing an Upgrade Project

About 54% of organizations using SAP say that minimizing downtime is an important part of planning and executing SAP ERP upgrade projects. But what factors influence downtime? What can be done to minimize downtime in an upgrade project? Downtime management means controlling the time the system is unavailable, so learn how to reduce downtime-related costs, and find out how the total amount of downtime can affect other decisions in an upgrade project.

Managing SAP ERP 6.0 Upgrade Projects, Ch. 5

Table of contents:
Managing system landscapes during SAP ERP 6.0 upgrades
How to minimize upgrade downtime during an SAP upgrade project

5.2 Downtime Minimization

This section discusses system downtime in the context of an SAP upgrade project. Downtime is probably the biggest challenge of an upgrade and a crucial topic because it is the time when the system cannot be used by the business. In an SAP customer feedback survey, 54% of organizations identified downtime minimization as being an important consideration in both planning and executing an upgrade project.

Managing downtime is not just about controlling the time the system is unavailable, but also about controlling the costs incurred during downtime. Furthermore, costs are not linear: for longer downtime costs can increase exponentially.

This excerpt from Managing SAP ERP 6.0 Upgrade Projects by Martin Riedel is reprinted here with permission from SAP Press; Copyright 2009. Download a pdf of this chapter.

The total amount of downtime permitted by the business has an effect on many of the other decisions you make when planning an upgrade project. Therefore, you must carefully determine the maximum downtime that will be available and precisely detail the upgrade tasks that have to be performed during that time.

In this section, we will look at the factors that influence downtime and what you can do to tackle the challenge of downtime within your upgrade project. You will learn how to decide on the upgrade strategy to use and also find details about SAP tools you can use to help minimize downtime. We will also provide you with recommendations for how to reduce downtime costs.

5.2.1 Definitions: Downtime, Uptime, Runtime

This section provides an overview of what the terms downtime, uptime, and runtime mean in the context of an upgrade project. Figure 5.6 shows the business and technical perspectives of the downtime and uptime phases.

Downtime (both technical and business), uptime, and runtime are described in Table 5.1.

Business Uptime Business Downtime In any system, you can further divide downtime into planned and unplanned downtime. Unplanned downtime concerns items over which you have little direct control such as hardware, or operating system failures, as well as human error.

Planned downtime, on the other hand, is very much under your control, in particular in terms of when it takes place. You are required to plan downtime for system and infrastructure maintenance and for the implementation of patches, upgrades, and changes to transports. Planned downtime can be minimized through the following:

• Scalable components that enable rolling maintenance
• Improved upgrade and patch processes
• Proven software lifecycle management and propagation engines (e.g., the transport management system or the change management service)

5.2.2 Why is Downtime Necessary?

One of the advantages of SAP's technology is that it allows customers to adapt, extend, and modify SAP software, and these extensions will be kept and adjusted to the new release during the upgrade process. Furthermore, most of the required processing steps that are part of the upgrade can be performed during system uptime.

Downtime is necessary whenever live, running transactions have to be replaced by new functionality and there is a potential risk for data inconsistency, for example due to a change in the processing logic, or a change to the data model or structure. You must prepare users for business downtime and make them aware of the need for both technical downtime and business downtime.

5.2.3 Downtime Facts and Figures

SAP has analyzed the average downtime (business downtime and technical downtime) for customers upgrading to SAP ERP 6.0.

The average business downtime was calculated separately, according to the SAP source release. Results showed 34 hours (for source release SAP R/3 Enterprise) and 48 hours (for source release SAP R/3 4.6C). For all upgrades to SAP ERP 6.0, the average technical downtime was 7.2 hours.

The chart shown in Figure 5.7 illustrates the minimum possible technical downtime for 110 customer upgrades to SAP ERP 6.0 SR3 where the downtime-minimized strategy was used.

5.2.4 Choosing an Upgrade Strategy

With the system switch technology, most upgrade activities are moved into uptime. When upgrading with the system switch upgrade procedure, SAP provides you with two upgrade strategies: the downtime-minimized strategy and the resource-minimized strategy. You must decide which strategy to use as determined by the requirements of your organization. Two factors are key to this decision:

• Maximum downtime permitted by your organization
• System resources available

The downtime and the consumption of system resources depend on the interaction of several parameters you can set for the upgrade. To optimize the duration of downtime and the consumption of system resources, parameter settings are grouped into preconfiguration modes. Instead of setting the parameters manually, you choose the preconfiguration mode that suits your system resource situation. You select the preconfiguration mode in the upgrade GUI during the upgrade procedure. By setting the preconfiguration mode, you can choose either a resource-minimized or a downtime-minimized upgrade strategy. For more details, see the SAP ERP 6.0 upgrade guides. Table 5.2 shows an overview of the three available preconfiguration modes.

SAP recommends using the downtime-minimized strategy for the majority of upgrades. The additional resources needed should be available because SAP ERP 6.0 technically requires them.

A downtime-minimized strategy results in shorter downtime, but has a higher demand on system resources for the parallel operation of both a production and shadow system.

The resource-minimized strategy means no additional system resources are needed during the upgrade, but results in longer downtime and an offline backup is required after the upgrade.

Manual selection Manual selection is also possible for setting the preconfiguration mode parameters. This gives you the flexibility to control all parameters.

5.2.5 Elements of Business Downtime During an Upgrade

The technical upgrade process involves a number of phases, including SAPup running the SAP upgrade application SAPup. However, business downtime is a period of time during the upgrade process that is shorter than the overall upgrade process duration, as you can see in Figure 5.8.

There are various elements of business downtime during the upgrade process, as follows:

• Business ramp-down and ramp-up, when productive use of the system is stopped and restarted (locking/unlocking users, rescheduling jobs, shutdown of interfaces)
• Post-upgrade transports and manual adjustments
• Business validation and acceptance testing, when the system is running but not yet available for productive use
• Pre- and post-upgrade system backups

5.2.6 Factors that Influence Upgrade Runtime and Downtime Duration

Runtime and downtime during an upgrade depend on many different factors, which can be broadly classified as hardware, software, configuration, and strategy. Downtime factors Downtime is influenced by the following factors:

• Hardware
  • Number and type of CPUs for the central application and database servers
  • Type and performance of the storage medium (I/O throughput)
• Software
  • Start release and target release
  • Version of the upgrade tools
• Configuration
  • Upgrade parameterization (e.g., the number of parallel processes, instance profile parameters, and database parameters)
  • Number of clients (e.g., 100, 010, and 200)
  • Number of installed languages
• Strategy
  • Upgrade strategy: downtime-minimized or resource-minimized
  • Usage of Incremental Conversion (ICNV), Customer-Based Upgrade (CBU), or Incremental Upgrade & Unicode Conversion (IUUC)

Runtime is influenced by the following factors:

• Hardware
  • Number and type of CPUs for the application and database servers
  • Type and performance of the storage medium (I/O throughput)
• Software
  • Start release and target release
  • Number of included support packages
  • Binding parts of an enhancement package into the upgrade process, and the number of technical usages selected
  • Number of modifications on standard SAP objects
  • Number of custom objects
  • Version of the upgrade tools
• Configuration
  • Upgrade parameterization (e.g., the number of parallel processes, instance profile parameters, and database parameters)
  • Import destination time
• Strategy
  • Usage of ICNV

In SAP ERP 6.0, SAP has improved the update process to allow for a onestep upgrade to SAP ERP 6.0:

• Direct upgrade path from R/3 or mySAP ERP 2004 to SAP ERP 6.0 with the option to include the latest parts of an SAP enhancement package.
• Cost and downtime reduction with no additional implementation steps necessary for enhancement packages if they are embedded with the upgrade.

5.2.7 Incremental Conversion (ICNV)

The primary goal of ICNV is to reduce downtime during an upgrade.

ICNV is a configurable process that can be can be stopped and restarted. It allows for the conversion of large tables during system uptime. Furthermore, you can select the specific tables to be processed by ICNV. ICNV requires additional resource usage of the database, as well as a sufficient number of background work processes. It is also recommended that you execute ICNV as early as possible. This requires more careful planning.

The procedure is designed to not be overly complicated and is fully integrated into the upgrade process. The conversion process is executed during uptime. The database load is expected to be higher during this process and therefore, it is possible to define exclusion times during which no ICNV processes are running.

The conversion process calculates the estimated end of the process. This information helps you plan the upgrade timings accurately. Large tables are converted during uptime but the switch to the new structure is made only during downtime (in PARCONV).

Configuration

ICNV offers several features to configure the incremental conversion process:

• Batch hosts can be specified.
• The number of running batch processes is adjustable.
• Exclusion times for processing can be specified for each table. (This enables you to run conversion jobs at times with relatively low table I/O.)
• The log files of the conversion processes for each table can be accessed.

After selecting the tables, you can choose to be guided through the necessary steps by the ICNV Assistant (see Figure 5.9). For the upgrade scenario, two steps must be started manually:

• Initialization
  • Extension by a flag field
  • Building an index on the flag field
  • Creation of an update and deletion of triggers
  • Replacement of the table by a view and renaming of the table
• Start of the data transfer

The remaining steps (transfer, switch, and delete entry in ICNV) are then performed by SAPup.

5.2.8 Customer-Based Upgrade (CBU)

A customer-based upgrade (CBU) is a special upgrade procedure that, compared with the standard procedure, can significantly reduce system downtime when you upgrade a production system.

SAP recommends a CBU when the time needed to implement customer transports means that the upgrade cannot be performed within the upgrade downtime window that is available.

CBU can be used to create and perform optimized, customer-specific upgrades.

A customer-based upgrade is intended for systems with extensive customer developments, a large volume of delta transports, and where there are high demands on system availability. The goal of a CBU is a significant reduction of business downtime with the inclusion of all requested package contents. This can be achieved through the creation of repository export data at a customer site from an upgraded copy of the production system. During a CBU, a customer-specific export of the substitution set for the upgrade takes place using a special upgrade procedure at your site. This individual export and the special upgrade tools for the CBU remove the need for the following actions when you upgrade the production system:

• Import of customer transports after the upgrade (with the exception of Customizing transports)
• Modification adjustment (transaction SPAU)
• ABAP load generation after the upgrade

However, as with many tools, there are prerequisites to be met before you can implement a CBU successfully:

• You must create a copy of the production system.
• After you have made the first copy of the production system, you must freeze the repository of the production system. Any transports and corrections you make in the production system after you make this copy are lost when you perform the CBU upgrade and need to be repeated manually later.
• The hardware you use to upgrade the second copy must be similar to the hardware of the production system. This is required for the platform-specific load generation and allows for a reliable estimate of the production upgrade downtime.
• The CBU is a customer-specific upgrade approach changing the standard upgrade procedure in some aspects. Therefore, it is strongly recommended to involve experienced consultants certified for this methodology, to minimize risks.

The advantages offered by a CBU can be significant. In general, a CBU allows you to plan the upgrade of the production system more precisely. You can also analyze and, if necessary, avoid or optimize critical or longrunning database modifications.

5.2.9 Unicode Conversion

Unicode is state-of-the-art technology for technical language support for all SAP solutions. It is the only future solution to support complex systems and system landscapes in a global, multinational, and multilingual environment. Multiple-Display Multiple-Processing (MDMP) configuration is no longer supported with SAP ERP 6.0.

SAP supports four paths to convert your system to Unicode depending on the version of SAP R/3 from which you are upgrading:

• Unicode conversion independent of an upgrade project (if upgrading from 4.7 or higher).
• Independent Unicode conversion before or after the upgrade (if upgrading from 4.7 or higher).
• Combined Upgrade & Unicode Conversion (if upgrading from SAP R/3 4.6C or higher).
• Twin Upgrade & Unicode Conversion (if upgrading from lower than 4.6C).

Section 3.1.4 "Unicode," of Chapter 3, Planning an Upgrade Project, describes the different approaches to a combination of upgrade and Unicode conversion. SAP provides guides that describe the detailed processes and procedures involved in Unicode conversion, with or without an upgrade. Extensive information on Unicode conversion is also available in the SAP NetWeaver Application Server Upgrade Guide by Bert Vanstechelman, Mark Maergerts, and Dirk Matthys (2nd edition, SAP PRESS 2007) and in Unicode in SAP Systems by Nils Bürckel, Alexander Davidenkoff, and Detlef Werner (SAP PRESS 2007).

In the context of an upgrade project, upgrading and converting to Unicode in the same project affects the runtime and downtime of the overall upgrade. It is therefore important to consider ways to minimize the runtime and downtime of the Unicode conversion as part of the upgrade process. All Unicode conversion paths follow the same basic steps during downtime:

1. Prepare the non-Unicode system.
   
2. Export the non-Unicode database and convert it to Unicode data.
   
3. Create a new Unicode database and import the Unicode data.
   
4. Perform post-conversion activities in the Unicode system.

Within any Unicode conversion (either independent or together with an upgrade), you can use the Near Zero Downtime approach for significant downtime reduction. This is described in the next section.

5.2.10 Near Zero Downtime

For customers with extremely high system availability requirements, SAP offers the Near Zero Downtime approach. It is a combined approach using both standard SAP upgrade tools and SAP migration tools and, in the case of a release upgrade, allows reducing business downtime to two to four hours.

With the Near Zero Downtime method, the upgrade is performed on a clone of the production system. During this upgrade, all changes to the production system are logged at the database level. After the completion of the upgrade on the clone, the real downtime begins. During this downtime, the data changes from the production system will be transferred to the clone. The data transfer is performed by the tool based on the Migration Workbench. The tool will also transform the data structure in case of changes in the data model between the old and the new release. After system validation, the clone takes over the role of the production system. See Figure 5.10 for an overview of the process.

The method can also be used for an upgrade combined with Unicode conversion. In this case, the business downtime can be reduced to five to seven hours, independent of the database size.

Compared to a standard upgrade, the Near Zero Downtime process has both advantages and disadvantages. The advantages of Near Zero Downtime are as follows:

• Greatly reduced downtime (approximately four hours), independent of the database size.
• Improved system availability.
• Reusability for the implementation of support packages, enhancement packages, operating system patches, and database patches.
• Late go/no-go decision and very fast reset of the system.
• The standard upgrade project remains unaffected.

The disadvantages of Near Zero Downtime are as follows:

• Additional project effort and a longer upgrade project (minimum of six months).
• Additional hardware requirements.
• Code freeze for four to six weeks prior to go-live.
• Restricted transport for one to two weeks prior to go-live.

5.2.11 Unicode Conversion Downtime

For Unicode conversion, you can follow various recommendations to minimize downtime during the conversion process.

The two main options for downtime reduction in Unicode conversion are to minimize the database size (e.g., through archiving or deletion of unused data) and to parallelize the conversion process. Downtime is highly dependent on hardware performance; therefore, hardware tuning (e.g., through additional CPUs and increasing I/O throughput) is recommended.

You can use SAP resources to get information on estimating downtime, for example see SAP Note 857081 "Unicode conversion: downtime estimate." This addresses expected downtime, potential bottlenecks, possible measures for improvement, how to analyze test results, and how to compare results of different migration projects.

The SAP Unicode Downtime Minimization service is recommended by SAP when you run or plan to run a Unicode conversion project with a large database and want to select the best option to minimize downtime.

The SAP Unicode Downtime Optimization service (which is run as a workshop) investigates all available downtime minimization options and aims to recommend the best one for your situation.

5.2.12 Best Practices -- Upgrade Tuning

This section details several best practices for minimizing downtime during an upgrade project. Upgrade tuning can minimize both upgrade runtime and downtime. In planning an upgrade, you can take steps to reduce the total upgrade time, but there is a trade-off between cost and time reduction. For example, you can invest in faster CPUs and better storage, use new backup tools, and implement automated testing tools to help with the go/no go decision, all of which can help minimize downtime but affect the overall cost of the upgrade.

You can carry out upgrade tuning actions within a standard upgrade (as described in the SAP ERP 6.0 upgrade guides). These will have an effect on the phases of the upgrade: prepare, upgrade uptime, upgrade downtime and follow-up activities. In general, the following will help reduce the amount of upgrade runtime and downtime:

• Using the latest upgrade software tools
• Using the latest software release DVDs (as available from SAP)
• Using the downtime-minimized strategy

However, you can take additional specific measures to reduce downtime:

• Include all required support packages into the upgrade: Downtime is reduced if support packages are included in the upgrade project rather than being imported after the technical upgrade. Also, it may be a requirement to include support packages in the upgrade.
• Include all required technical usages from enhancement packages into the upgrade, and using the SAP Enhancement Package Installer: This drastically reduces the duration of technical downtime. Embedding the SAP enhancement packages within the upgrade to SAP ERP 6.0 requires only one upgrade window and therefore only one period of downtime (embedding enhancement packages in the upgrade is only possible from SAP ERP SR3 on).
• Use transport(s) created for automatic modification adjustment (SPDD/SPAU).
• Use Incremental Conversion (for details, see Section 5.2.7).
• Use as many parallel upgrade processes as possible.
• Create a well thought out cutover plan.
• Consider a backup strategy.

You can also employ additional upgrade tuning measures such as hardware improvements, SAP tools and services, and cutover planning that are not described as part of a standard upgrade in the upgrade guides.

However, when considering to apply any of these measures, you should run at least three test upgrades. It is recommended that the first time you test the upgrade you should run a standard upgrade, following the procedures described in the SAP ERP upgrade guides. The results you get from this will give you a baseline. You can then run a "tuned" upgrade, using one or more of the tuning measures described in the text that follows and compare the results with those of the standard upgrade baseline. You should then run an additional tuned upgrade to again determine whether the tuning steps are effective. It is not worthwhile to pursue a tuned upgrade if the standard upgrade produces acceptable results in terms of downtime.

5.2.13 Testing

It is highly recommended that you perform as much testing as possible before finally embarking on the actual upgrade to determine the duration of the upgrade runtime. This is essential to your planning of the upgrade. There are techniques you can use to reduce the runtime and therefore you should perform pre-upgrade tests to discover how to tune the upgrade process.

For example, a test upgrade on a copy of the production system and a thorough analysis of the upgrade log files can identify many possibilities for effective manual tuning activities. Therefore, a highly recommended preparation for achieving an optimal reduction of production system downtime is to perform multiple upgrade tests, including manual tuning measures.

One of the main purposes of running test upgrades is to measure downtime and to find out if the upgrade can take place within the downtime window available to you. This section suggests ways to minimize downtime as much as possible if you are not able to achieve the desired downtime. For example, understanding and analyzing the factors that influence downtime will enable you to adapt your environment appropriately with the goal of minimizing both downtime and runtime.

Further downtime minimization -- beyond standard upgrade technology -- can also be achieved by using specific SAP services, such as the SAP Downtime Assessment Service and the Customer-based Upgrade service, as well as the Near Zero Downtime approach.

5.2.14 Splitting Downtime

SAP usually releases older versions of SAP ERP software on database and operating system versions higher than those used initially. This enables customers to run an SAP release on the latest database and operating system versions, if required. Thus, the maintenance periods of SAP releases can be prolonged. During the upgrade of an SAP/database/operating system combination, this feature may be used to split the system downtime, for example, into two different weekends:

1. Perform the upgrade of the database and operating system to the version required by the SAP ERP target release. It is recommended that you do this at least four weeks before the SAP ERP upgrade.
   
2. Upgrade to SAP ERP.

5.3 Training

This section contains recommendations and options for training should consider during an upgrade project. This can include training for the following:

• The administrator, to perform the technical upgrade.
• Developers, to handle modification adjustments, re-implementation of custom developments, and new development on the new release.
• The project team, to handle the upgrade project.
• Power users, to cover delta and new functionality in the new release and to perform delta Customizing.
• End-users, to cover delta and new functionality in the new release.

Within all phases of the project potential requirements for training exist. Relevant, targeted training is important to ensure the smooth running of the entire upgrade project and the adaptation to new processes and functionality. You should devote an appropriate portion of the upgrade project budget to training needs.

For upgrades from source releases SAP R/3 4.6C on, training efforts and costs typically amount to no more than 5 % of the project budget. For additional information, see the (unnumbered) Section "Cost and Effort Factors of an Upgrade Project," in Chapter 3, Planning an Upgrade Project.

You can download a pdf of this chapter. You can also visit SAP Press to purchase a copy of Managing SAP ERP 6.0 Upgrade Projects.

You'll find more downloadable excerpts from books by SAP experts in our SAP chapter download library.