Oracle E-Business Suite Online Patch Phases executing adop

Following description about Oracle E-Business Suite is high level and from documentation https://docs.oracle.com/cd/E26401_01/doc.122/e22954/T202991T531062.htm#5281339
for in depth and detail description refer it.


The online patching cycle phases:

• Prepare
• Apply
• Finalize
• Cutover
• Cleanup

Prepare phase:
Start a new online patching cycle, Prepares the environment for patching.

$ adop phase=prepare

Apply phase:

Applies the specified patches to the environment. Apply one or more patches to the patch edition.

$ adop phase=apply patches=123456,789101 workers=8

Finalize phase:

Performs any final steps required to make the system ready for cutover. Perform the final patching operations that can be executed while the application is still online.

$ adop phase=finalize

Cutover phase:

Shuts down application tier services, makes the patch edition the new run edition, and then restarts application tier services. This is the only phase that involves a brief downtime.

Perform the transition to the patched environment.

$ adop phase=cutover

Cleanup phase:

Removes obsolete code and data from old editions. Remove old objects that are no longer needed.

$ adop phase=cleanup

Abort command:

Aborts the online patching cycle by dropping the database patch edition. This phase cannot be specified with any other phase.

Online patching cycle can be terminated, with the actions taken being discarded. Only available up to (but not including) the cutover phase. After cutover, the system is running on the new edition, and abort is no longer possible for that patching cycle.

$ adop phase=abort

Note: After an online patching cycle is started, you should not perform any configuration changes in the run edition file system. Any that are made will not be propagated, and will therefore be lost after cutover is complete.

• The prepare, apply, and fs_clone phases all require at least 10GB of free disk space. All other phases require 1GB of free space. A warning message will be displayed if less than the needed amount is available.

• The directories where you extracted the patches applied in a given patching cycle must be retained, in the same location and with the same contents, until the next prepare phase completes.

This is also a requirement for patches applied in hotpatch or downtime mode.

• Any customizations must be applied to the patch edition during the apply phase, normally after any Oracle E-Business Suite patches have been applied.
Special Phases:

abort:

used to terminate a patching cycle before it is complete, and roll back any changes that have been made. It can also be run in conjunction with a full cleanup operation.

fs_clone:

Recreates the patch edition file system as an exact copy of the run edition file system. This phase cannot be specified with any other phase. Use of fs_clone is normally not required.

Situations that do require fs_clone are will explicitly document that requirement.

This command must be invoked from the run file system, before the next prepare phase is run.

If running this phase, ensure that your current working directory is not within the patch edition file system.

The fs_clone phase is a command (not related to adcfgclone.pl) that is used to synchronize the patch file system with the run file system. Normally, the fs_clone phase should only be run when mentioned as part of a specific documented procedure.

Note: You may perform a procedure that as a final step instructs you to run fs_clone. You do not have to do this immediately, the key requirement is to run fs_clone before you start the next patching cycle. And if you are performing multiple procedures, each of which requires fs_clone to be run at the end, you only need to run it once before the start of the next patching cycle.

During the prepare phase, adop performs the following steps:

• Whether to perform a cleanup
• Validates system configuration
• If the database is prepared, database user is edition-enabled.
• If the patch service has been created.
• If logon trigger exists and is enabled.
• Integrity of the database data dictionary.
• EBS Technology Codelevel Checker (ETCC) has been run, to verify that all required patches have been applied to the database.
• System configuration on each application tier node.
• Existence of the "Online Patching In Progress" (ADZDPATCH) concurrent program.

ADZDPATCH - Prevents certain predefined concurrent programs from being started, and as such needs to be active while a patching cycle is in progress (that is, while a database patch edition exists ADZDPATCH is cancelled when the cutover phase is complete.

Invokes the TXK script txkADOPPreparePhaseSynchronize.pl to synchronize the patches which have been applied to the run APPL_TOP, but not the patch APPL_TOP.

Checks the database for the existence of a patch edition, and creates one if it does not find one.

Calls the $AD_TOP/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl script again to confirm that the database connection to the patch edition is working.

Optional User Checks:

Validate your system for patching -

$ adop -validate

Confirm there is adequate free space - Execute $AD_TOP/sql/ADZDSHOWTS.sql It will check SYSTEM tablespace minimum of 25 GB free, APPS_TS_SEED tablespace minimum of 5 GB free.

Perform the following steps in the prepare phase:

1. Set the environment by executing (sourcing) the run file system environment file:

$ source <EBS install base>/EBSapps.env run

Confirm that the environment is properly set

$ echo $FILE_EDITION

run

$ echo $TWO_TASK

dbSID

2. Download patches to be applied and place them in the $PATCH_TOP directory. This directory is pre-created by the install in the non-editioned file system (fs_ne) and should not be changed.

3. Unzip the patch:

$ unzip p99999999.zip

4. Prepare the system for patching by running the following command to start a new patching cycle:

In this step, the patch file system application tier is synchronized with the run file system application tier.

$ adop phase=prepare

Quickest option -

$ adop phase=prepare sync_mode=delta

The delta_sync_drv.txt file includes examples for setting up synchronization using rsync on UNIX or RoboCopy on Windows.

To include custom patching actions in the default file system synchronization, you must include the required commands in the Custom Synchronization Driver, $APPL_TOP_NE/ad/custom/adop_sync.drv.

Prepare phase in multi node environment:

In a multi-node environment, one application tier node will be designated as the primary node. This is the node where the Admin Server is located, and will usually also be the node that runs Oracle HTTP Server.

All other application tier nodes are designated as secondary nodes.

Online Patching Cycle Steps - Apply Phase:

adop applies the specified patches to the system. In an online patching cycle, patches are applied to the patch edition of the database and file system.

$ adop phase=apply input_file=<inputfile.txt>

An example input_filemight liook like this:

workers=<number of workers>

patches=

<patch number 1>:<driver file 1>.drv, 
<patch number 2>:<driver file 2>.drv ...

Files to examine are

$APPL_TOP/admin/<SID>/out/adzdcmped.out
$APPL_TOP/admin/<SID>/log/u<patch_number>.log

The analytics parameter should only be used when required, because of the extra processing needed.

Using analytics parameters:

$ adop phase=apply analytics=yes
will execute ADZDCMPED.sql, ADZDSHOWED.sql, ADZDSHOWOBJS.sql, ADZDSHOWSM.sql
Online Patching Cycle Steps - Finalize Phase:

$ adop phase=finalize

Actions of the finalize phase are:

• Pre-compute DDL that needs to be run at cutover.
• Compile all invalid objects.
• Validate that the system is ready for cutover.
• If finalize_mode=full, compute statistics for key data dictionary tables for improved performance.

Online Patching Cycle Steps - Cutover Phase:

$ adop phase=cutover

Note: No users should remain on the system during cutover, as there will be a short downtime period while the application tier services are restarted. Also, any third-party processes connected to the old run edition of the database should be shut down, or they will be terminated automatically. If desired, you can defer running cutover until a time which will cause minimal disruption to users.

• Shut down internal concurrent manager
• Shut down application tier services
• Cutover database:

Promote patch database edition to become the new run database edition, using adzdpmgr.pl script.

Ref.: https://smartdogservices.com/how-and-when-does-an-edition-get-dropped/

EBR (Edition-Based Redefinition) manages more than one copy of code-type objects, such as procedures, triggers, and views. Storage objects like tables and indexes do not support editioning.

With 12.2, however, online patching uses other features of EBR to manage the non-editioned objects (tables and views) during patching. The features include logical views (editioning views), which allow for data model changes, cross-edition triggers, which are used to synchronize data during the patching process, and edition data storage for seed data updates.

The full clean-up in 12.2 executes the following:

 

• Old code objects that are no longer visible in the run edition are dropped.
• Old rows or columns that are no longer visible in the run edition are deleted or dropped.
• Old database editions that are no longer contain actual objects are dropped.

5. Cutover file system:

Promote patch file system to become the new run file system, switching the $FILE_EDITION values in the patch and run environments. The current patch APPL_TOP becomes the new run APPL_TOP, and the current run APPL_TOP

becomes the new patch APPL_TOP. Once cutover is complete, it is not possible to revert to the previous edition.

6. Terminate old database sessions: Terminate any database connections to the old run edition of the database.

7. Start application tier services: Application tier services are restarted, on the new run edition. The system is now available again to users.

Note: The adop utility invokes the TXK script txkADOPCutOverPhaseCtrlScript.pl to perform tasks 1, 2, 3, 5, and 6. Task 4 is performed by AutoConfig.

You can terminate the internal concurrent manager by executing the adcmctl.sh abort command from a different shell.

Refer Document 1584097.1, Oracle EBusiness Suite Release 12.2: Backup and Recovery Guidelines For Online Patching Cutover.

Deferring Application Tier Restart at Cutover:

$ adop phase=cutover mtrestart=no

Perform any additional steps that require the services to be shut down, and then start the application tier services manually using the adstrtal.sh script.

Patching the Database Tier:

Steps for post-cutover:

On the application tier, as the applmgr user:

1. Change directory to the run file system $APPL_TOP and source your environment file.

2. Run $ perl <AD_TOP>/bin/admkappsutil.pl

This will create the appsutil.zip file in <INST_TOP>/admin/out.

On the database tier, as the oracle user:

Copy or ftp the appsutil.zip file to the RDBMS_ORACLE_HOME, then run the following commands:

$ cd <RDBMS_ORACLE_HOME>

$ unzip -o appsutil.zip

3. Run AutoConfig on the database tier.

4. Run AutoConfig on the run file system of each application tier node.

5. Start the application tier services.

Online Patching Cycle Steps - Cleanup Phase:

$ adop phase=cleanup

Note: If you fail to run the cleanup phase explicitly, it will be run automatically on the next prepare cycle, but this will cause a delay in starting your next online patching cycle.

Dropping (removing) obsolete: Crossedition triggers, Seed data, Editioned code objects (covered objects), Indexes, Columns, Editions

cleanup_mode=quick - Performs minimum cleanup, including removal of obsolete crossedition triggers and seed data.

cleanup_mode=standard - Does the same as quick mode, and also drops (removes) obsolete editioned code objects (covered objects). This is the default mode, so does not need to be specified.

cleanup_mode=full - Performs maximum cleanup, which drops all obsolete code and data from earlier editions.

Online Patching Cycle Steps - Abort Phase:

$ adop phase=abort

In the abort phase, adop does the following:

1. Confirms that there is an in-progress online patching cycle, so the abort call is therefore valid.

2. Checks for the existence of a patch edition and drops one if it exists.

3. Cancels the ADZDPATCH concurrent program, if it is running.

4. Deletes the rows inserted for the pending session ID from the
ad_adop_sessions and
ad_adop_session_patches tables.

Note: After running abort, a full cleanup must be performed.

$adop phase=cleanup cleanup_mode=full
This will remove any columns that were added by the patch but are no longer needed because of the abort. If they are not removed, they may cause problems in a later patching cycle.

Combined command to abort the patching cycle and full cleanup:

$ adop phase=abort, cleanup cleanup_mode=full

If any attempt was made to apply patches to the patch edition, after abort you must run the fs_clone phase
$ adop phase=fs_clone
to recreate the patch file system.

Dropping Old Editions With the actualize_all Phase:

System performance will be affected when the number of old database editions reaches 25 or more, consider dropping all old database editions by running the
$adop actualize_all
phase and then performing a full cleanup.

Note: This procedure will take a large amount of time (significantly longer than a normal patching cycle), and should only be performed when there is no immediate need to start a new patching cycle.

Before starting, you should ensure that the system has the recommended database patches and latest AD-TXK code level installed.

To proceed, run following commands in the order shown:

$ adop phase=prepare

$ adop phase=actualize_all

$ adop phase=finalize finalize_mode=full

$ adop phase=cutover

$ adop phase=cleanup cleanup_mode=full

You have now completed removal of the old database editions. 

Configuration Management and Patching:

Refer - Document 1355068.1, Oracle E-Business Suite 12.2 Patching Technology Components

Online configuration changes:

Performed within the context of an online patching cycle. This is the recommended strategy. By running

$ adop phase=prepare

Then make the desired configuration changes to the patch file system. They may include:

• Oracle WebLogic Server configuration changes
• HTTP Server configuration changes
• File system changes performed by the AD utilities
• After making the configuration changes, you must run the following command to promote them.

$adop phase=cutover

You must also run the command to propagate configuration changes to secondary file system.

$adop phase=fs_clone

Offline configuration changes:

Applied directly to the run file system, outside an online patching cycle. Use following command to verify that no patching cycle is currently active.

$adop -status

After making the desired configuration changes, you must explicitly run following command to propagate the changes to the patch file system.

$adop phase=fs_clone

Note: This offline scenario will require a period of downtime for users.

Single File System Development Environments:

• Single file system environment must have a single-node application tier: multinode application tiers are not supported.
• cannot use online patching in a single file system environment.
• cannot convert a single file system environment back to using a dual file system.
• cannot clone from a single file system environment.

Set up a single file system development environment by installing Oracle EBusiness Suite Release 12.2 in the normal way, and then deleting the $PATCH_BASE directory with the command:

$ rm -rf $PATCH_BASE

Single file system development environment. These are:

• apply phase in downtime mode
• cleanup phase

Restrictions on Applying Patches in hotpatch Mode:

You should not attempt to apply a patch in hotpatch mode unless explicitly directed to do so by the patch readme.

For designed and tested Patch only. Hotpatch mode applies changes to the run edition while this edition is in active use, which may result in one or more of the following issues

• Runtime transactions may fail due to invalid objects, loss of PL/SQL package state.

• Application code and database objects may be temporarily inconsistent.

• Seed data may change, and may be temporarily inconsistent.

• Tables that are patched will be temporarily inconsistent.

• Code and data cached in application tier server memory may be inconsistent with changes made by the hotpatch.

• Runtime processing may hold long-term locks on code or data, leading to execution failures in the hotpatch.

Also, when you use hotpatch mode to apply a patch that contains a downloadable resource (such as a Forms-related client JAR file), that resource will only become

available after you restart the Oracle WebLogic Server Managed Servers. Until you perform the restart, you may receive an error on the client or server about the integrity

of the resource file.

Emergency Application of Patches:

Deploy patches directly to the run file system and run edition, with the following important restrictions being strictly adhered to:

• The patch using adop downtime mode.

• No patching cycle can be in progress.

• To prevent massive invalidation in the database, code objects deployed to the database should not include parent objects that could cause extensive invalidation

on dependent objects. If this does happen, a significant amount of time will be required for code recompilation.

• Directories that contain code deployed to the application tier during emergency patch deployment must be registered with the custom synchronization driver, to ensure successful automatic file system synchronization by the adop synchronization process.

• Deployment of grants to base objects results in code actualization and recompilation, and the subsequent risk of code invalidation.

For guidance on how to perform such grant operations safely, refer to My Oracle Support Knowledge  Document:1987947.1, Granting Privileges On An Object May Cause Invalidations.

Note: Most Oracle E-Business suite patches are not tested in downtime or hotpatch mode. It is therefore important that this type of deployment is only used in an emergency,

and not incorporated into standard maintenance practices. 

Considerations When Re-Applying Patches:

if you try to apply a patch that has already been applied, and you do not specify options=forceapply, adop will display an error.

Two more scenarios that may occur in this type of situation:

• When a failed patch session is restarted with abandon=no, restart=yes, the patches applied in current adop session will not be applied even if options=forceapply is specified.

For example, you run the command

$adop phase=apply options=forceapply patches=1111,2222 ,

and application of patch 1111 is successful but patch 2222 fails. After correcting the problem, you try to rerun adop with the command

$adop phase=apply options=forceapply patches=1111,2222 abandon=no, restart=yes

In this example, patch 1111 would be skipped as it had successfully been applied, and application of patch 2222 would resume.

If you wanted to apply patch 1111 again, you would need to specify

abandon=yes, restart=no.

• If you apply multiple patches with merge=yes , and you do not specify

options=forceapply, the patches will be applied only if at least one of the patches has not been successfully applied before.

Note: This check will be performed for AD and non-AD patches separately, as adop applies these two categories of patch in different sessions.

Document 396009.1, Database Initialization Parameters for Oracle E-Business Suite Release 12

Monitoring ADOP:

Online Patching Monitoring utility (adopmon) to provide a continuously refreshed view of key adop actions, in a coceptually similae way to the tail -f UNIX command.

$ adopmon

The adopreports utility is invoked by entering the command:

$ adopreports <APPS username> <APPS password>

This displays the adopreports Main Menu:

Online Patching Diagnostic Reports Main Menu

--------------------------------------------

1. Run edition reports

2. Patch edition reports

3. Other generic reports

4. Exit

Analyzing Log Files:

$ adopscanlog

$ adopscanlog -help

$ adopscanlog -latest=yes

$ adopscanlog -latest=yes -phase=<phase_name>

$ adopscanlog -session_id=<number>

$ adopscanlog ?scan_dir=<full path of directory to scan>

Common Issues:

Defragmentation of table:

To defragment the table, perform the following steps.

1. Run the prepare phase.

2. Run the command:

SQL>ADZDTABRDF.sql <owner> <table> [interim table name]

Note: Supplying the interim table name is only needed if you want to redefine a non-partitioned table to a partitioned table.

In such a case, you must create the partitioned table before running this script.

3. Run the cutover phase.

The fragmentation will now have been eliminated.

Defaults File Becomes Corrupt:

The system-generated defaults file is located at $APPL_TOP/admin/<SID>/adalldefaults.txt

If this file becomes corrupt, running AutoConfig will automatically instantiate a new copy.

Cutover Fails:

Document 1584097.1, Oracle E-Business Suite Release 12.2: Backup and Recovery Guidelines For Online Patching Cutover
To resume a failed session after cutover exits with cutover_status=3, you may receive an 'Invalid Credentials' error.

$ adop phase=cutover action=nodb

If cutover fails to complete, check log messages for any problems that may require correction. Then try executing the cutover command again. When cutover is

re-executed after a previous failure, adop will restart cutover processing at the failure point for any nodes that did not complete, and the processing may be successful this time.

Abandoned Node:

In a multi-node environment, the user executes adop commands only on the primary node. The primary instance of adop then uses ssh remote invocation to execute patching actions on all secondary nodes.

At the end of each primary node adop command, a summary report is produced showing the status of the command on each secondary node. If a secondary node fails to complete a phase, you should investigate the related log messages.

Abandoned nodes must either be removed from the system, or recloned (copied) from an available node. If you start a new adop session (prepare, apply, fs_clone) while abandoned nodes are present in the system, adop will display an error:

Document 1677498.1, How to Restore An Abandoned Node in Oracle EBusiness Suite Release 12.2

Applying a Single Patch to an NLS Installation:

If an Oracle E-Business Suite system contains languages other than American English (US), the recommended method is to apply the US patch first and then apply the

translation patch for each installed language. If you have installed more than one additional language, you can merge all the translation patches and apply them as a single, merged NLS patch.

To see a list of all completed patching sessions with patch details:

Run the adphrept.sql script (located in <AD_TOP>/patch/115/sql). This script produces an XML report showing individual patching sessions.

$ cd $AD_TOP/patch/115/sql

$ sqlplus <APPS username>/<APPS password> @adphrept.sql ALL ALL 01/01/2012 01/31/2012 ALL ALL ALL ALL ALL N N N N N jan09.xml

To display information about files changed by patches:

Run the adfhrept.sql script (located in <AD_TOP>/patch/115/sql) to produce an XML report named adfilerep.xml. Use the following parameters:

$ cd $AD_TOP/patch/115/sql

$ sqlplus <APPS username>/<APPS password> @adfhrept.sql admorgb.pls N 01/01/2013 01/31/2013 ALL ALL ALL N N N N N

To see a list of all patches in a given date range:

The adpchlst.sql report (located in <AD_TOP>/patch/115/sql) produces a list (adpchlst.lst) of all patches in a date range, without patch detail.

$ cd $AD_TOP/patch/115/sql

$ sqlplus <APPS username>/<APPS password> @adpchlst.sql 07/01/2014 07/31/2014

Doc ID 1091294.1 - How to Use the My Oracle Support Conflict Checker Tool for Patches Installed with OPatch [Video]
Doc ID 1913040.1 - Patch conflict analysis may not find resolutions for large merged patches.

Ref.:

https://docs.oracle.com/cd/E26401_01/doc.122/e22954/T202991T531062.htm#5281339