Oracle E-Business Suite Patching using adop

Following description about Oracle E-Business Suite is high level and from documentation
https://docs.oracle.com/cd/E26401_01/doc.122/e22954/T202991T531062.htm#5281339
for in depth and detail description refer it.
 

E- Business Suite patching:
Oracle E- Business Suite technology layer include:

• AD - Oracle Applications DBA (AD)
• FND - Oracle Application Object Library (FND)
• AU - Oracle Applications Utilities (AU)
• AK - Oracle Common Modules (AK)
• WF - Oracle Workflow (WF)
• ALR - Oracle Alert (ALR)
• FWK - Oracle Application Framework (FWK)
• XDO - Oracle XML Publisher (XDO)
• RUP - Release update pack (RUP)
• TXK - EBS 12.2 technology stack (TXK) and Applications Database (AD) components
• CPU - Critical Patch Updates
• PSUs - Patch set updates
• ATG - Application technology group
• PAA - Oracle Patch Application Assistant
• OAM - Oracle Applications Manager

Patch files Structures:

• The README.txt or README.html file describes what the patch does.
• Driver File - u<patchnum>.drv, contains the commands necessary to change files and database objects, and to generate new objects.

Patch format:

• Individual bug fix
• Product family release update pack (product family RUP)
• Release update pack (RUP)
• Pre-upgrade patch
• Consolidated upgrade patch

Codelevel:
This unique set of product features for a point release is referred to as a a codelevel, and assigned a unique number.
Codelevel R12.GL.A.1 is the first set of fixes to codelevel R12.GL.A, R12.GL.A.2 is the second, and so on.
Codelevels are cumulative - Each one contains the initial set of features plus all the fixes created to date for that product or product family.
All codelevels created after the initial point release for a product or product family are aggregated into release update packs (RUPs).
RUP1 is equivalent to R12.0.1, RUP2 is equivalent to R12.0.2, and so on. RUPs provide bug fixes. In addition, they may also provide feature enhancements, which provide new functionality that has a limited impact on your system.
Patching Utilities:
adop (AD Online Patching) - apply patches to the Oracle E-Business Suite file system or database. Oracle Applicaiton DBA online patching utility.
admrgpch (AD Merge Patch ) - AD Merge Patch tool was used to merge multiple patches into a single patch, so that the common tasks only needed to be performed once.
12.2, all the functionality of AD Merge Patch has been included in the adop patching tool. By default, adop will apply a list of patches individually, but if desired you can specify the merge=yes parameter to have adop
merge the listed patches and apply the merged patch. Can still be run independently of adop.
admsi.pl (Patch Application Assistant) - Helps you track and perform manual steps during patching, and provides consistency in the format of manual steps.
OAM (Oracle Applications Manager - Query Applied Patches, File history, Patch wizard, Timing reports, Registered Flag Files, Software updates.
adop:
always runs from the run edition file system.
automatically sets its environment correctly, regardless of the edition it is run from.
no new updates to files or database objects in your system, adop takes no action
previously failed patching session, it will attempt to recover that session
adop does:
Reads patch metadata, uploads patch information, reads and validate the patch driver file, reads the product driver files, compares version numbers of object modules, backs up all existing files that will be changed, copies files, archive files in libraries, relinks executables, generates forms, reports, messages, graphics, and Java archive (JAR) files, compiles JSP files and invalid database objects, updates database objects, runs autoConfig to update configuration files, saves patch information to the database.
adop help and examples:
adop -help Get information about basic operation
adop -examples Display examples of the various adop parameters
adop parameters:
skipsyncerror [prepare phase], yes/no, defaulty - no
sync_mode [prepare phase], patch/delta, default - patch
abandon [apply phase], yes/no, default - no
apply_mode [apply phase], online/downtime/hotpatch, default - online, Important: You must only use the downtime and hotpatch modes when explicitly directed, for example by the patch readme.
flags [apply phase] - autoskip/noautoskip,  flags=autoskip, Used to specify numerous options. To see a full list, enter:$ adop -examples, autoskip/noautoskip, default is noautoskip.
merge [apply phase] - yes/no, default - no
options [apply phase] - specify various options during the apply phase.
restart [apply phase] - yes/no, default - no
cm_wait [cutover phase] - Integer representing number of minutes to wait, recommendation - on production do not specify cm_wait, specify in non production systems.
mtrestart [cutover phase] - whether to restart application tier services after cutover, yes/no, default - yes.
cleanup_mode [cleanup phase] - quick/full, Full cleanup is required after aborting an online patching cycle.
Patch phase:
prepare/apply/finalize/cutover/cleanup/fs_clone/abort/actualize_all, use a comma-separated list to specify multiple phases.
For example, 'phase=prepare, apply' loglevel - STATEMENT/PROCEDURE/EVENT/WARNING/ERROR/UNEXPECTED, default - EVENT.
input_file - specify the name of the input_file supplied to adop.
patches - specify the patches adop is to apply, 'patches=123456', you would specify patch number 111 and the  Korean language version of patch 222 as 'patches=111, 222_KO:u222.drv
hotpatch - patches are to be applied in hotpatch mode, hotpatch=yes, use hotpatch mode when explicitly directed, for example by the patch readme.
prompt - whether adop prompts the user whether to continue after warnings, yes/no, default - yes
workers - number of parallel workers to be employed, If you omit the 'workers' argument, a suitable number of workers will be be chosen automatically.
defaultsfile - path to the custom adop defaults file, default - $APPL_TOP/admin/ $TWO_TASK
patchtop - location where the patches are unloaded, default - $PATCH_TOP
allowcoredump - core dump should be generated if adop crashes, yes/no, default-no.
analytics - generate reports that can be helpful in debugging, yes/no, default-no 
Patch Parameters Order:
Following order of precedence is used.

1. Command Line

2. Input File

3. Defaults File

Note: You should never need to edit the defaults file because you can always specify a parameter on the command line (highest precedence) or in the input file (second highest precedence).

Patch Log Files:

The adop log files are located on the non-editioned file system (fs_ne). This location is referred to as the $AD_LOG_HOME.

Sessions:

You can only run one adop session at a time on a particular,

Patch Directory (Patchtop):

The directory where the patch files have been unzipped is referred to as the patchtop. The default patch top directory is $NE_BASE/EBSapps/patch, which is pointed to by the $PATCH_TOP environment variable.

If you want to merge patches before applying them, you need to download and unzip all the individual patches in the same location as that of the merged driver file.

Patch Driver File:

The unified driver, named u<patchnum>.drv, contains the commands necessary to change files and database objects, and to generate new objects. It contains copy, database, and generate portions and performs the copy, database, and generate actions in the stated order. The adop utility runs the unified driver on all APPL_TOPs.

Number of Parallel Workers:

By default, adop runs database updates and file generation commands in parallel using multiple workers. The default number of workers is computed based on the system hardware configuration, but the number can be specified explicitly using the 'workers' parameter.

Principles of Non-Interactive Patching:

Non-interactive patching saves time by automating the patching process. It is used with all the major phases of adop, including the apply phase.

Error Messages:

When adop is using parallel processing and an error occurs, the job fails. Review the main adop log file and the adworkxxx.log file to determine the source of the error, resolve the issues and continue.

Restart adop using the adctrl command. If the message indicates that a worker has failed its job, you can fix the problem and restart the worker while the manager is running. Some failed jobs are deferred (not immediately reassigned) by the manager. These jobs do not cause the manager or otherworkers to stop.

Successful Completion Message:

adop displays a success message when processing is complete. If you do not see a such a completion message, you should investigate and identify the reason.

Backup Directory:

When adop runs, a backup directory is created in the directory where you unzip the patch. The old version of each file updated by the patch is copied into the backup directory.

adop Patching Modes:

• test mode
• downtime mode
• preinstall mode

adop test Mode:

In test mode, adop does not apply the patch. Instead, it lists each file it would have copied, relinked, executed, or generated, and shows exactly what actions it would have performed had it applied the patch. It also runs autoConfig in test mode to determine any impending changes to the configuration files. This allows you to see the effects of a patch on your system before you apply it. To run adop in test mode, add the apply=no parameter

to the adop command you would use if you were actually going to apply the patch.

adop downtime mode:

adop will first confirm that the application tier services are down, and will then proceed to apply the patch to the run edition of the Oracle EBS database and file system. Completes more quickly than in online mode, but at the cost of increased system downtime. example:$ adop phase=apply patches=123456 apply_mode=downtime.

Note: Be aware that:

• Release 12.2 patches are not normally tested in downtime mode.

• Downtime mode is only supported for production use where explicitly documented, or when directed by Oracle Support or Development.

Preinstall Mode:

Used during the upgrade process to update AD utilities, apply pre-upgrade patches,or work around other patching issues. adop asks all startup questions except those relating to the database. To run adop in preinstall mode, include preinstall=y on the adop command line. Because adop does not read driver files in preinstall mode, it copies all product files in the patch to the APPL_TOP directory.

Note: Run adop in preinstall mode only if the patch readme instructs you to do so.

adop Options:

The "options" argument is used to pass options that control how the patch is applied. It takes the form of a comma-separated list. Enter a single option, or a comma-separated list of options such as

adop options=nocopyportion, nogenerateportion.

actiondetails:print details of actions, Default: actiondetails

autoconfig: Run autoConfig automatically.Default: autoconfig.

Use options=noautoconfig if you are applying a number of patches in sequence and want to run AutoConfig once, after applying the last patch of the sequence.

Release 12.2 means that there is no need to shut down application tier services before running AutoConfig.

compiledb: automatically compile invalid objects in the database after running actions normally found in the database portion of the driver. Use options=nocompiledb to save time when multiple non-merged patches are applied in the same patching operation.

Merging multiple patches and applying a single merged patch is usually a better strategy.

compilejsp: whether to automatically compile out-of-date JSP files.

copyportion: run commands normally found in the copy portion of the driver. Default: copyportion databaseportion:run commands normally found in the database portion of the driver.

forceapply: reapply a patch that has already been applied.

generateportion: run commands normally found in the generate portion of the driver.

integrity: verify that the version of each file referenced in a copy action matches the version present in the patch. Default: nointegrity

parallel: update the database in parallel (such as SQL) and actions that generate files in parallel (such as genform). Default: parallel

phtofile: place patch history information after applying the patch. Default: nophtofile

validate: connect to all registered Oracle E-Business Suite schemas at the start of the patch.Default: novalidate

Running adop:

1: Set environment: $.<EBS_ROOT>/EBSapps.env run

2: Unzip the patch: Download and unzip the patch into the patch top directory $PATCH_TOP environment variable.

3: Review the information in the readme file: Review either readme file for information about the patch and for instructions on using Oracle Patch Application Assistant (PAA) to generate customized instructions for your system.

4: Run Oracle Patch Application Assistant: Run PAA (admsi.pl) to generate customized instructions for your system.

5: Run adop: PAA describe how to run adop using the adop command. Note: You can add arguments on the command line to refine the way adop runs.

Commands to execute a complete online patching cycle for patch 123456 is as follows:

$ . <EBS_ROOT>/EBSapps.env run

$ adop phase=prepare

$ adop phase=apply patches=123456

$ adop phase=finalize

$ adop phase=cutover

$ . <EBS_ROOT>/EBSapps.env run

$ adop phase=cleanup

Monitoring Status:

$ adop -status

$ adop -status <session ID>

$ adop -status -detail

Aborting an Online Patching Cycle:

$ adop phase=abort

Note: Used before successful completion of the cutover phase. After cutover, the system is running on the new edition, and abort is no longer possible for that patching cycle.

Aborting a patching cycle will drop the patch edition, but you must then run the cleanup and fs_clone phases before starting a new patching cycle. The cleanup must be a full cleanup.

For example:

$ adop phase=prepare

$ adop phase=apply patches=123456

[Patch application encounters problems and you want to abort]

$ adop phase=abort

$ adop phase=cleanup cleanup_mode=full

$ adop phase=fs_clone

Optionally, you can combine the abort and cleanup commands:

$ adop phase=abort,cleanup cleanup_mode=full

Note: You cannot abort application of a patch applied in hotpatch mode or downtime mode.

Restarting adop:

Investigate and resolve the problem that caused the failure, then restart adop. After you restart adop, it will ask if you want to continue with the previous session (at the point where the processing stopped), or start a new session.

Restart a failed patch from where it left off, you only need to specify restart=yes on the command line: adop phase=apply patches=1234 restart=yes

Restart a failed patch from the very beginning, you need to specify abandon=yes on the command line: adop phase=apply patches=1234 abandon=yes

Ignore a previously failed patch and apply a different one instead, you need to specify the new patch number and abandon=yes on the command line: adop phase=apply patches=5678 abandon=yes

AD Merge Patch:

Note: The functionality of AD Merge Patch is now included in the adop tool. Specify merge=yes when invoking the tool. AD Merge Patch is located in the AD_TOP/bin.

1: Set the environment:set the environment to indicate the location of the configuration parameters that define your system. This task is common to many AD utilities.

2: Run AD Merge Patch:$ admrgpch -s <source_directory> -d <destination_directory>

$ admrgpch -s /u01/R122_EBS/fs_ne/EBSapps/patch -d /u01/R122_EBS/fs_ne/EBSapps/patch/test

Now patches 111 and 222 have been merged, and the merged patch's patchtop is /u01/R122_EBS/fs_ne/EBSapps/patch/test:

Running Oracle Patch Application Assistant:

1: Set environment: set the environment to apply the configuration parameters that define your system. This task is common to many AD utilities.

2: Unzip the patch: Download the patch into the patch top directory ($PATCH_TOP) and unzip it.

3: Review the information in the readme file: Review either of these files for information about the patch and for instructions on using Oracle Patch Application Assistant to generate customized instructions for your system.

4: Run Oracle Patch Application Assistant: Run PAA (admsi.pl) to generate customized instructions for your system.
Ref.:
https://docs.oracle.com/cd/E26401_01/doc.122/e22954/T202991T531062.htm#5281339