71. Common causes of
"The opatch Conflict check failed"
A true patch conflict,
the solution is to open an SR and request merge patch
A Clusterware patch may
report a conflict even it is super set of the currently installed patch.
Normally OPatch should rollback the installed patch in favor of the super set,
but in some rare case, OPatch errors out; the solution is to manually rollback
and apply the new patch.
Common causes of
"The opatch Applicable check failed"
ZOP-46: The patch(es) are not applicable on the Oracle Home because some
patch actions are not applicable. All required components, however, are
installed.
Prereq "checkApplicable" for patch 13653086 failed.
..
Copy Action: Desctination File "/u02/grid/11.2.0/bin/crsd.bin"
is not writeable.
'oracle.crs, 11.2.0.2.0': Cannot copy file from 'crsd.bin' to
'/u02/grid/11.2.0/bin/crsd.bin'
--Some of the files in
target home are still being locked - some processes still running, or "rootcrs.pl -unlock" / "roothas.pl
-unlock" has not been executed
--Unzipped patch files
in stage area are unreadable by grid or database user: "Copy Action:
Source File
"/patches/B202GIPSU4/12827731/files/bin/appvipcfg.pl"
does not exists or is not readable". The solution is to unzip the patch by
any OS user that has primary group oinstall, for example, grid user.
--Database control
(Enterprise Manager)/EM agent is not stopped: "Copy Action: Desctination
File
"/ocw/grid/11.2/bin/emcrsp.bin"
is not writeable.". To stop, as the ORACLE_HOME owner execute: $
<ORACLE_HOME>/bin/emctl
stop dbconsole
A directory is provides
when prompted "OPatch is bundled with OCM, Enter the absolute OCM response
file path", the solution is to enter absolute OCM response file directory
and filename For other potential causes, review opatch logfile. The OPatch log
file can be found in
$ORACLE_HOME/cfgtoollogs/opatch.
72. Common causes of
"The opatch Component check failed"
--Incorrect patch
location specified
--The patch was unzipped
in non-empty directory
--The proper version of
the OPatch executable was is not installed into the ORACLE_HOME being patched
and OPatch was not launched from that ORACLE_HOME
The current working
directory is $ORACLE_HOME/OPatch of the ORACLE_HOME that is being patched
--The
$ORACLE_HOME/.patch_storage cannot be created, the solution is to create the
directory manually with corresponding grid or database user with permission of
750
--The listener is
running from database home, the solution is to shutdown listeners from database
home
--The grid and database
users are different, the workaround is to apply manually
--The patch is intended
for other version, i.e. when applying 11.2.0.2 GI PSU5 to 11.2.0.3 GI home
73. Common causes of
"The opatch version check failed"
ZOP-49: Not able to
execute the prereq. OPatch cannot inform if the patch satisfies minimum version
requirement.
--Incorrect patch
location specified
--OPatch version does
not meet the requirements for the patch (specified in the README), you must
upgrade OPatch.
--The grid or database
user cannot read unzipped patch directory/files, check directory permissions
74. OPatch reports:
"Prerequisite check CheckSystemSpace failed"
On AIX, due to
unpublished bug 9780505 opatch needs about 22GB of free disk space in
$GRID_HOME to apply a GIPSU/Bundle compared to about 4.4GB on other platforms.
"opatchutil cleanup -oh <home-path>"
can be used to reclaim space in .patch_storage, refer to Document 550522.1 and
Document 1088455.1 for a full explanation on this issue as well as and other
tips to work around this issue.
75. "opatch
apply" or "opatchnapply" failure if clusterware home is not
unlocked
If
"opatchnapply" or "opatch apply" is executed without
unlocking clusterware home first, the following will be reported:
ApplySession failed: ApplySession failed to
prepare the system. ApplySession was not able to
create the patch_storage area: /ocw/grid/.patch_storage/<patch#_date>
..
OPatch failed with error code 73
OR
chmod: changing permissions of
`/ocw/grid/bin':
Operation not permitted
..
mv: cannot move `/ocw/grid/bin/oracle' to
`/ocw/grid/bin/oracleO': Permission denied
..
The solution is to
carefully follow the instructions in the README packaged with the patch which
will provide instructions use "opatch auto" if applicable or the
manual opatch process.
If "opatch
napply" or "opatch apply" is executed without unlocking
clusterware home first,
76. OPatch reports:
"Patch nnn requires component(s) that are not installed in
OracleHome"
These not-installed
components are oracle.crs:<version>, oracle.usm:<version>
The patch is not
applicable to the target home, i.e. if apply 11.2.0.3 GI PSU1 to 11.2.0.2 GI
home, the error will be reported, refer to Document 763680.1 for
troubleshooting instructions.
77. What files to
review if opatch auto fails?
The key to OPatch Auto
troubleshooting is understanding where and how the OPatch Auto logs are
generated. When OPatch auto is invoked,
it will generate a opatchauto<timestamp>.log
log in the $ORACLE_HOME/cfgtoollogs
directory of the ORACLE_HOME that opatch auto was invoked. This log will
contain information around the execution of OPatch auto itself and will be the
starting point for troubleshooting issues. Now, OPatch auto will also invoke
additional OPatch sessions using the OPatch executable from the home that is
being patch to perform the pre-req checks and actually
apply the patch. Each of
these subsequent invokationsOPatch will generate a new
opatch<timestamp>.log log under $ORACLE_HOME/cfgtoollogs/opatch within
the ORACLE_HOME that is being patched. These logs are often times where the
root cause of an OPatch auto failure will be found.
Should OPatch auto fail
to restart the stack (noted in the error message) one should consult the
rootcrs or roothas log (11.2 only) in $GRID_HOME/cfgtoollogs/crsconfig,
as OPatch auto actually makes calls "rootcrs.pl"
or "roothas.pl".
Should an SR need to be
opened up in the event of a OPatch auto failure it is recommended to collect
the OPatchlogs (discussed above) as well as the output of the diagcollection
script as described Document 330358.1> and providethis diagnostic data at
the time of SR creation.
78. How to apply a
patch after the root script (root.sh or rootupgrade.sh) has failed?
In some cases root
script fails because of missing patch. When this happens we must first
determine whether the root script had made it to the point in which
theownership/permissions of the GI software has changed. This will allow us to
make the determination if unlocking theGI software is necessary prior to
installing the patch. To determine if unlocking is required, go to the GI home
and execute:
# find $GRID_HOME -user root
If the above command
returns no files, unlocking the software is not necessary. Should the command
return rootowned files you must unlock the GI Home by executing the following
as the root user:
# $GI_HOME/perl/bin/perl
$GI_HOME/crs/install/rootcrs.pl -unlock
Once the GI home is
unlocked (or it was determined that unlocking was not necessary, refer to
answer for the question"How do I apply a Grid Infrastructure patch before
the root script (root.sh or rootupgrade.sh) is executed?" to install the
required patch.
79. What's the
difference between manual opatch apply and opatch auto?
In a single instance
environment to apply database patch it is very simple. Shutdown all processes
running from a home to be patched and invoke "opatch apply" or
"opatchnapply". Very simple.
Now, to apply a
Clusterware patch to a Clusterware Home or a GI patch to Grid Infrastructure
Home, the clusterware stack needs to be down, the clusterware home needs to be
unlocked, the configuration needs to be saved, then the patch can be applied;
once the patch is applied, the configuration needs to be restored, the
clusterware home needs to be re-locked again, then the clusterware stack can
start. Once the Clusterware patch or GI patch was applied to the Clusterware/GI
home, you then have to go and apply the database components of that patch to
the Database Home with a whole other list of steps. This is a complex process
that had proven to be error prone if instructions were not carefully followed.
This is where OPatch
Auto comes in. The whole goal of OPatch auto is for an administrator to only
have to execute a single command to apply a patch to a RAC system, thus
simplifying the process.
80. How do I install
the latest OPatch release?
Prior to applying a
patch to a system it is HIGHLY recommended (often required) to download and
install the latest version of the OPatch utility into every ORACLE_HOME on
every cluster node that is to be patched. The latest version of OPatch can be
found on MOS under Patch 6880880. Be sure to download the release of OPatch
that is applicable for your platform and major release (e.g. Linux-x86-64
11.2.0.0.0). Once OPatch has been downloaded and staged on your target system(s)
it can be installed by executing the following as the Oracle Software
installation owner:
% export
ORACLE_HOME=<home-path>
% unzip -d $ORACLE_HOME
p6880880_112000_Linux-x86-64.zip
81. How to find out
the opatch version?
To find out the opatch version
perform the following as the Oracle software owner:
% export ORACLE_HOME=<home-path>
% $ORACLE_HOME/OPatch/opatch version
82. Do I need
downtime to apply a Clusterware or Grid Infrastructure patch?
If the Clusterware/Grid
Infrastructure home is not shared (common), Clusterware/Grid Infrastructure
patches can be applied in rolling fashion so no downtime (of the entire
cluster) is needed.
83. Can I upgrade the
Clusterware or Grid Infrastructure to a higher version while leaving database
at a lowerversion?
Yes, as long as
Clusterware/Grid Infrastructure is at higher version than the RAC Database
homes on the cluster this is perfectly acceptable
84. Do Exadata
patches apply to GI or RAC homes?
General speaking,
Exadata patches should only be applied to Exadata environments.
85. Which oracle home
does a patch apply to?
A home must meet the
patch version specification to be applicable.
In CRS environments, a
Clusterware patch (interim, MLR, bundle or PSU) applies to all homes (CRS, ASM
anddatabase), while a Database patch applies to the ASM and Database home only
In Grid Infrastructure
environments, a GI patch (interim, bundle or PSU) applies to all homes (GI and
Database), while a Database patch could be applicable to GI/ASM home if the fix
applies to ASM (in this case patch README will tellclearly that it applies to
GI/ASM home).
86. How do I tell
from the patch readme which home the patch applies to?
The patch README usually
tells which home it applies to.
Common identifier in
patch README:
# Product Patched : ORCL_PF ==> Applies to
database and pre-11.2 ASM home
# Product Patched : RDBMS ==> Applies to
database and pre-11.2 ASM home
# Product Patched : CRS/RDBMS ==> Applies
to both clusterware and database homes
Oracle Database 11g Release 11.2.0.2.3 ==>
Applies to database home.
Oracle Clusterware 11g Release 2 (11.2.0.2.2)
==> Applies to both clusterware and database
homes
87. Which PSU patch
applies to what home in mixed environments (clusterware and database at
different version)?
Example 1:
11.2.0.2 GI + 11.2.0.2 DB,
11.1.0.7 DB and 10.2.0.5 DB
In this environment, 11.2.0.2 GI
PSU applies to both 11.2.0.2 GI and 11.2.0.2 DB home(DB PSU
does not apply to any home),
11.1.0.7 CRS PSU and 11.1.0.7 database PSU applies to 11.1.0.7
DB home, and 10.2.0.5 CRS PSU and
10.2.0.5 database PSU applies to 10.2.0.5 DB home.
Example 2:
11.1.0.7 CRS + 11.1.0.7 CRS,
10.2.0.5 DB
In this environment, 11.1.0.7 CRS
PSU applies to 11.1.0.7 CRS home, 11.1.0.7 CRS PSU and
11.1.0.7 database PSU applies to
11.1.0.7 database home, 10.2.0.5 CRS PSU and 10.2.0.5 DB
PSU applies to 10.2.0.5 DB home.
88. What's Oracle's
patching recommendation?
Oracle recommends to
apply the latest patchset and latest PSU as a general best practice.
89. What does a
Clusterware/Grid Infrastructure patch contain?
Clusterware and Grid
Infrastructure patches have at least two portions, one for the clusterware
home, and the other for the Database home(s). Those two portions have same
version number. The clusterware home portion is in the top level directory of
the unzipped location, while the other portion is in
custom/server/<bug#>. For example, if 11.2.0.2.2 GI PSU is unzipped to
/op/db/11.2.0.2/GIPSU2, Grid Infrastructure home portion will be in
/op/db/11.2.0.2/GIPSU2/12311357 and database home specific portion will be in /op/db/11.2.0.2/GIPSU2/12311357/custom/server/12311357.
This is just an example, full details will be in the README for the patch that
is being applied. ALWAYS consult the patch README prior to applying any patch.
90. What's the
difference between Clusterware/Grid Infrastructure patches and Database
patches?
Generally speaking
Clusterware/Grid Infrastructure patches modify files that belong to the
Clusterware or GridInfrastructure product, while Database patches change files
that belong to the database product. As they apply to different sets of files,
they do not conflict with each other.
Please note:"files" in this context can refer to
binaries/executables, scripts, libraries etc
Clusterware files can
reside in all types of Oracle software homes like clusterware home, database
home and ASM homePrior to 11gR2, RDBMS files reside in DB/ASM homes only, while
with 11gR2 RDBMS files will also reside in the GI home (as ASM is now part of
GI). A GI PSU is a special type of clusterware patch as it also includes a
database PSU and modifies database binaries.
91.What's the
difference between a Clusterware/Grid Infrastructure bundle patch and a PSU?
A Clusterware or Grid
Infrastructure (GI) patch can be in the form of bundle or Patchset Update
(PSU). The biggest difference between a GI/Clusterware bundle and a PSU is that
PSUs are bound to a quarterly release schedule while a bundle may be released
at any time throughout the course of a given quarter. If a GI/Clusterware
bundle is released in a given quarter, the fixes in that bundle will be
included in the PSU that will be released for that quarter. This concept allows
for development to provide critical bug fixes in a more granular timeline if
necessary.
92. What are Critical
Patch Updates (CPUs)?
CPU Patches are
collection of high-priority fixes for security related issues and are only
applicable to the Database Home (and pre-11.2 ASM Home(s)) . CPUs are released
quarterly on the same cycle as PSUs and are cumulative with respect to prior
security fixes but may contain other fixes in order to address patch conflicts
with non-security patches.
PSUs always contain the
CPU for that respective quarter. PSUs and CPUs are NOT compatible meaning if
you apply the 11.2.0.2.2 Database PSU and then want to apply the 11.2.0.2 July
CPU this would result in the rollback of the 11.2.0.2.2 Database PSU. That
said, once a PSU patching strategy is adopted it must be maintained. PSU
patching is preferred over CPU Patching.
93. Can a Database
PSU be applied to a clusterware home?
No, only CRS PSUs, GI
PSUs or other Clusterware/GI patches can be applied to a Clusterware/GI home.
94. Will the 5th
digit of the version be changed after PSU is applied?
A PSU will not
physically update the 5th-digit of the release information, the updates to the
5th digit in the release are for Documentation purposes only. So the third GI
PSU that was released for 11.2.0.2 will have a documentation version of
11.2.0.2.3. You will NOT see this change reflected in the actual software
version if you query it from the inventory, clusterware or database.
96. What's a Patch
Set Update (PSU)?
As the name implies PSUs
are patches that are applied on top of a given patchset release. They are
released on a quarterly basis and contain fixes for known critical issues for
the patchset. PSUs are subject to thorough testing and do not include changes
that would alter the functionality of the software. With this in mind, PSUs are
designed to provide "low risk" and "high value" allowing
for customers to more easily adopt proactive patching strategies. Consider the
following PSU facts:
All PSUs are installed
via "opatch" and are not considered an "upgrade".
Database PSUs always
contain the CPU for the respective quarter that the PSU is released in. PSUs
and CPUs are NOT compatible meaning if you apply the 11.2.0.2.2 Database PSU
and then want to apply the 11.2.0.2 July CPU this would result in the rollback
of the 11.2.0.2.2 Database PSU. That said, once a PSU patching strategy is
adopted it must be maintained.
Independent PSUs are
released for both the Database and Clusterware or Grid Infrastructure
installations.
Clusterware PSUs
(pre-11.2) are referred to as CRS PSUs
Grid Infrastructure PSUs
are referred to as GI PSUs
GI PSUs do contain the
Database PSU for the corresponding release, e.g. 11.2.0.2.3 GI PSU
contains the 11.2.0.2.3
Database PSU
Database PSUs hold true
to their name
Both Clusterware/Grid
Infrastructure and Database PSU patches are cumulative. Clusterware PSU refers
to CRS PSU for pre-11gR2 and GI PSU for 11gR2.
GI PSUs are always
cumulative meaning that you can apply higher version GI PSU directly without
having to apply a lower version one first. For example, the 11.2.0.2.2 GI PSU
can be applied to a 11.2.0.2 home without having to apply GI PSU 11.2.0.2.1
first.
Database PSUs can be
subject to overlay PSU packaging. In these cases, the PSUs are still
cumulative, but a higher PSU may require a lower PSU to be applied first; for
example, to apply database PSU 10.2.0.4.7, you must apply database PSU
10.2.0.4.4 first. If a previous PSU is a prerequisite to a later PSU the
requirement will be clearly documented in the PSU readme.
97. What's the PSU
release schedule?
Generally speaking PSU
are released on quarterly basis for both Clusterware/Grid Infrastructure and
Database. There's cases where a Clusterware PSUs is not released for
corresponding Database PSU. For example, there's database PSU 10.2.0.5.4 but no
CRS PSU 10.2.0.5.4.
98. What's a
Patchset?
Compared to all other
patch types, a Patchset is released the least frequently. It contains fixes for
most known issues for the release and potentially also introduces new features.
A patchset is cumulative and when applied it changes the fourth digit of the
product release banner - for example, 10.2.0.5 is 4th patch set for 10.2, and
11.2.0.2 is the 1st patch set for 11.2.
A patchset must be
installed via the Oracle Universal Installer (OUI) and is generally considered
an "upgrade".
Prior to 11gR2, a base
release had to be installed before a patchset could be applied. For example, to
install 10.2.0.5 on Linux, the 10.2.0.1 base release has to be installed first
and then upgraded to 10.2.0.5.
Prior to 11gR2, the same
patchset download is used to patch the Clusterware, ASM and Database homes. For
example, Patch 8202632 is the 10.2.0.5 patchset, this same patch (Patch
8202632) will be used to patch the 10.2.0.x Clusterware, 10.2.0.x ASM and
10.2.0.x Database to 10.2.0.5.
Starting with 11gR2,
patchset releases are now full releases and no longer require a
"base" release, e.g. 11.2.0.2 can be installed directly without
having to install 11.2.0.1 first.
Prior to 11gR2 - even
though the CRS and RDBMS base releases were provided on separate media
(downloadable zip file or separate DVD/CD) - the patchsets for both products
were delivered as one i.e. the same patchset could be applied to the CRS as
well as the RDBMS home.
Starting with 11gR2 the
patchsets for Grid Infrastructure and RDBMS are delivered separately (as they
are full releases).
Clusterware
patchsets can be applied in a rolling fashion, while database patchsets cannot.
For example, you can rolling upgrade clusterware to 11.2.0.2, but you have to
shutdown database on all nodes to upgrade to database 11.2.0.2.
99. What Types of
Patches are available for Oracle Clusterware, Grid Infrastructure and/or the
Database?
Generally patches for
Clusterware, Grid Infrastructure and/or the Database are catagorized into the
following:
Patchsets
Patchset Updates (PSUs)
Critical Patch Updates (CPUs)
Bundle Patches
Interim (one-off) Patches
100. What type of
home do I have?
In a pre-11.2
Clusterware environment you will have:
A single Clusterware
(CRS) home One or more Database (RDBMS) homes at the same or at a lower release
level than the Clusterware home (to the 4th dot in the release) Optionally a
separate dedicated RDBMS home to run ASM (Automatic Storage Management) at the
same or at a lower release level than the Clusterware home (to the 4th dot in
the release)
In pre-11.2
installations, ASM was implemented using the RDBMS binary installation so this
should be treated as a typical RDBMS home.
In a 11.2 Grid
Infrastructure environment you will have:
A single Grid Infrastructure
(GI) Home that implements both ASM and the Clusterware
Note: During upgrade
from pre-11gR2 to 11gR2 GI, the ASM upgrade can optionally be performed
independently of the
Clusterware (not recommended). In this case the pre-11gR2 ASM Home would still
be active after the GI Upgrade. This is NOT the recommended method of upgrading
to Grid Infrastructure and will therefore not be discussed in this note.
Note: Starting with
11gR2 all upgrades of and to Grid Infrastructure MUST be performed
"out-of-place" (into a new software home) for this reason there may
be more than one clusterware home, however only one will be the active. This
"inactive" Clusterware or GI home should be removed once satisfied
that the upgrade was successful.
One or more database (RDBMS)
homes at the same or at a lower release level than the GI home (to the 4th dot
in the release)
101. What is cloning
and how does it work?
Cloning is the process
of copying an existing Oracle installation to a different location and updating
the copied Oracle Home to work in the new environment.
During cloning, OUI
replays the actions that were performed during the actual installation of the
home.
Cloning is similar to
installation except that OUI runs the actions in a special mode that is
referred to as clone mode.
102. What are the
different phases of cloning?
The cloning process has
two phases:
1.
Source Preparation Phase
During this phase,
prepare_clone.pl parses files in the source Oracle Home to extract and store
the required values. At the source, run a script called prepare_clone.pl.
This is a Perl script
that prepares the source for cloning by recording the information required for
cloning.
This script is generally
found in the following location:
$ORACLE_HOME/clone/bin/prepare_clone.pl
NOTE: The need to perform the
preparation phase depends on the Oracle product you are installing. This script
needs to be executed only for the Application Server Cloning.
Database and CRS Oracle Home
cloning do not require this.
Archive and compress the
source Oracle Home using your preferred archiving tool.
For example: You can use WinZip on Microsoft Windows
system computers and tar or gzip on UNIX. Make sure that the tool that you use
preserves the permissions and file timestamps.
When archiving the home,
also ensure that you skip the *.log, *.dbf, listener.ora, sqlnet.ora, and
tnsnames.ora for archiving.
2.
Cloning Phase
On the destination
system, you unarchive the Oracle Home and run the clone.pl script.
This Perl script
performs all parts of the cloning operation automatically by running OUI and
various otherutilities.
This script uses the
cloning functionality in OUI. When you run the clone.pl script, it handles the
specifics that OUI may have missed.
The Central Inventory of
the box where the home is being cloned is updated as is the Oracle Home
inventory ($ORACLE_HOME/inventory).
Note: The cloned home and source home will not
be identical in size, because the cloned home will have additional files
created during the cloning operation.
103. Why is cloning
required?
Cloning is required when
you want to deploy multiple Oracle Home(s) from an existing Oracle Home.
104. Advantages and
disadvantages of cloning?
Advantages
Creating an installation
that is a copy of a production, test, or development installation.
Cloning enables you to
create a new installation with all patches applied to it in a single step. This
contrasts with going through the installation process by performing separate
steps to install, configure, and patch the installation.
Rapidly deploying an
instance and the applications that it hosts.
Preparing an Oracle Home
and deploying it to many hosts.
You can also customize
various aspects of cloning, for example, to specify custom port assignments, or
to preserve custom settings.
Disadvantages
The cloning process
copies all of the files from the source Oracle Home to the destination Oracle
Home. Thus, any files used by the source instance located outside the source
Oracle Home's directory structure are not copied to the destination location.
Cloning is not possible across platforms
105. Where to find
the cloning log files?
All the logs can be
found in <Central_Inventory>/logs folder and
<ORACLE_HOME>/clone/logs.
107. Is cloning
faster than the actual installation?
Yes, cloning is much faster
than the actual installation. When cloning a source Oracle Home, all source
home patches and settings are seamlessly cloned to the new home. Cloning is
much faster than manually creating new homes and applying all source home
patches and settings.
108. Is it required
to copy the Central Inventory for cloning the Oracle Home?
Copying Central
Inventory is not required because:
If it is not the first
Oracle Home on the target Server:
Cloning will update the
central inventory with the information regarding the new home.
If it is the first
oracle home on the target Server:
Cloning will create a
central inventory with the information regarding the new home.
109. Do you need Perl
in your environment for cloning to work?
Yes, Perl is required.
Since cloning is done using the clone.pl script, which is a Perl script.
NOTE : Perl 5.6 or
higher is required for cloning and should be set in the environment.
110. What is the
difference between cloning and copying an Oracle Home ?
Cloning re-plays all the
actions that were performed during the installation of the Oracle Home like
relinking,updating the inventory, etc., and hence the cloned home can be
patched using OPatch, patchsets can be installed to it, etc., other than
extracting files from the software kit.
By copying the Oracle
Home from one server to another, all the above mentioned actions are not
performed and also it is not possible to apply patches or patchsets to copied
Oracle Home.
Simply copying the
Oracle Home is not supported.
If the Oracle Home has
to be deployed to multiple destinations with the same configuration, cloning is
the only supported method.
No comments:
Post a Comment