Jaspreet Singh Anand "Apps DBA Blog"

Tuning Concurrent Managers

2008-10-14T18:36:00.001+05:30

The Master Concurrent Managers

There is a lot of talk about "the" concurrent manager in Oracle Applications. Actually, there are many Concurrent Managers, each governing flow within each Oracle Apps areas. In addition there are "super" Concurrent Managers whose job is to govern the behavior of the slave Concurrent Managers. The Oracle e-Business suite has three important master Concurrent Managers:

Internal Concurrent Manager — The master manager is called the Internal Concurrent Manager (ICM) because it controls the behavior of all of the other managers, and because the ICM is the boss, it must be running before any other managers can be activated. The main functions of the ICM are to start up and shutdown the individual concurrent managers, and reset the other managers after one them has a failure.
Standard Manager — Another important master Concurrent Manager is called the Standard Manager (SM). The SM functions to run any reports and batch jobs that have not been defined to run in any specific product manager. Examples of specific concurrent managers include the Inventory Manager, CRP Inquiry Manager, and the Receivables Tax Manager.
Conflict Resolution Manager — The Conflict Resolution Manager (CRM) functions to check concurrent program definitions for incompatibility rules. However, the ICM can be configured to take over the CRM's job to resolve incompatibilities.
Now that we understand the functions of the master Concurrent Managers, let's take a quick look at techniques that are used by Oracle Apps DBAs to monitor the tune the behavior of the Concurrent Managers.
Tuning the Concurrent Manager

All successful Oracle Apps DBAs must understand how to monitor and tune each of the Concurrent Managers. This article will explore some of the important techniques for monitoring and tuning the Oracle Apps Concurrent Manager processes. The topics will include:

Tuning the Concurrent Manager
Tuning the Internal Concurrent Manager

Purging Concurrent Requests

Troubleshooting Oracle Apps performance problems

Adjusting the Concurrent Manager Cache Size

Analyzing the Oracle Apps Dictionary Tables
Monitoring Pending Requests in the Concurrent Manager

Changing the dispatching priority within the Concurrent Manager
Let's start by looking at tuning the ICM, and drill-down into more detail.
Tuning the Internal Concurrent Manager (ICM)

The ICM performance is affected by the three important Oracle parameters PMON cycle, queue size, and sleep time.

PMON cycle — This is the number of sleep cycles that the ICM waits between the time it checks for concurrent managers failures, which defaults to 20. You should change the PMON cycle to a number lower than 20 if your concurrent managers are having problems with abnormal terminations.
Queue Size — The queue size is the number of PMON cycles that the ICM waits between checking for disabled or new concurrent managers. The default for queue size of 1 PMON cycle should be used.
Sleep Time — The sleep time parameter indicates the seconds that the ICM should wait between checking for requests that are waiting to run. The default sleep time is 60, but you can lower this number if you see you have a lot of request waiting (Pending/Normal). However, reducing this number to a very low value many cause excessive cpu utilization.
All of the concurrent managers, with the exception of the ICM and CRM, can be configured to run as many processes as needed, as well as the time and days a manager can process requests. However, the number of processes needed is dependent on each organization's environment. An Applications DBA must monitor the concurrent processing in order to decide how to configure each manager. For a fresh install of the applications, initially configure the standard manager to run with five processes, and all the other managers with two processes. After the applications have been in operation for a while, the concurrent managers should be monitored to determine is more operating system process should be allocated.

Purging Concurrent Requests

One important area of Concurrent Manager tuning is monitoring the space usage for the subsets within each concurrent manager. When the space in FND_CONCURRENT_PROCESSES and FND_CONCURRENT_REQUESTS exceed 50K, you can start to experience serious performance problems within your Oracle Applications. When you experience these space problems, a specific request called "Purge Concurrent Requests And/Or Manager Data" should be scheduled to run on a regular basis. This request can be configured to purge the request data from the FND tables as well as the log files and output files on accumulate on disk.
Adjusting the Concurrent Manager Cache Size

Concurrent manager performance can also be enhanced by increasing the manager cache size to be at lease twice the number of target processes. The cache size specifies the number of requests that will be cached each time the concurrent manager reads from the FND_CONCURRENT_REQUESTS table. Increasing the cache size will boost the throughput of the managers by attempting to avoid sleep time.
Analyzing Oracle Apps Dictionary Tables for High Performance

It is also very important to run the request Gather Table Statistics on these tables:
FND_CONCURRENT_PROCESSES

FND_CONCURRENT_PROGRAMS

FND_CONCURRENT_REQUESTS

FND_CONCURRENT_QUEUES.
Run the request "Analyze All Index Column Statistics" on the indexes of these tables. Since the APPLSYS user is the owner of these tables, so you can also just run the request Analyze Schema Statistics for APPLSYS.

To troubleshoot performance, a DBA can use three types of trace. A module trace, such as PO or AR, can be set by enabling the module's profile option Debug Trace from within the applications. Second, most concurrent requests can be set to generate a trace file by changing the request parameters. To enable trace for a specific request, log in as a user with the System Administrator responsibility. Navigate to Concurrent -> Program -> Define. Query for the request that you want to enable trace. At the bottom right of the screen you can check the box Enable Trace. (Figure 1)

Figure 1: Troubleshooting Concurrent Manager Performance.

Another popular way to troubleshoot the Concurrent Managers is to generate a trace file. This is done by setting the OS environment variable FNDSQLCHK to FULL, and running the request from the command line.

Monitoring Pending Requests in the Concurrent Managers
Occasionally, you may find that requests are stacking up in the concurrent managers with a status of "pending". This can be caused by any of these conditions:

1. The concurrent managers were brought down will a request was running.
2. The database was shutdown before shutting down the concurrent managers.
3. There is a shortage of RAM memory or CPU resources.
When you get a backlog of pending requests, you can first allocate more processes to the manager that is having the problem in order to allow most of the requests to process, and then make a list of the requests that will not complete so they can be resubmitted, and cancel them.

To allocate more processes to a manager, log in as a user with the System Administrator responsibility. Navigate to Concurrent -> Manager -> Define. Increase the number in the Processes column. Also, you may not need all the concurrent managers that Oracle supplies with an Oracle Applications install, so you can save resources by identifying the unneeded managers and disabling them.

Figure 2: Allocating more processes to the Concurrent Manager.

However, you can still have problems. If the request remains in a phase of RUNNING and a status of TERMINATING after allocating more processes to the manager, then shutdown the concurrent managers, kill any processes from the operating system that won't terminate, and execute the following sqlplus statement as the APPLSYS user to reset the managers in the FND_CONCURRENT_REQUESTS table:
update fnd_concurrent_requests
set status_code='X', phase_code='C'
where status_code='T';

Changing Dispatching Priority within the Concurrent Manager
If there are requests that have a higher priority to run over other requests, you can navigate to Concurrent --> Program --> Define to change the priority of a request. If a priority is not set for a request, it will have the same priority as all other requests, or it will be set to the value specified in the user's profile option Concurrent:Priority.

Also, you can specify that a request run using an SQL optimizer mode of FIRST_ROWS, ALL_ROWS, RULE, or CHOOSE, and this can radically effect the performance of the SQL inside the Concurrent request. If several long running requests are submitted together, they can cause fast running requests to have to wait unnecessarily. If this is occurring, try to schedule as many long running requests to run after peak business hours. Additionally, a concurrent manager can be created to run only fast running requests.
Using data Dictionary Scripts with the Concurrent Manager

Few Oracle Applications DBAs understand that sophisticated data dictionary queries can be run to reveal details about the workings within each Concurrent Manager. Oracle provides several internal tables that can be queried from SQL*Plus to see the status of the concurrent requests, and the most important are FND_CONCURRENT_PROGRAMS and FND_CONCURRENT_REQUESTS.

APPS DBA INTERVIEW Questions and Answers

2008-10-14T18:29:00.000+05:30

Questions
1. What happens if the ICM goes down?
2. How will you speed up the patching process?
3. How will you handle an error during patching?
4. Provide a high-level overview of the cloning process and post-clone manual steps.
5. Provide an introduction to AutoConfig. How does AutoConfig know which value from the XML file needs to be put in which file?
6. Can you tell me a few tests you will do to troubleshoot self-service login problems? Which profile options and files will you check?
7. What could be wrong if you are unable to view concurrent manager log and output files?
8. How will you change the location of concurrent manager log and output files?
9. If the user is experiencing performance issues, how will you go about finding the cause?
10. How will you change the apps password?
11. Provide the location of the DBC file and explain its significance and how applications know the name of the DBC file.
Answers
1. All the other managers will keep working. ICM only takes care of the queue control requests, which means starting up and shutting down other concurrent managers.
2.
o You can merge multiple patches.
o You can create a response file for non-interactive patching.
o You can apply patches with options (nocompiledb, nomaintainmrc, nocompilejsp) and run these once after applying all the patches.
3. Look at the log of the failed worker, identify and rectify the error and restart the worker using adctrl utility.
4. Run pre-clone on the source (all tiers), duplicate the DB using RMAN (or restore the DB from a hot or cold backup), copy the file systems and then run post-clone on the target (all tiers).
Manual steps (there can be many more):
o Change all non-site profile option values (RapidClone only changes site-level profile options).
o Modify workflow and concurrent manager tables.
o Change printers.
5. AutoConfig uses a context file to maintain key configuration files. A context file is an XML file in the $APPL_TOP/admin directory and is the centralized repository.
When you run AutoConfig it reads the XML files and creates all the AutoConfig managed configuration files.
For each configuration file maintained by AutoConfig, there exists a template file which determines which values to pick from the XML file.
6.
o Check guest user/password in the DBC file, profile option guest user/password, the DB.
o Check whether apache/jserv is up.
o Run IsItWorking, FND_WEB.PING, aoljtest, etc.
7. Most likely the FNDFS listener is down. Look at the value of OUTFILE_NODE_NAME and LOGFILE_NODE_NAME in the FND_CONCURRENT_REQUESTS table. Look at the FND_NODES table. Look at the FNDFS_ entry in tnsnames.ora.
8. The location of log files is determined by parameter $APPLCSF/$APPLLOG and that of output files by $APPLCSF/$APPLOUT.
9.
o Trace his session (with waits) and use tkprof to analyze the trace file.
o Take a statspack report and analyze it.
o O/s monitoring using top/iostat/sar/vmstat.
o Check for any network bottleneck by using basic tests like ping results.
10.
o Use FNDCPASS to change APPS password.
o Manually modify wdbsvr.app/cgiCMD.dat files.
o Change any DB links pointing from other instances.
11.
o Location: $FND_TOP/secure directory.
o Significance: Points to the DB server amongst other things.
o The application knows the name of the DBC file by using profile option "Applications Database Id."

Q. Explain Oracle Applications architecture – Components?

Desktop tier
Application EBS (Forms, Reports, Concurrent Processing, Web, Admin, Disc)Q. Find application server version ?

FND_PRODUCT_GROUPS
Q. How will you find the Apache, forms, reports version in an EBS suite?
http –v – Apache
f60gen – Forms
ar60run – Reports
Q. Find jinitiator version?

Client side – Control panel
Server side – SID_hostname.xml (s_jinit_ver_)

Q. Whatis US directory in $AD_TOP or under various product TOP’s .
US directory is default language directory in Oracle Applications. If you have multiple language Installed in your Applications then you will see other language directories beside US, that directory will contain reports, fmx and other code in that respective directory like FR for France, AR for arabic, simplifies chinese or spanish.
Q. Whats main concurrent Manager types.
ICM - Internal Concurrent Manager which manage concurrent Managers
Standard Managers - Which Manage processing of requests.
CRM - Conflict Resolution Managers , resolve conflicts in case of incompatibility.
Q. Where is Concurrent Manager log file location ?
By default standard location is $APPLCSF/$APPLLOG , in some cases it can go to $FND_TOP/log as well.
Q. List some ad utilities and their functions?
adadmin, adctrl, adpatch, adlicmgr, admrgpch, adsplice and adworker
Q. List out the modules related to oracle Apps DBA ?

FND – Application Object Library, AU – Application Utilities, AD – Application DBA
Q. What are profile options, what are various type of profile options ?
User, application, responsibility and site
Q. List some profile options with their function?
ICX: Session timeout, ICX: Forms Launcher, ICX: Limit Connect
Q. How will you take the forms trace?

http://hostname:port/dev60cgi/f60cgi?play=&record=collect&log=filename
Q. How you put Applications 11i in Maintenance mode ?
Use adadmin to change Maintenance mode is Oracle Apps. With AD.I you need to enable maintenance mode in order to apply apps patch via adpatch utility. If you don’t want to put apps in maintenance mode you can use adpatch options=hotpatch feature.
Q. What are the different types of oracle patches available ?
One-off, mini, diagnostics, cu, rup, language, help, platform specific, OS, OPatch,
Interoperability, family pack and maintenance pack.

Q. What are the different patch drivers available?

C driver, D driver, G driver and U driver.
Q. Explain the steps for applying apps patch and database patch ?

Q. What is autoconfig?
Q. How will you find the autoconfig is enabled?
Q. List out the following logfile locations?

Patch log, autoconfig log, reports output, reports log and Concurrent manager log.
Q. List out the important env files?

Appsora.env, appsSID_hostname.env, adovars.env, APPSSID_hostname
Q. How to enable diagnostics for oracle applications?

FND: Enable Diagnostics.. Tell about RDA and its use?
Q. Describe about Discoverer Configuration and Oracle 10g AS installation ?
So start sharing information which can be useful to others like scripts, document by

Q. Explain Single Node or Multi Node installation
Single Node - All components of Apps 11i i.e. Database, CM, Forms, Web Server on single machine.
Multi Node - Apps 11i components on more than one Machine.
Q. Explain single user or multi user installation ?
Single User - Both Database and Application tier are owned by Single Operating System Account
Multi User - 11i Database Tier is owned by one O.S. User where as Application Tier is owned by different user.
Note* In both cases Apps is installed as user ROOT
Q. What is difference between express install or advanced install (system specific)
Express Install - Single User, Single Machine install where Apps 11i is installed on default location, Not much options during install.
Advanced Install - Advanced install with option to define o.s. Users, Mount Points and advanced installation.
Q. What is difference between fresh database and vision database install types ?
Fresh Database - Database is installed with Apps but with no data
Vision Database - Database installed with Apps with dummy data
Q. What are various components installed after 11.5.10 (11i) install ?
–9iAS (1.0.2.2.2) web server, Developer 6i Forms & Reports, Discoverer, Jinitiator
Q. What is O.S. level software requirement for installing Apps ?
ar, ld, make & X Display server for all unix machines (linux, solaris, IBM, HP Unix )
with following additional as per O.S.
Linux- gcc, g++, ksh
HP-Unix- cc, acc
IBM AIX - cc, linkx1C
For Windows you need
Microsoft C++, MKS Toolkit, GNU Make
Q. What is minimum approx. disk requirement for 11.5.10 (Note these disk requirement changes as per type of installation, languages installed and release )
For 11.5.10
Application Tier File System - 26 GB
Database Tier (Fresh install) - 31 GB
Database Tier (Vision install) - 65 GB
Q. What is staging area ?
Staging Area is special directory structure where you can dump 11i installation software so that you don’t have to insert CD’s during install and these disks will automatically be picked by Installer.
Q. How you set up staging area ?
Use adautostg.pl to create staging area orcreate required directory manually for staging like (following directories under Stage11i - startCD, oraApps, oraDB, oraiAS, oraAppsDB, oraNLS and inside these directories Disk1, Disk2…).

----------

Patching:

X. Utility used to apply application patch is ??
“adpatch”
X.Various options available with adpatch are
option=nocheckfile, noautoconfig, nocompiledb, nocompilejsp, nocopyportion, nodatabaseportion, nogenerateportion, hotpatch, integrity, prereq, noprereq, nomaintainmrc, validate, nojcopy, noforcecopy, forcecopy, nolink, nogenform, nogenrep
X. For database patch utility used to apply patch is ??
“opatch”
X..log and .lgi files for adpatch will be under $APPL_TOP/admin/SID/log directory.
X. What are .lgi files for ?
X. While applying patch put the system in maintenance mode by running
$adadmin and then select option 5 (change maintenance mode) after that select options enable/disable maintenance mode. (You can also change system to maintenence mode using $AD_TOP/patch/115/sql/adsetmmd.sql)
X.By default where adadmin log file will go??
$APPL_TOP/admin/$SID/log/abc.log
-
Workers:

X. How to check what adpatch is doing ??
“adctrl” is the utility to do check status of ad worker, including adpatch.
X. What are various stages of adpatch worker
First of all workers will be in “waiting” stage after that jobs will be “assigned” then “running“, If error doesn’t come it will move to “complete” (waiting -> assigned -> running -> complete)
If error comes it will go like waiting -> assigned -> running -> failed -> fixed -> restarted -> completed
X. What is Default number of workers:
2 times the number of CPU on the database server.
-
Deferred jobs
Have you ever seen worker status as deferred, while using adctrl ??
Once you start applying patch , manager assigns jobs and unique ID to each worker. Manager will also insert one row into FND_INSTALL_PROCESSES table for each worker with information about who is doing what ???
1) CONTROL_CODE 2) STATUS, these two columns are important for us from the fnd_install_processes. Manager will monitor the workers by these two columns about status of the assigned job.Once worker is done with first job, manager will assign it the second job to do with the status update as .
Once all the worker’s status is completed for all the jobs, manager will tell the workers to shutdown and will drop fnd_install_process table.
-
X. What happens if worker is working on some particular job say updating some table but that table is locked ??? so what the worker will do in such situation…
here comes AD_DEFERRED_JOBS table which will be created at the time of FND_INSTALL_PROCESSES and ad_deferred_jobs too will be dropped with FND_INSTALL_PROCESSES. First time when worker checks and find that table is locked, that job fails and manager will automatically defers the job, and it will assigns a new job to the worker.
The job which was waiting in ad_deferred_jobs table will be assigned again till time it will not become failed or completed.
In such situation above, worker will wait till either the lock is released or timed out is reached and We (DBA) need not to take any action till the job status becomes fails…. once it shows its failed we can fix it and restart it.
X. Why APPS schema present in the 11i application ??
Well it reduces traffic because all the product schemas will grant full privileges to Apps schema. so it will have full access to the complete Oracle application.
All the products like AP,AR,FIN has the code for respective products and APPS will have access to all these code objectives too. Apps will have all the synonyms for base tables and sequences as well.
OBJECT_NAME
——————————————————————————–
OBJECT_TYPE STATUS OWNER
—————— ——- ——————————
BEN_PL_PCP_S
SYNONYM VALID APPS
BEN_PL_PCP_TYP_S
SYNONYM VALID APPS
BOM_CTO_ORDER_DEMAND_S1
SYNONYM VALID APPS
File System

X. What all directories will be there under any “product” top. (To know more about various top’s in 11i click here)
cd $AP_TOP (Example of product top AP) (Below are various directory under each product top - ad, ap, gl..)
admin bin forms help html lib log mds media mesg out patch reports sql xml
$ Here directory “forms” contains all .fmx (compiled version of forms)
X.Where are .fmb (Forms) stored
They will be under $AU_TOP/forms/US
Note* .fmb and .fmx are different
X. When you run f60gen to compile the forms : .fmx will be put to $PRODUCT_TOP/forms/US/ of respective product of form.
X.What is stored in “mesg” directory under each product top ??
“mesg” contains language specific message files and error message for the product.
X. What is stored in “bin” directory under each product top
“bin” directory contains executable files like in $AP_TOP/bin you will see APPBCF APTZGF apxamex.ctl apxboav.ctl apxdiner.ctl apxgecmc.ctl apxusbv.ctl APXXTR
X. How to compile apps schema and when to compile??
You can use “adadmin” utility to compile apps schema (other methods like utlrp.sql exists). Usually you compile apps after application of the patches, maintanance patch, upgrade, runtime error due to AD_DDL packages or scenarios where there are invalid objects in apps schema.
X. How many database connections are allowed during fresh installation of oracle application.
100.
-
ADADMIN :- day to day use
AD Administration Main Menu
1. Generate Applications Files menu
2. Maintain Applications Files menu —Snapshot details
3. Compile/Reload Applications Database Entities menu ————-COMPILE APPS schema(invalid objects or runtime error)
4. Maintain Applications Database Entities menu
5. Change Maintenance Mode —–While patching
6. Exit AD Administration
Maintain Snapshot Information
1. List snapshots
2. Update current view snapshot
3. Create named snapshot
4. Export snapshot to file
5. Import snapshot from file
6. Delete named snapshot(s)
Hence question on maintenance snapshot can be like “What all maintain snapshot can do”
a. You can create copy of existing snapshot.
b. You can create a list of current view and named snapshot stored in your system.
c. You can update current view snapshot with any changes to the snapshot since the last update
d. You can export an existing snapshot to a file for storage or to another system.
e. You can delete the current view snapshot
And answer is
We can export/import snapshot:Delete named snapshots (You can’t delete current view snapshot):Update current view snapshot:Create n

11i Cloning (Multi Node to Single Node)

2008-10-13T14:45:00.000+05:30

Merging from Multinode to Single node environments

Author - Jaspreet Singh Anand
Version 1.0
Created - 01-08-2007

Source: PRODUCTION -- Multinode Environment (RAC Environment)
10.14.6.23 ---- FORM/WEB TIER (Primary node)
10.14.6.13 ---- CONCURRENT+DB+ADMIN TIER(Primary Concurrent Node)
10.14.6.16 ---- CONCURRENT+DB+ADMIN TIER(Secondary Concurrent Node)

Important Note: Always start the merging from WEB node and then go to remaining Admin/Concurrent Nodes

Target: TEST -- Single node (Non RAC Environment)
10.5.72.127 ---- FORM/WEB/CONCURRENT/DB/ADMIN TIER

NODE A : 10.14.6.23 (FORM/WEB TIER) PRIMARY NODE Source
NODE B : 10.14.6.13 (CONCURRENT+DB+ADMIN TIER) Source
NODE C : 10.14.6.16 (CONCURRENT+DB+ADMIN TIER) Source
NODE E : 10.5.72.127 FORM/WEB/CONCURRENT+DB+ADMIN TIER Target

SOURCE SYSTEM

First refer note 230672.1 apply all the prereq patches and then refer Note: 233428.1 Section 4 for further steps.

1.Login as ora user (10.14.6.13) NODE B
Prepare the source system database tier for cloning
$cd /appsutil/scripts/
$perl adpreclone.pl dbTier

2.Login as ora user (10.14.6.16) NODE C
Prepare the source system database tier for cloning
$cd /appsutil/scripts/
$perl adpreclone.pl dbTier

3.Login as appl user (10.14.6.23) NODE A
cd $COMMON_TOP/admin/scripts/context_name1
$perl adpreclone.pl appsTier merge

It will create clone directort in common_top/clone/contextname1

4.Login as appl user (10.14.6.13) NODE B
$cd $COMMON_TOP/admin/scripts/context_name2
$perl adpreclone.pl appltop merge

It will create APPL_TOPs direcory in common_top/clone/appl/contextname2

5.Login as appl user (10.14.6.16) NODE C
$cd $COMMON_TOP/admin/scripts/context_name3
$perl adpreclone.pl appltop merge

It will create APPL_TOPs direcory in common_top/clone/appl/contextname3

6.Copy following from Node A(Primary Forms Tier) to Node E(Target Node) by tarring

<806 ORACLE_HOME>

7.On NODE A
$nohup tar -cvf finprdappl.tar finprdappl &
$nohup tar cvf finprdcomn.tar finprdcomn &
$nohup tar -cvf finprdora.tar finprdora &

8.Copy the Tar files to Target location(As an appl user)
$cd /oraprod/oracle
$scp -rC finprdappl.tar appl@10.5.72.127:/oraprod/oracle
$scp -rC finprdcomn.tar appl@10.5.72.127:/oraprod/oracle
$scp -rC finprdora.tar appl@10.5.72.127:/oraprod/oracle

9.And then untar it by using following command
$tar -xvf finprdappl.tar
$tar -xvf finprdcomn.tar
$tar -xvf finprdora.tar

10.Copy the required files from NODE B & C(SOURCE) to NODE E(TARGET)
$COMMON_TOP/clone/appl/contextname2 (NODE B) to $COMMON_TOP/clone/appl(NODE E)
$COMMON_TOP/clone/appl/contextname3 (NODE C) to $COMMON_TOP/clone/appl(NODE E)

Login as appl user 10.14.6.13 (NODE B)
$cd /oraprod/oracle/finprdappl/clone/appl
$scp -R appl applscm@10.5.72.127:/oraprod/oracle/finprdcomn/clone

Login as appl user 10.14.6.16 (NODE C)
$cd /oraprod/oracle/finprdappl/clone/appl
$scp -R appl applscm@10.5.72.127:/oraprod/oracle/finprdcomn/clone

11.From NODE B(Source) and NODE C(Source) to NODE E(TARGET)
Copy the database tier file system
Copy the database (DBF) files from the source to the target system
Copy the source database ORACLE_HOME(9.2.0) to the target system

TARGET SYSTEM

1.On Node E (Login as ora user and run the following :- )

Use the following method if database was backup-ed using COLD BACKUP methology
$cd /appsutil/clone/bin
$perl adcfgclone.pl dbTier

Use this method if database was backup-ed using HOT BACKUP methology
$cd /appsutil/clone/bin
$perl adcfgclone.pl dbTechStack

Create the target database control files manually
Start the target system database in open mode

Run the library update script against the database
$cd /appsutil/install/
$sqlplus "/ as sysdba" @adupdlib.sql
where is "sl" for HP-UX, "so" for any other UNIX/LINUX platform
Configure the target database (the database must be open)

$cd /appsutil/clone/bin
$perl adcfgclone.pl dbconfig

Database and DB Listener should be up and running

2.Login as appl user 10.5.72.127 (NODE E)
$cd $COMMON_TOP/clone/bin
$perl adcfgclone.pl appsTier

3.Finishing Task
* Run the following tasks in adadmin for all products:
* generate JAR files
* generate message files
* relink executables
* copy files to destination
* Depending on which tier you chose as the primary node, certain files may be missing. Run adadmin to
verify files required at runtime. If any files are listed as missing files, you
must manually copy them to the merged APPL_TOP.
* Remove the temporary directory /clone/appl(NODE E) to reduce disk space usage.

Oracle 11i Cloning pre and post Steps(Basic)

2008-10-13T14:43:00.000+05:30

+++++++++++
+PRE CLONE+
+++++++++++

1.Backup Configuration Files (On Target)
----------------------------------------

A) DB Tier(RDBMS HOME) (On Target)
===============================
Take backup of 1) $ORACLE_HOME/*.env (All env files)
2) $ORACLE_HOME/dbs (Full Folder)
3) $ORACLE_HOME/appsutil/.xml
4) $TNS_ADMIN/*.ora (Full *.ora files)

B) APPS Tier(APPLICATION) - CONC/ADMIN (On Target)
===============================================
Take backup of 1)$APPL_TOP/*.env (Includes env and custom.env files as well)
2)$APPL_TOP/admin/*.* (All files under this folder)
3)$OA_HTML/bin/*.* (All files under this folder)
4)$FND_TOP/secure (All folders under this folder)
5)$FND_TOP/resource (Alls files under this folder)
6)$806_ORACLE_HOME/network/admin/$CONTEXT_NAME/*.ora
7)$806_ORACLE_HOME/*.env (All Files)
8)$IAS_ORACLE_HOME/*.env (All Files)
9)$IAS_ORACLE_HOME/network/admin/$CONTEXT_NAME/*.ora

C) APPS Tier(APPLICATION) - WEB/FORMS (On Target)
==============================================
Take backup of 1)$APPL_TOP/*.env (Includes env and custom.env files as well)
2)$APPL_TOP/admin/*.* (All files under this folder)
3)$OA_HTML/bin/*.* (All files under this folder)
4)$FND_TOP/secure (All folders under this folder)
5)$FND_TOP/resource (Alls files under this folder)
6)$806_ORACLE_HOME/network/admin/$CONTEXT_NAME/*.ora
7)$806_ORACLE_HOME/*.env (All Files)
8)$IAS_ORACLE_HOME/*.env (All Files)
9)$IAS_ORACLE_HOME/network/admin/$CONTEXT_NAME/*.ora
10)$IAS_ORACLE_HOME/Apace (Full folder)

At APPS TIER (On Source)
========================

Run adpreclone.pl

Please set PATH to perl/bin and PERL5LIB before starting

#perl adpreclone.pl appsTier (On both web/Conc Tiers)

#perl adpreclone.pl dbTier (On RDBMS HOME)

++++++++++++
+POST CLONE+
++++++++++++

a) Cold backup Stuff
---------------------------------------------
$perl ./adcfgclone.pl dbTier db_tier_context.xml
---------------------------------------------

---------------------------------------------------
$perl ./adcfgclone.pl appsTier apps_tier_context.xml
$perl ./adcfgclone.pl appsTier apps_tier_context.xml
---------------------------------------------------

1)UTL_FILE DIRECTORY
2)APPLTMP and APPLPTMP
3)AOLJTEST
4)MENU TREE VIEW BAR
5)Active Users Conc. Req.
6)Temp Tablespace
7)Custom TOPs
8)OAM Login Checks
9)Normal Login Checks
10)Conc. Req. Schedule (If any)
11)TNS_ENTRY (RDBMS_HOME)
12)Passwords (apps,sysadmin,system,sys,dbsnmp)
13)Sof Links (6iserver,jdk and etc...)
14)DB Links
15)Online Help
16)Servlets Check
17):8000/pls/DEV (FND Web Ping)
18)X-server accessibilty

19)EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;
Run AutoConfig

RAC SRVCTL Syntax

2008-10-13T14:31:00.000+05:30

Add two-node cluster to config
srvctl add database -d db9i -o /u01/app/oracle/product/9.2.0.1.0
srvctl add instance -d db9i -i db9i1 -n server01
srvctl add instance -d db9i -i db9i2 -n server02

# remove instance from cluster
srvctl remove instance -d db9i -i db9i2

# Start and stop cluster
srvctl start database -d db9i
srvctl stop database -d db9i

# Start and stop individual instance
srvctl start instance -d db9i -i db9i1
srvctl stop instance -d db9i -i db9i1

# Get status of whole database or specific instance
srvctl status database -d db9i
srvctl status instance -d db9i -i db9i1

# Get current database config
srvctl config database -d db9i

===================================================================================

[oratest@ndxdevr1 ~]$ srvctl add instance -d TEST -i TEST1 -n ICndxdevr1
[oratest@ndxdevr1 ~]$ srvctl add instance -d TEST -i TEST2 -n ICndxdevr2
[oratest@ndxdevr1 ~]$ srvctl config database -d TEST
ICndxdevr1 TEST1 /u02/app/TEST/testdb/ORA920
ICndxdevr2 TEST2 /u02/app/TEST/testdb/ORA920
[oratest@ndxdevr1 ~]$ srvctl getenv instance -d TEST -i TEST1
[oratest@ndxdevr1 ~]$ srvctl status
Usage: srvctl verb noun [options]
verbs: start|stop|status|add|remove|modify|getenv|setenv|unsetenv|config
nouns: database|instance
For detailed help on each verb and noun and its options use:
srvctl verb noun -h
[oratest@ndxdevr2 bdump]$ srvctl status database -d TEST
Instance TEST1 is running on node ICndxdevr1
Instance TEST2 is running on node ICndxdevr2

===================================================================================

srvctl start instance -d orcl -i orcl2 :Start an instance
srvctl start database -d orcl :Start all instances
srvctl stop database -d orcl :Stop all instances and close database
srvctl stop instance -d orcl -i orcl2 :Stop an instance
srvctl start service -d orcl -s rac :Start a service
srvctl stop service -d orcl -s rac :Stop a service
srvctl status service -d orcl :Check status of a service
srvctl status instance -d orcl -i orcl1 :Check an individual instance
srvctl status database -d orcl :Check status of all instances
srvctl start nodeapps -n server1 :Start gsd, vip, listener, and ons.
srvctl stop nodeapps -n server2 :Stop gsd, vip, listener, and ons

RAC init.ora (parameter file)

2008-10-13T14:28:00.001+05:30

#*.aq_tm_processes=1
*.background_dump_dest='/oracle/admin/rac/bdump'
*.cluster_database_instances=4
*.cluster_database=true
*.compatible='9.2.0.0.0'
*.control_files='/data/oradata/rac/control01.ctl','/data/oradata/rac/control02.ctl','/data/
oradata/rac/control03.ctl'
*.core_dump_dest='/oracle/admin/rac/cdump'
*.db_block_size=16384
*.db_cache_size=536870912
*.db_domain=''
*.db_file_multiblock_read_count=16
*.db_name='rac'
*.sga_max_size=2602012616
#*.dispatchers='(PROTOCOL=TCP) (SERVICE=racXDB)'
*.fast_start_mttr_target=300
*.hash_join_enabled=TRUE
*.instance_name='rac'
rac3.instance_name='rac3'
rac2.instance_name='rac2'
rac1.instance_name='rac1'
rac4.instance_name='rac4'
rac4.instance_number=4
rac3.instance_number=3
rac1.instance_number=1
rac2.instance_number=2
*.java_pool_size=115343360
*.job_queue_processes=10
*.large_pool_size=67108864
*.open_cursors=300
*.pga_aggregate_target=134217728
*.processes=150
*.query_rewrite_enabled='FALSE'
*.remote_login_passwordfile='exclusive'
*.shared_pool_size=536870912
*.sort_area_size=524288
*.star_transformation_enabled='FALSE'
rac1.thread=1
rac2.thread=2
rac3.thread=3
rac4.thread=4
*.timed_statistics=TRUE
*.undo_management='AUTO'
*.undo_retention=10800
#*.undo_tablespace='UNDOTBS1'
rac2.undo_tablespace='UNDOTBS2'
rac1.undo_tablespace='UNDOTBS1'
rac3.undo_tablespace='UNDOTBS3'
rac4.undo_tablespace='UNDOTBS4'
*.user_dump_dest='/oracle/admin/rac/udump'
nls_date_format='dd mon yyyy hh24:mi:ss'

Set Archiving "ON" onto the RAC database

2008-10-13T14:14:00.000+05:30

The ALTER DATABASE ARCHIVELOG command can only be performed if the database in mounted in exclusive mode. This means the whole clustered database must be stopped before the operation can be performed. First we set the relevant archive parameters:

ALTER SYSTEM SET log_archive_start=TRUE SCOPE=spfile;
ALTER SYSTEM SET log_archive_dest_1='location=/u01/oradata/MYSID/archive/' SCOPE=spfile;
ALTER SYSTEM SET log_archive_format='arch_%t_%s.arc' SCOPE=spfile;

Since we need to mount the database in exclusive mode we must also alter the following parameter:

ALTER SYSTEM SET cluster_database=FALSE SCOPE=spfile;

From the command line we can stop the entire cluster using:

srvctl stop database -d MYSID

With the cluster down we can connect to a single node and issue the following commands:

STARTUP MOUNT;
ARCHIVE LOG START;
ALTER DATABASE ARCHIVELOG;
ALTER SYSTEM SET cluster_database=TRUE SCOPE=spfile;
SHUTDOWN IMMEDIATE;

Notice that the CLUSTER_DATABASE parameter has been reset to it's original value. Since the datafiles and spfile are shared between all instances this operation only has to be done from a single node.

From the command line we can now start the cluster again using:

srvctl start database -d MYSID

The current settings place all archive logs in the same directory. This is acceptible since the thread (%t) is part of the archive format preventing any name conflicts between instances. If node-specific locations are required the LOG_ARCHIVE_DEST_1 parameter can be repeated for each instance with the relevant SID prefix.

RAC ON and RAC OFF

2008-10-13T14:10:00.000+05:30

Use the following steps to disable RAC (known as RAC OFF):

Log in as the Oracle software owner (which is typically the UNIX account oracle) in all nodes.
Shut down all the instances from all the nodes using a NORMAL or IMMEDIATE option.
Change the working directory to $ORACLE_HOME/rdbms/lib:

cd $ORACLE_HOME/rdbms/lib

Run the following make command to relink the Oracle binaries without the RAC option:

make -f ins_rdbms.mk rac_off

This normally runs for few minutes and should not pose any errors.

. Now relink the Oracle binaries:

make -f ins_rdbms.mk ioracle

Now the Oracle binaries are relinked with the RAC OFF option. You may have to edit the init.ora or SPFILE parameters accordingly. If errors occur in step 4, you may need to contact Oracle Support and log a service request with the trace and log files.

Use the following steps to enable RAC (known as RAC ON):

Log in as the Oracle software owner (typically the UNIX account oracle) in all nodes.
Shut down all the instances from all the nodes using a NORMAL or IMMEDIATE option.
Change the working directory to $ORACLE_HOME/rdbms/lib:

cd $ORACLE_HOME/rdbms/lib

Run the following make command to relink the Oracle binaries without the RAC option:

make -f ins_rdbms.mk rac_on

This normally runs for a few minutes and should not pose any errors.

Now relink the Oracle binaries:

make -f ins_rdbms.mk ioracle

Now the Oracle binaries are relinked with the RAC ON option. You may need to edit the init.ora or SPFILE parameters accordingly. If any errors occur in step 4, you may need to contact Oracle Support and log a service request with the trace and log files.