Monday, November 26, 2012

Disk Array Migration for ASM Databases


This blog identifies the DBA steps required to migrate databases from one disk array to another.

Identify the ASM diskgroups and their disk (LUN) sizes
Login to the ASM instance and identify the LUNs allocated to server.

SQL> SET PAGES 999
SQL> col name format a15

For 11g ASM:

SQL> select d.group_number, g.name, d.os_mb
from v$asm_disk d left outer join v$asm_diskgroup g
on (d.group_number = g.group_number)
order by g.group_number, d.os_mb;

For 10g ASM:

SQL> select d.group_number, g.name, d.total_mb
from v$asm_disk d left outer join v$asm_diskgroup g
on (d.group_number = g.group_number)
order by g.group_number, d.total_mb;

You may discard LUNs with group number 0, as they are not part of any diskgroup. Request Sysadmin team to provision the new LUNs to the container, same LUN size or total of multiple LUNs in the group is equal or more to what is allocated to the group. 

 To find out what is allocated and free in each diskgroup, you may use:

SQL> select name, total_mb, free_mb from v$asm_diskgroup;

Once LUNs are provisioned by Unix team, use OEM or SQL to add the LUNs to respective diskgroups [existing diskgroup]. NO outage required for this step. Remember to login to 11g ASM using the SYSASM privilege, for 10g login using SYSDBA privilege.

SQL> ALTER DISKGROUP ADD DISK ‘/full_path_of_device’;

To find out which LUNs are available to add, you may use SQL:

SQL> col path format a40
SQL> select path, os_mb from v$asm_disk where group_number = 0;

{for 10g ASM, please substitute os_mb with total_mb}

Then, go ahead and drop the old LUNs. No need to wait for the add operation to complete.

SQL> ALTER DISKGROUP DROP DISK name_of_old_LUN;

The name of the LUN can be found by querying the V$ASM_DISK...

SQL> select path,name from v$asm_disk where group_number=3;

Keep the rebalance power low, so that the migration activity does not impact database performance.

You may check the progress of rebalance periodically using V$ASM_OPERATION, once no rows are in this view, you may advice SYSADMIN team to pull out the old LUNs from the server. V$ASM_OPERATION also shows the work so far completed and how long it will take to complete the work.


Wednesday, November 7, 2012

SYSAUX Tablespace Growth - Review and Fix

Received an alert from one of the databases that SYSAUX tablespace is nearing its allocated 24GB space. 24GB is too much for SYSAUX, especially on a development database... so need to make sure the contents are reviewed before adding more space...

Oracle10g+ versions have a view V$SYSAUX_OCCUPANTS, showing what components are in SYSAUX tablespace and how much space is used. It also shows the procedure to use if you decide to move this component from SYSAUX to another tablespace.



SQL> col OCCUPANT_NAME format a20
SQL> select occupant_name, space_usage_kbytes


     from v$sysaux_occupants
     where space_usage_kbytes > 10240
     order by space_usage_kbytes desc;

OCCUPANT_NAME        SPACE_USAGE_KBYTES
-------------------- ------------------
SM/AWR                          2875776
LOGMNR                           573184
SM/ADVISOR                       448384
XDB                              177216
SDO                               79936
STREAMS                           76928
AO                                49536
XSOQHIST                          49536
XSAMD                             31872
SM/OTHER                          17984
TEXT                              14720

11 rows selected.

SQL>

So, it really does not add up anywhere near 24GB, the largest component is AWR and is only occupying less than 3GB... What else is in SYSAUX, that does no belong to the standard Oracle components?


SQL> col SEGMENT_NAME format a25
SQL> col segment_type format a10
SQL> col bytes format "999,999,999,999"
SQL> select segment_name, bytes 
     from dba_segments 
     where tablespace_name = 'SYSAUX'
     and bytes > 1048576000 order by bytes;

SEGMENT_NAME                         BYTES
------------------------- ----------------
SYS_LOB0002516712C00008$$   20,408,434,688

1 row selected.

SQL>

It's a LOB segment, let us find out which table...

SQL> select table_name, column_name
  2  from dba_lobs
  3  where segment_name = 'SYS_LOB0002516712C00008$$';

TABLE_NAME
------------------------------
COLUMN_NAME
---------------------------------------------
STREAMS$_APPLY_SPILL_MSGS_PART
MESSAGE

Aaaha! We recently enabled Change Data Capture feature on this database, and it is using STREAMS... Now, need to see what's causing it to grow and how to fix...!

Also, shows that V$SYSAUX_OCCUPANTS do not consider the LOB segments, when calculating space occupied  :-(





Friday, September 28, 2012

Time Flies!

Just thought that I have not updated the blog for sometime, but did not realize it is almost 10 months since the last update... Time really flies...

Oracle Openworld is here, though this is my sixth or seventh OOW at SFO, 2012 is the first OOW where I am part of the event outside of participation...

Will be presenting on Thu 10/4 - EBS performance improvement tips... I know it is the last day of the conference and many may not stay back to attend the afternoon sessions in particular... If you are around and you are supporting an Oracle EBS instance [11i or R12], I think you would learn a thing or two from this presentation. As the title of the presentation "Improving Performance of E-Business Suite Application - Practical Tips from a DBA's Diary..." says, the presentation is full of proven tips and techniques. If you happen to attend, please provide me feedback...

I will be at the Oracle Certification Lounge on Wed 10/3 at 1PM for a meet the author session. If you have any certification related questions or need guidance, don't forget to stop by... you may also discuss the contents of  my 11g OCA certification exam study guide, and how it can help you get certified...

Also, I will be available most of the Exhibition hall hours at the OneNeck booth [415 Moscone South]... Stop by to say hello, or if you need to have a quick chat on architecture, tuning,  upgrades, administration or any topic related to EBS system administration or Oracle administration... No Exadata or EM12c yet :-(

Have a great day!


Tuesday, January 3, 2012

Sequences Nearing Limit

Happy New Year 2012!

During the holidays we hit a production issue on the EBS instance. It was one of the Sequences reaching the maximum limit, but took a lot of time for our admins and Oracle support to figure out where the issue was... If we had the following SQL as part of the monitoring jobs, could have avoided the issue altogether...


SELECT sequence_owner,
       sequence_name,
       last_number,
       max_value,
       cache_size
FROM   dba_sequences
WHERE  last_number > max_value - CASE
                                   WHEN max_value > 50000 THEN 10000
                                   ELSE 200
                                 END
       AND cycle_flag = 'N'
       AND max_value != -1; 

This was added to monitoring tool immediately... :-)


Sunday, November 20, 2011

Oracle EBS Housekeeping Jobs

For a long time, I searched for a comprehensive list of housekeeping jobs [Standard EBS jobs provided by Oracle] that should be scheduled periodically on the EBS instance. Since I could not find a single source, came up with my own after a lot of reading... There may be more, but these should be scheduled on the EBS to keep the instance performing healthy... The arguments and run interval may need to be modified to suit the workload and business requirement... but this will get you started... Few jobs may need multiple scheduling with different parameters...

Job       : Gather Schema Statistics 
Executable: FNDGSCST 
Frequency : Daily 
Parameters: ALL, 10, , NOBACKUP, , LASTRUN, GATHER AUTO, , N 
Frequency : Weekly 
Parameters: ALL, 10, 4, NOBACKUP, , LASTRUN, GATHER, , Y  


Job       : JTF Item InterMedia Index Optimizing operation
Executable: JTFOPTI 
Frequency : Weekly 
Parameters: FULL, 2 


Job       : JTF Item InterMedia Index Sync Operation
Executable: JTFSYNC 
Frequency : Weekly 
Parameters: (None) 


Job       : MES InterMedia Index Sync Operation
Executable: AMVSYNC 
Frequency : Weekly 
Parameters: (None) 


Job       : OCM InterMedia Index Synchronizing for Attribute Bundles 
Executable: IBCSYNCATTR 
Frequency : Weekly 
Parameters: (None) 


Job       : Purge Debug Log and System Alerts FNDLGPRG 
Frequency : Weekly 
Parameters: (Date - 30 days)


Job       : Purge FND_STATS History Records 
Executable: FNDPGHST 
Frequency : Weekly 
Parameters: DATE, 01-Jan-00, 30-JUN-11 


Job       : Purge Obsolete Generic File Manager Data 
Executable: FNDGFMPR Daily 
Parameters: Y, , 


Job       : Purge Obsolete Workflow Runtime Data FNDWFPR 
Frequency : Weekly
Parameters: , , 60, PERM, N, 500, N Weekly , , 60, TEMP, N, 500, N 


Job       : Purge Signon Audit data FNDSCPRG 
Frequency : Weekly 
Parameters: (Date - 60 days)


Job       : Workflow Agent Activity Statistics Concurrent Program  
Executable: FNDWFAASTATCC 
Frequency : Daily 
Parameters: (None) 


Job       : Workflow Background Process 
Executable: FNDWFBG 
Frequency : 12 HOURS 
Parameters: , , , N, N, Y 
Frequency : 1 HOUR
Parameters: , , , N, Y, N 
Frequency : 5 MINUTES 
Parameters: , , , Y, N, N 


Job       : Workflow Control Queue Cleanup 
Executable: FNDWFBES_CONTROL_QUEUE_CLEANUP 
Frequency :  12 HOURS 
Parameters: (None) 


Job       : Workflow Directory Services User/Role Validation
Executable: FNDWFDSURV 
Frequency : Daily 
Parameters: 10000, Y, Y, Y 


Job       : Workflow Mailer Statistics Concurrent Program
Executable: FNDWFMLRSTATCC 
Frequency : Daily 
Parameters: (None) 


Job       : Workflow Work Items Statistics Concurrent Program
Executable: FNDWFWITSTATCC 
Frequency : Daily 
Parameters: (None)  


Enjoy!