Oracle ASM Cluster File System – Das lang erwartete fehlende Feature

• S309036 Oracle ASM Cluster File System: The Awaited Missing Feature
• S309060 Edition-Based Redefinition: The Key to Minimal-Downtime Application Upgrades

If you are interested in above topics, come and visit the presentations. There are still some seats available.

See you at the Oracle Open World,

Matthias

root@node1:/root# /opt/oracle/product/10.2.0/crs/install/root102.sh
Creating pre-patch directory for saving pre-patch clusterware files
Completed patching clusterware files to /opt/oracle/product/10.2.0/crs
Relinking some shared libraries.
/tmp/ccTBYA0i.s: Assembler messages:
/tmp/ccTBYA0i.s:43: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:44: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:45: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:46: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:47: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:48: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:49: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:50: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:51: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:52: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:53: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:54: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:55: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:56: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:57: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:58: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:59: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:60: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:61: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:62: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:63: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:64: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:65: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:66: Error: can not do 8 byte relocation
mv: cannot stat `ntcontab.o': No such file or directory
/usr/bin/ar: /opt/oracle/product/10.2.0/crs/lib/ntcontab.o: No such file or directory
make: *** [ntcontab.o] Error 1
/tmp/ccrcYDIC.s: Assembler messages:
/tmp/ccrcYDIC.s:28: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:29: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:30: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:31: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:32: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:33: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:34: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:35: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:36: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:37: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:38: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:39: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:40: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:41: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:42: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:43: Error: can not do 8 byte relocation
mv: cannot stat `nnfgt.o': No such file or directory
/usr/bin/ar: /opt/oracle/product/10.2.0/crs/lib/nnfgt.o: No such file or directory
make: *** [nnfgt.o] Error 1
Relinking of patched files is complete.
WARNING: directory '/opt/oracle/product/10.2.0' is not owned by root
WARNING: directory '/opt/oracle/product' is not owned by root
WARNING: directory '/opt/oracle' is not owned by root
Preparing to recopy patched init and RC scripts.
Recopying init and RC scripts.
ocrcheck failed. Check /opt/oracle/product/10.2.0/crs/srvm/log for more details
root@node1:/root#

The relink for “ntcontab.o” and “nnfgt.o” failed with a strange error message. During the analyze, I discovered that I was able to relink the two files manually as the oracle user by using the following commands:

% cd $ORA_CRS_HOME/network/lib
% make -f ins_net_client.mk ntcontab.o
% make -f ins_net_client.mk nnfgt.o

So the problem had to be related to the root user. Finally the cause was clarified. The system administrators had compiled GNU binaries (in this case gcc) and had configured the PATH variable to use the self compiled binaries first. When removing the self compiled binaries from the PATH variable, we were also able to relink “ntcontab.o” and “nnfgt.o” as the root user.

Additionally we performed a relink of the CRS client shared libraries following Note 743649.1:

% cd $ORA_CRS_HOME/bin
% ./genclntsh

Afterwards the reexecution of root102.sh was successfull.

References:

• Note 743649.1 on how to relink the client shared libraries of the CRS
• Note 438747.1 on how to relink nnfgt.o and ntcontab.o

SQL> show parameter audit

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
audit_file_dest                      string      /opt/oracle/admin/MYSID/adump
audit_sys_operations                 boolean     FALSE
audit_syslog_level                   string
audit_trail                          string      NONE
SQL> !ls -al /opt/oracle/admin/MYSID/adump

total 20
drwxr-xr-x 2 oracle oinstall 2064 2009-04-02 10:51 .
drwxr-xr-x 8 oracle oinstall  192 2008-03-13 01:35 ..
-rw-r----- 1 oracle oinstall  618 2009-04-02 10:36 ora_11430.aud
-rw-r----- 1 oracle oinstall  618 2009-04-02 10:36 ora_11431.aud
-rw-r----- 1 oracle oinstall  618 2009-04-02 10:36 ora_11432.aud
-rw-r----- 1 oracle oinstall  618 2009-04-02 10:36 ora_11433.aud
-rw-r----- 1 oracle oinstall  618 2009-04-02 10:36 ora_11434.aud
-rw-r----- 1 oracle oinstall  618 2009-04-02 10:36 ora_11435.aud
-rw-r----- 1 oracle oinstall  618 2009-04-02 10:36 ora_11505.aud
-rw-r----- 1 oracle oinstall  618 2009-04-02 10:36 ora_11512.aud
-rw-r----- 1 oracle oinstall  612 2009-04-02 10:45 ora_18425.aud
-rw-r----- 1 oracle oinstall  612 2009-04-02 10:45 ora_18427.aud
...
...

This is an expected behaviour which is also described in in note 308066.1 AUDIT_SYS_OPERATIONS Set To FALSE Yet Audit Files Are Generated. Events for generation of audit file entries are:

• Connections to the instance with administrator privileges (SYSDBA/SYSOPER)
• Database shutdown
• Database startup

 oracle@dbserver1:~/admin/MYSID/adump> rm *.aud
-bash: /bin/rm: Argument list too long

To cleanup the files without using ranges, you can use the following shell code:

• Listing files to delete

oracle@dbserver1:~/admin/MYSID/adump> for i in `ls | grep ".aud"`; do echo $i; done

• Removing files

oracle@dbserver1:~/admin/MYSID/adump> for i in `ls | grep “.aud”`; do echo $i; rm $i; done

DBAs can use bbed to simulate block corruptions and how to fix them. The following steps will describe how to compile and start bbed on a Linux server.

Linking

Oracle didn’t changed the necessary step since version 8. So for an up to date 10.2  installation you have to call “make -f ins_rdbms.mk $ORACLE_HOME/rdbms/lib/bbed” in the $ORACLE_HOME/rdbms/lib directory:

oracle@myserv:/opt/oracle/product/10.2.0/db_1/rdbms/lib> make -f ins_rdbms.mk $ORACLE_HOME/rdbms/lib/bbed
Linking BBED utility (bbed)
rm -f /opt/oracle/product/10.2.0/db_1/rdbms/lib/bbed
gcc -o /opt/oracle/product/10.2.0/db_1/rdbms/lib/bbed -L/opt/oracle/product/10.2.0/db_1/rdbms/lib/ -L/opt/oracle/product/10.2.0/db_1/lib/ -L/opt/oracle/product/10.2.0/db_1/lib/stubs/ /opt/oracle/product/10.2.0/db_1/lib/s0main.o /opt/oracle/product/10.2.0/db_1/rdbms/lib/ssbbded.o /opt/oracle/product/10.2.0/db_1/rdbms/lib/sbbdpt.o `cat /opt/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 /opt/oracle/product/10.2.0/db_1/rdbms/lib/defopt.o -ldbtools10 -lclntsh `cat /opt/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /opt/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lclient10 -lnnetd10 -lvsn10 -lcommon10 -lgeneric10 -lmm -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /opt/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /opt/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lclient10 -lnnetd10 -lvsn10 -lcommon10 -lgeneric10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lclient10 -lnnetd10 -lvsn10 -lcommon10 -lgeneric10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /opt/oracle/product/10.2.0/db_1/lib/sysliblist` -Wl,-rpath,/opt/oracle/product/10.2.0/db_1/lib -lm `cat /opt/oracle/product/10.2.0/db_1/lib/sysliblist` -ldl -lm -L/opt/oracle/product/10.2.0/db_1/lib

After this you can move the binary to the bin-directory.

oracle@myserv:/opt/oracle/product/10.2.0/db_1/rdbms/lib> mv bbed ../../bin/.

Starting bbed

As bbed is an Oracle internal only utility and can cause major harm on datafiles when used by an unexperienced person, Oracle protected the startup with a password:

oracle@myserv:/opt/oracle/product/10.2.0/db_1> bbed
Password:
BBED: Release 2.0.0.0.0 - Limited Production on Sun Sep 7 13:22:13 2008
Copyright (c) 1982, 2005, Oracle. All rights reserved.
************* !!! For Oracle Internal Use only !!! ***************
BBED>

The password can already be found on some websites, so I decided to put it into my article too. It is “blockedit”, but remember to use it carefully.

List of commands

After starting bbed you can get an overview of the available commands with “help all”:

BBED> help all
SET DBA [ dba | file#, block# ]
SET FILENAME 'filename'
SET FILE file#
SET BLOCK [+/-]block#
SET OFFSET [ [+/-]byte offset | symbol | *symbol ]
SET BLOCKSIZE bytes
SET LIST[FILE] 'filename'
SET WIDTH character_count
SET COUNT bytes_to_display
SET IBASE [ HEX | OCT | DEC ]
SET OBASE [ HEX | OCT | DEC ]
SET MODE [ BROWSE | EDIT ]
SET SPOOL [ Y | N ]
SHOW [ | ALL ]
INFO
MAP[/v] [ DBA | FILENAME | FILE | BLOCK ]
DUMP[/v] [ DBA | FILENAME | FILE | BLOCK | OFFSET | COUNT ]
PRINT[/x|d|u|o|c] [ DBA | FILE | FILENAME | BLOCK | OFFSET | symbol | *symbol ]
EXAMINE[/Nuf] [ DBA | FILE | FILENAME | BLOCK | OFFSET | symbol | *symbol ]:
</Nuf>:
N - a number which specifies a repeat count.
u - a letter which specifies a unit size:
b - b1, ub1 (byte)
h - b2, ub2 (half-word)
w - b4, ub4(word)
r - Oracle table/index row
f - a letter which specifies a display format:
x - hexadecimal
d - decimal
u - unsigned decimal
o - octal
c - character (native)
n - Oracle number
t - Oracle date
i - Oracle rowid
FIND[/x|d|u|o|c] numeric/character string [ TOP | CURR ]
COPY [ DBA | FILE | FILENAME | BLOCK ] TO [ DBA | FILE | FILENAME | BLOCK ]
MODIFY[/x|d|u|o|c] numeric/character string
      [ DBA | FILE | FILENAME | BLOCK | OFFSET | symbol | *symbol ]
ASSIGN[/x|d|u|o] <target spec>=<source spec>
<target spec> : [ DBA | FILE | FILENAME | BLOCK | OFFSET | symbol | *symbol ]
<source spec> : [ value | <target spec options> ]
SUM [ DBA | FILE | FILENAME | BLOCK ] [ APPLY ]
PUSH [ DBA | FILE | FILENAME | BLOCK | OFFSET ]
POP [ALL]
REVERT [ DBA | FILE | FILENAME | BLOCK ]
UNDO
HELP [ | ALL ]
VERIFY [ DBA | FILE | FILENAME | BLOCK ]
CORRUPT [ DBA | FILE | FILENAME | BLOCK ] 
BBED>

If you would like to get some more information on how to use bbed and how to disassemble Oracle Data Blocks, you can have a look at the following whitepaper: Disassembling the Oracle Data Block by Graham Thornton

alter system set event = '10298 trace name context forever, level 32' scope=spfile;
shutdown immediate;
startup;

This will disable the NFS checks and will allow the usage of NFS in problematic situations (like when you are experiencing problems with rman as described in Note 356199.1 RMAN Backup On Nfs Filsystem Auto Mounted With Correct Options Still Fails Ora-27054.)

Caution: Always be careful with internal events!

[oracle@myserver nfsexp]$ expdp userid=system parfile=schema_expdp.pfile

Export: Release 10.2.0.3.0 - 64bit Production on Sunday, 01 February, 2009 1:15:25

Copyright (c) 2003, 2005, Oracle.  All rights reserved.
Password:

Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

If you are analyzing the data pump master session you will encounter an endless wait on “wait for unread message on broadcast channel”:

02:13:52 SQL> col module for a16
02:14:01 SQL> col program for a30
02:14:07 SQL> col username for a10
02:14:17 SQL> select sid, serial#, username, program, module, event, seconds_in_wait from v$session where username = 'SYSTEM';

SID    SERIAL# USERNAME     PROGRAM            MODULE
---------- ---------- ---------- ------------------------------ ----------------
EVENT                                 SECONDS_IN_WAIT
---------------------------------------------------------------- ---------------
5310    59497 SYSTEM     oracle@myserver (DM02) Data Pump Master
wait for unread message on broadcast channel                    3545

Additionally the dump file stucks at a size of 4K and no logfile messages will be written:

[oracle@myserver nfsexp]$ ll
total 2
-rw-r-----  1 oracle oinstall 4096 Feb  1 01:15 schema.dmp
-rw-r--r--  1 oracle oinstall    0 Feb  1 01:15 schema_expdp.log
[oracle@myserver nfsexp]$

You can only abort the expdp utility with CTRL-C and try to cleanup the orphaned data pump job like described in note 336014.1 How To Cleanup Orphaned DataPump Jobs In DBA_DATAPUMP_JOBS?. Unfortunately a cleanup of the orphaned data pump job was not possible like described in this note. Neither the stop_job was working nor a kill -9 on the dm02 was working. It seems that bouncing the instance is the only way to get rid of the data pump job in a “DEFINING” state.

A bug regarding the NFS expdp issue is filled at Oracle under the number 6433651 JOB HANGS IN DEFINING STATUS WITH FILE ON 4K SIZE USING NFS MOUNT POINT. Unfortunately till now no fix or workaround is available, as the event “10298 trace name context forever, level 32″ for deactivating the NFS mount option checking has no effect on this problem. I’ll keep an eye on this issue and will update this entry on new findings.

To obtain the latest version of the utility use the following link to download it in a zip-file format: strmmon.zip

Compilation on systems prior 10gR2:

1.) Download the latest version
2.) extract the version on your database server in a strmmon directory
3.) make sure that you have set ORACLE_HOME and LD_LIBRARY_PATH correctly
4.) compile the binary inside the directory strmmon using “make -f strmmon.mk strmmon”

Compilation on 10gR2 systems:

1.) Download the latest version
2.) extract the version on your database server in a strmmon directory
3.) copy the files strmmon.c and strmmon.html to your $ORACLE_HOME/rdbms/demo directory
4.) cd $ORACLE_HOME/rdbms/demo
5.) make -f rdbms_demo.mk strmmon

Example on 10.2.0.1:

oracle@orasrv:~$ mkdir strmmon
oracle@orasrv:~$ cd strmmon/
oracle@orasrv:~/strmmon$ mv /tmp/strmmon.zip .
oracle@orasrv:~/strmmon$ unzip strmmon.zip
Archive:  strmmon.zip
inflating: strmmon.mk
inflating: strmmon.htm
inflating: strmmon.c
oracle@orasrv:~/strmmon$ cp strmmon.c $ORACLE_HOME/rdbms/demo
oracle@orasrv:~/strmmon$ cp strmmon.htm $ORACLE_HOME/rdbms/demo
oracle@orasrv:~/strmmon$ cd $ORACLE_HOME/rdbms/demo
oracle@orasrv:/u01/app/oracle/product/10.2.0/db_1/rdbms/demo$ make -f demo_rdbms.mk strmmon
make -f /u01/app/oracle/product/10.2.0/db_1/rdbms/demo/demo_rdbms.mk build EXE=strmmon OBJS=strmmon.o
make[1]: Entering directory `/u01/app/oracle/product/10.2.0/db_1/rdbms/demo'
/usr/bin/gcc -c -I/u01/app/oracle/product/10.2.0/db_1/rdbms/demo -I/u01/app/oracle/product/10.2.0/db_1/rdbms/public -I/u01/app/oracle/product/10.2.0/db_1/plsql/public -I/u01/app/oracle/product/10.2.0/db_1/network/public -I/u01/app/oracle/product/10.2.0/db_1/precomp/public strmmon.c
/usr/bin/gcc -L/u01/app/oracle/product/10.2.0/db_1/lib/ -L/u01/app/oracle/product/10.2.0/db_1/rdbms/lib/ -o strmmon strmmon.o -lclntsh   `cat /u01/app/oracle/product/10.2.0/db_1/lib/sysliblist` -ldl -lm
make[1]: Leaving directory `/u01/app/oracle/product/10.2.0/db_1/rdbms/demo'
oracle@r03pc00:/u01/app/oracle/product/10.2.0/db_1/rdbms/demo$

Execution example:

oracle@orasrv:/u01/app/oracle/product/10.2.0/db_1/rdbms/demo$ ./strmmon
Usage: strmmon -interval <seconds> -count <number> [-user <user name>]
[-passw <password>] [-dbname <database name>] [-sysdba]
[-long]

oracle@orasrv:/u01/app/oracle/product/10.2.0/db_1/rdbms/demo$ ./strmmon -interval 1 -count 5 -user strmadmin -passw SOME_PW -dbname a.oracle.com

STREAMS Monitor, v 2.6  Copyright Oracle Corp. 2002, 2005.
Interval = 1, Count=5

Logon=strmadmin@a.oracle.com ORACLE 10.2.0.1.0

Streams Pool Size = 132M

NET: <client bytes per sec> <dblink bytes per sec>
Cxxx: <lcrs captured per sec> <lcrs enqueued per sec> <capture latency>
PRxx: <messages received per sec>
Qx  : <msgs enqueued per sec> <msgs spilled per sec>
PSxx: <lcrs propagated per sec> <bytes propaged per sec>
Axxx: <lcrs applied per sec> <txns applied per sec> <dequeue latency>
<F>: flow control in effect
<B>: potential bottleneck
AR: apply reader
AS(n): n number of apply server
<x%I x%F x%xx>: <idle wait events percentage> <flow control wait events percentage> <other wait event percentage and name>
xx->: database instance name

2009-01-15 16:14:18 || orcl-> |      |
2009-01-15 16:14:19 || orcl-> | NET 16K 0 | <B> C002 0 0 0sec <97%I 0%F ->  |  -  C001 6 0 0sec <97%I 0%F ->  | Q54516 0 0 | PS01 0 0 0 | Q53876 0 0 | PS02 0 0 0
2009-01-15 16:14:20 || orcl-> | NET 13K 0 | <B> C002 0 0 0sec <97%I 0%F ->  |  -  C001 0 0 0sec <97%I 0%F ->  | Q54516 0 0 | PS01 0 0 0 | Q53876 0 0 | PS02 0 0 0
2009-01-15 16:14:21 || orcl-> | NET 13K 0 |  -  C002 6 0 3sec <100%I 0%F ->  |  -  C001 0 0 0sec <97%I 0%F ->  | Q54516 0 0 | PS01 0 0 0 | Q53876 0 0 | PS02 0 0 0
2009-01-15 16:14:23 || orcl-> | NET 6K 0 |  -  C002 0 0 3sec <48%I 0%F ->  |  -  C001 0 0 0sec <48%I 0%F ->  | Q54516 0 0 | PS01 0 0 0 | Q53876 0 0 | PS02 0 0 0
oracle@r03pc00:/u01/app/oracle/product/10.2.0/db_1/rdbms/demo$

References:

• Metalink Note 290605.1: Streams Monitoring Program

event="10269 trace name context forever, level 10"

It is also possible to prevent SMON from free space coalescing by setting the PCTINCREASE value on effected tablespaces and segments to 0, but if you need a general solution you might use the described event.