I wrote this article based on my beta tests of the Oracle ASM Cluster File System back in 2009 for the German Oracle User Group (DOAG) magazine. It’s written in German and introduces this awaited missing feature of Oracle ASM.
Oracle ASM Cluster File System – Das lang erwartete fehlende Feature
I was invited to hold the following sessions at the Oracle Open World 2009:
If you are interested in above topics, come and visit the presentations. There are still some seats available.
See you at the Oracle Open World,
Matthias
While performing the setup of an Oracle 10gR2 Real Application Cluster on Oracle Enterprise Linux 5 Update 3, I came across this issue. After applying the 10.2.0.4 patchset on the CRS binaries successfully, I had to stop the CRS on the nodes and execute the root102.sh. While executing the root102.sh script as the root user the following problem was encountered:
root@node1:/root# /opt/oracle/product/10.2.0/crs/install/root102.sh
Creating pre-patch directory for saving pre-patch clusterware files
Completed patching clusterware files to /opt/oracle/product/10.2.0/crs
Relinking some shared libraries.
/tmp/ccTBYA0i.s: Assembler messages:
/tmp/ccTBYA0i.s:43: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:44: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:45: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:46: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:47: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:48: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:49: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:50: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:51: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:52: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:53: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:54: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:55: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:56: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:57: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:58: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:59: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:60: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:61: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:62: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:63: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:64: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:65: Error: can not do 8 byte relocation
/tmp/ccTBYA0i.s:66: Error: can not do 8 byte relocation
mv: cannot stat `ntcontab.o': No such file or directory
/usr/bin/ar: /opt/oracle/product/10.2.0/crs/lib/ntcontab.o: No such file or directory
make: *** [ntcontab.o] Error 1
/tmp/ccrcYDIC.s: Assembler messages:
/tmp/ccrcYDIC.s:28: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:29: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:30: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:31: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:32: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:33: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:34: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:35: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:36: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:37: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:38: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:39: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:40: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:41: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:42: Error: can not do 8 byte relocation
/tmp/ccrcYDIC.s:43: Error: can not do 8 byte relocation
mv: cannot stat `nnfgt.o': No such file or directory
/usr/bin/ar: /opt/oracle/product/10.2.0/crs/lib/nnfgt.o: No such file or directory
make: *** [nnfgt.o] Error 1
Relinking of patched files is complete.
WARNING: directory '/opt/oracle/product/10.2.0' is not owned by root
WARNING: directory '/opt/oracle/product' is not owned by root
WARNING: directory '/opt/oracle' is not owned by root
Preparing to recopy patched init and RC scripts.
Recopying init and RC scripts.
ocrcheck failed. Check /opt/oracle/product/10.2.0/crs/srvm/log for more details
root@node1:/root#
The relink for “ntcontab.o” and “nnfgt.o” failed with a strange error message. During the analyze, I discovered that I was able to relink the two files manually as the oracle user by using the following commands:
% cd $ORA_CRS_HOME/network/lib
% make -f ins_net_client.mk ntcontab.o
% make -f ins_net_client.mk nnfgt.o
So the problem had to be related to the root user. Finally the cause was clarified. The system administrators had compiled GNU binaries (in this case gcc) and had configured the PATH variable to use the self compiled binaries first. When removing the self compiled binaries from the PATH variable, we were also able to relink “ntcontab.o” and “nnfgt.o” as the root user.
Additionally we performed a relink of the CRS client shared libraries following Note 743649.1:
% cd $ORA_CRS_HOME/bin
% ./genclntsh
Afterwards the reexecution of root102.sh was successfull.
References:
Even if the Oracle Documentation tells that the FALSE setting for the parameter AUDIT_SYS_OPERATIONS disables SYS auditing, it will be performed:
SQL> show parameter audit
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
audit_file_dest string /opt/oracle/admin/MYSID/adump
audit_sys_operations boolean FALSE
audit_syslog_level string
audit_trail string NONE
SQL> !ls -al /opt/oracle/admin/MYSID/adump
total 20
drwxr-xr-x 2 oracle oinstall 2064 2009-04-02 10:51 .
drwxr-xr-x 8 oracle oinstall 192 2008-03-13 01:35 ..
-rw-r----- 1 oracle oinstall 618 2009-04-02 10:36 ora_11430.aud
-rw-r----- 1 oracle oinstall 618 2009-04-02 10:36 ora_11431.aud
-rw-r----- 1 oracle oinstall 618 2009-04-02 10:36 ora_11432.aud
-rw-r----- 1 oracle oinstall 618 2009-04-02 10:36 ora_11433.aud
-rw-r----- 1 oracle oinstall 618 2009-04-02 10:36 ora_11434.aud
-rw-r----- 1 oracle oinstall 618 2009-04-02 10:36 ora_11435.aud
-rw-r----- 1 oracle oinstall 618 2009-04-02 10:36 ora_11505.aud
-rw-r----- 1 oracle oinstall 618 2009-04-02 10:36 ora_11512.aud
-rw-r----- 1 oracle oinstall 612 2009-04-02 10:45 ora_18425.aud
-rw-r----- 1 oracle oinstall 612 2009-04-02 10:45 ora_18427.aud
...
...
This is an expected behaviour which is also described in in note 308066.1 AUDIT_SYS_OPERATIONS Set To FALSE Yet Audit Files Are Generated. Events for generation of audit file entries are:
If you want to cleanup the audit_file_dest, you may came across the problem that there are too many files located inside the directory to delete them in a single command:
oracle@dbserver1:~/admin/MYSID/adump> rm *.aud
-bash: /bin/rm: Argument list too long
To cleanup the files without using ranges, you can use the following shell code:
oracle@dbserver1:~/admin/MYSID/adump> for i in `ls | grep ".aud"`; do echo $i; done
oracle@dbserver1:~/admin/MYSID/adump> for i in `ls | grep “.aud”`; do echo $i; rm $i; done
bbed stands for Block Browser and EDitor and is an Oracle internal utility for directly accessing data blocks in data files. It can be used to show the content of a block or even change it. This powerful features should be used CAREFULLY as you can end up with corrupt datafiles/database. That’s why it should never be used in production systems unless for good reason.
DBAs can use bbed to simulate block corruptions and how to fix them. The following steps will describe how to compile and start bbed on a Linux server.
Linking
Oracle didn’t changed the necessary step since version 8. So for an up to date 10.2 installation you have to call “make -f ins_rdbms.mk $ORACLE_HOME/rdbms/lib/bbed” in the $ORACLE_HOME/rdbms/lib directory:
oracle@myserv:/opt/oracle/product/10.2.0/db_1/rdbms/lib> make -f ins_rdbms.mk $ORACLE_HOME/rdbms/lib/bbed
Linking BBED utility (bbed)
rm -f /opt/oracle/product/10.2.0/db_1/rdbms/lib/bbed
gcc -o /opt/oracle/product/10.2.0/db_1/rdbms/lib/bbed -L/opt/oracle/product/10.2.0/db_1/rdbms/lib/ -L/opt/oracle/product/10.2.0/db_1/lib/ -L/opt/oracle/product/10.2.0/db_1/lib/stubs/ /opt/oracle/product/10.2.0/db_1/lib/s0main.o /opt/oracle/product/10.2.0/db_1/rdbms/lib/ssbbded.o /opt/oracle/product/10.2.0/db_1/rdbms/lib/sbbdpt.o `cat /opt/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 /opt/oracle/product/10.2.0/db_1/rdbms/lib/defopt.o -ldbtools10 -lclntsh `cat /opt/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /opt/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lclient10 -lnnetd10 -lvsn10 -lcommon10 -lgeneric10 -lmm -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /opt/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /opt/oracle/product/10.2.0/db_1/lib/ldflags` -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lclient10 -lnnetd10 -lvsn10 -lcommon10 -lgeneric10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lclient10 -lnnetd10 -lvsn10 -lcommon10 -lgeneric10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lcore10 -lsnls10 -lnls10 -lxml10 -lcore10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /opt/oracle/product/10.2.0/db_1/lib/sysliblist` -Wl,-rpath,/opt/oracle/product/10.2.0/db_1/lib -lm `cat /opt/oracle/product/10.2.0/db_1/lib/sysliblist` -ldl -lm -L/opt/oracle/product/10.2.0/db_1/lib
After this you can move the binary to the bin-directory.
oracle@myserv:/opt/oracle/product/10.2.0/db_1/rdbms/lib> mv bbed ../../bin/.
Starting bbed
As bbed is an Oracle internal only utility and can cause major harm on datafiles when used by an unexperienced person, Oracle protected the startup with a password:
oracle@myserv:/opt/oracle/product/10.2.0/db_1> bbed
Password:
BBED: Release 2.0.0.0.0 - Limited Production on Sun Sep 7 13:22:13 2008
Copyright (c) 1982, 2005, Oracle. All rights reserved.
************* !!! For Oracle Internal Use only !!! ***************
BBED>
The password can already be found on some websites, so I decided to put it into my article too. It is “blockedit”, but remember to use it carefully.
List of commands
After starting bbed you can get an overview of the available commands with “help all”:
BBED> help all
SET DBA [ dba | file#, block# ]
SET FILENAME 'filename'
SET FILE file#
SET BLOCK [+/-]block#
SET OFFSET [ [+/-]byte offset | symbol | *symbol ]
SET BLOCKSIZE bytes
SET LIST[FILE] 'filename'
SET WIDTH character_count
SET COUNT bytes_to_display
SET IBASE [ HEX | OCT | DEC ]
SET OBASE [ HEX | OCT | DEC ]
SET MODE [ BROWSE | EDIT ]
SET SPOOL [ Y | N ]
SHOW [ | ALL ]
INFO
MAP[/v] [ DBA | FILENAME | FILE | BLOCK ]
DUMP[/v] [ DBA | FILENAME | FILE | BLOCK | OFFSET | COUNT ]
PRINT[/x|d|u|o|c] [ DBA | FILE | FILENAME | BLOCK | OFFSET | symbol | *symbol ]
EXAMINE[/Nuf] [ DBA | FILE | FILENAME | BLOCK | OFFSET | symbol | *symbol ]:
</Nuf>:
N - a number which specifies a repeat count.
u - a letter which specifies a unit size:
b - b1, ub1 (byte)
h - b2, ub2 (half-word)
w - b4, ub4(word)
r - Oracle table/index row
f - a letter which specifies a display format:
x - hexadecimal
d - decimal
u - unsigned decimal
o - octal
c - character (native)
n - Oracle number
t - Oracle date
i - Oracle rowid
FIND[/x|d|u|o|c] numeric/character string [ TOP | CURR ]
COPY [ DBA | FILE | FILENAME | BLOCK ] TO [ DBA | FILE | FILENAME | BLOCK ]
MODIFY[/x|d|u|o|c] numeric/character string
[ DBA | FILE | FILENAME | BLOCK | OFFSET | symbol | *symbol ]
ASSIGN[/x|d|u|o] <target spec>=<source spec>
<target spec> : [ DBA | FILE | FILENAME | BLOCK | OFFSET | symbol | *symbol ]
<source spec> : [ value | <target spec options> ]
SUM [ DBA | FILE | FILENAME | BLOCK ] [ APPLY ]
PUSH [ DBA | FILE | FILENAME | BLOCK | OFFSET ]
POP [ALL]
REVERT [ DBA | FILE | FILENAME | BLOCK ]
UNDO
HELP [ | ALL ]
VERIFY [ DBA | FILE | FILENAME | BLOCK ]
CORRUPT [ DBA | FILE | FILENAME | BLOCK ]
BBED>
If you would like to get some more information on how to use bbed and how to disassemble Oracle Data Blocks, you can have a look at the following whitepaper: Disassembling the Oracle Data Block by Graham Thornton
If you are experiencing problems between Oracle and NFS mounts it is possible to deactivate the NFS checks of Oracle by using the event 10298. Example:
alter system set event = '10298 trace name context forever, level 32' scope=spfile;
shutdown immediate;
startup;
This will disable the NFS checks and will allow the usage of NFS in problematic situations (like when you are experiencing problems with rman as described in Note 356199.1 RMAN Backup On Nfs Filsystem Auto Mounted With Correct Options Still Fails Ora-27054.)
Caution: Always be careful with internal events!
Today I tried to perform a data pump export with expdp on a directory located on a NFS share. Unfortunately expdp will stuck in the defining state forever:
[oracle@myserver nfsexp]$ expdp userid=system parfile=schema_expdp.pfile
Export: Release 10.2.0.3.0 - 64bit Production on Sunday, 01 February, 2009 1:15:25
Copyright (c) 2003, 2005, Oracle. All rights reserved.
Password:
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
If you are analyzing the data pump master session you will encounter an endless wait on “wait for unread message on broadcast channel”:
02:13:52 SQL> col module for a16
02:14:01 SQL> col program for a30
02:14:07 SQL> col username for a10
02:14:17 SQL> select sid, serial#, username, program, module, event, seconds_in_wait from v$session where username = 'SYSTEM';
SID SERIAL# USERNAME PROGRAM MODULE
---------- ---------- ---------- ------------------------------ ----------------
EVENT SECONDS_IN_WAIT
---------------------------------------------------------------- ---------------
5310 59497 SYSTEM oracle@myserver (DM02) Data Pump Master
wait for unread message on broadcast channel 3545
Additionally the dump file stucks at a size of 4K and no logfile messages will be written:
[oracle@myserver nfsexp]$ ll
total 2
-rw-r----- 1 oracle oinstall 4096 Feb 1 01:15 schema.dmp
-rw-r--r-- 1 oracle oinstall 0 Feb 1 01:15 schema_expdp.log
[oracle@myserver nfsexp]$
You can only abort the expdp utility with CTRL-C and try to cleanup the orphaned data pump job like described in note 336014.1 How To Cleanup Orphaned DataPump Jobs In DBA_DATAPUMP_JOBS?. Unfortunately a cleanup of the orphaned data pump job was not possible like described in this note. Neither the stop_job was working nor a kill -9 on the dm02 was working. It seems that bouncing the instance is the only way to get rid of the data pump job in a “DEFINING” state.
A bug regarding the NFS expdp issue is filled at Oracle under the number 6433651 JOB HANGS IN DEFINING STATUS WITH FILE ON 4K SIZE USING NFS MOUNT POINT. Unfortunately till now no fix or workaround is available, as the event “10298 trace name context forever, level 32″ for deactivating the NFS mount option checking has no effect on this problem. I’ll keep an eye on this issue and will update this entry on new findings.
The Streams Monitoring Utility strmmon is a very useful utility for monitoring current Streams activity. You can use it to obtain the current figures for capture, propagation and apply activity in vmstat-style.
To obtain the latest version of the utility use the following link to download it in a zip-file format: strmmon.zip
Compilation on systems prior 10gR2:
1.) Download the latest version
2.) extract the version on your database server in a strmmon directory
3.) make sure that you have set ORACLE_HOME and LD_LIBRARY_PATH correctly
4.) compile the binary inside the directory strmmon using “make -f strmmon.mk strmmon”
Compilation on 10gR2 systems:
1.) Download the latest version
2.) extract the version on your database server in a strmmon directory
3.) copy the files strmmon.c and strmmon.html to your $ORACLE_HOME/rdbms/demo directory
4.) cd $ORACLE_HOME/rdbms/demo
5.) make -f rdbms_demo.mk strmmon
Example on 10.2.0.1:
oracle@orasrv:~$ mkdir strmmon
oracle@orasrv:~$ cd strmmon/
oracle@orasrv:~/strmmon$ mv /tmp/strmmon.zip .
oracle@orasrv:~/strmmon$ unzip strmmon.zip
Archive: strmmon.zip
inflating: strmmon.mk
inflating: strmmon.htm
inflating: strmmon.c
oracle@orasrv:~/strmmon$ cp strmmon.c $ORACLE_HOME/rdbms/demo
oracle@orasrv:~/strmmon$ cp strmmon.htm $ORACLE_HOME/rdbms/demo
oracle@orasrv:~/strmmon$ cd $ORACLE_HOME/rdbms/demo
oracle@orasrv:/u01/app/oracle/product/10.2.0/db_1/rdbms/demo$ make -f demo_rdbms.mk strmmon
make -f /u01/app/oracle/product/10.2.0/db_1/rdbms/demo/demo_rdbms.mk build EXE=strmmon OBJS=strmmon.o
make[1]: Entering directory `/u01/app/oracle/product/10.2.0/db_1/rdbms/demo'
/usr/bin/gcc -c -I/u01/app/oracle/product/10.2.0/db_1/rdbms/demo -I/u01/app/oracle/product/10.2.0/db_1/rdbms/public -I/u01/app/oracle/product/10.2.0/db_1/plsql/public -I/u01/app/oracle/product/10.2.0/db_1/network/public -I/u01/app/oracle/product/10.2.0/db_1/precomp/public strmmon.c
/usr/bin/gcc -L/u01/app/oracle/product/10.2.0/db_1/lib/ -L/u01/app/oracle/product/10.2.0/db_1/rdbms/lib/ -o strmmon strmmon.o -lclntsh `cat /u01/app/oracle/product/10.2.0/db_1/lib/sysliblist` -ldl -lm
make[1]: Leaving directory `/u01/app/oracle/product/10.2.0/db_1/rdbms/demo'
oracle@r03pc00:/u01/app/oracle/product/10.2.0/db_1/rdbms/demo$
Execution example:
oracle@orasrv:/u01/app/oracle/product/10.2.0/db_1/rdbms/demo$ ./strmmon
Usage: strmmon -interval <seconds> -count <number> [-user <user name>]
[-passw <password>] [-dbname <database name>] [-sysdba]
[-long]
oracle@orasrv:/u01/app/oracle/product/10.2.0/db_1/rdbms/demo$ ./strmmon -interval 1 -count 5 -user strmadmin -passw SOME_PW -dbname a.oracle.com
STREAMS Monitor, v 2.6 Copyright Oracle Corp. 2002, 2005.
Interval = 1, Count=5
Logon=strmadmin@a.oracle.com ORACLE 10.2.0.1.0
Streams Pool Size = 132M
NET: <client bytes per sec> <dblink bytes per sec>
Cxxx: <lcrs captured per sec> <lcrs enqueued per sec> <capture latency>
PRxx: <messages received per sec>
Qx : <msgs enqueued per sec> <msgs spilled per sec>
PSxx: <lcrs propagated per sec> <bytes propaged per sec>
Axxx: <lcrs applied per sec> <txns applied per sec> <dequeue latency>
<F>: flow control in effect
<B>: potential bottleneck
AR: apply reader
AS(n): n number of apply server
<x%I x%F x%xx>: <idle wait events percentage> <flow control wait events percentage> <other wait event percentage and name>
xx->: database instance name
2009-01-15 16:14:18 || orcl-> | |
2009-01-15 16:14:19 || orcl-> | NET 16K 0 | <B> C002 0 0 0sec <97%I 0%F -> | - C001 6 0 0sec <97%I 0%F -> | Q54516 0 0 | PS01 0 0 0 | Q53876 0 0 | PS02 0 0 0
2009-01-15 16:14:20 || orcl-> | NET 13K 0 | <B> C002 0 0 0sec <97%I 0%F -> | - C001 0 0 0sec <97%I 0%F -> | Q54516 0 0 | PS01 0 0 0 | Q53876 0 0 | PS02 0 0 0
2009-01-15 16:14:21 || orcl-> | NET 13K 0 | - C002 6 0 3sec <100%I 0%F -> | - C001 0 0 0sec <97%I 0%F -> | Q54516 0 0 | PS01 0 0 0 | Q53876 0 0 | PS02 0 0 0
2009-01-15 16:14:23 || orcl-> | NET 6K 0 | - C002 0 0 3sec <48%I 0%F -> | - C001 0 0 0sec <48%I 0%F -> | Q54516 0 0 | PS01 0 0 0 | Q53876 0 0 | PS02 0 0 0
oracle@r03pc00:/u01/app/oracle/product/10.2.0/db_1/rdbms/demo$
References:
If you are still using dictionary managed tablespaces and you are dropping segments with lot of extents, you might encounter the problem that smon will start to use 100% cpu capacity and aquire the space transaction enqueue. Other processes that require this enqueue will be blocked and you might encounter ORA-1575. You can disable the automatic SMON free space coalescing by setting the following event:
event="10269 trace name context forever, level 10"
It is also possible to prevent SMON from free space coalescing by setting the PCTINCREASE value on effected tablespaces and segments to 0, but if you need a general solution you might use the described event.
