my blog has moved to a different server.

The new address is:

http://blog.ronnyegner-consulting.de

my blog has moved to a different server.

The new address is:

http://blog.ronnyegner-consulting.de

When backing up servers with any kind of backup software you will most certainly backup your data.

But how about the system configuration?

When restoring a system from scratch do you know how big the disks were, what partition sizes, what disks used (in case you have more than one disk), if you used an LVM and so on.

Think about it!

In this article i wanted to summarize what system information should be collected to enable a fast and error free restore.

Feel free to add your own comments and suggestions.

Linux

• General Information
  • Output of “hostname”
  • Output of “domainname”
  • Output of “uname -a”
  • Output of “ifconfig -a”
• Partition Information
  • Output of “fdisk -l”
  • Output of “df -h”
• Volume Manager
  • Output of “vgdisplay”
  • Output of “lvdisplay”
  • Output of “pvdisplay”
  • vgcfgbackup

Solaris

• General Information
  • Output of “hostname”
  • Output of “domainname”
  • Output of “ifconfig -a”
  • Output of “uname -a”
• Partition Information
  • “format” information for each disk
  • Output of “df -h
• Volume Manager
  • ZFS
    • Output of “zpool list”
    • Ouput of  “zpool status”
    • Output of “zfs info”
    • Output of “zfs list -o all
  • VxVM
    • Output of “vxdevlist”
    • <need some help here>
  • SVM (SDS)
    • Output of “mestastat”
  • Backup of VTOC

Oracle

• ASM
  • ASM device mapping information (for instance with this script)
• Clusterware
  • Backup of OCR
  • Backup of Voting Disk
• Database
  • “backup control file to trace”
  • (s)pfile

my blog has moved to a different server.

The new address is:

http://blog.ronnyegner-consulting.de

There is a discussion from December 2008 what Linux (SLES vs. Red vHat vs. Oracle Enterprise Linux) to use for running oracle on Linux by Yann Neuhaus. You can read his post here.

After nearly one year i wanted to catch up the article and check if the pros and cons are still valid or if there changed anything.

SuSE Linux Enterprise Server

Pros according to the original post

Very large package repository

Centralized administration tool (YaST), allowing to solve a large part of the administration tasks: disk management (including Logical Volume Manager), User handling, Network configuration, and so on …

SLES is by default delivered with OCFS2, the Oracle Cluster Filessystem. This could be useful for Failover clusters (Clusterware) or for RAC (Real Application Clusters) deployment

Heartbeat is provided by SuSE as a Failover cluster technology. Even if not officially supported by Oracle, Heartbeat offers a quite interesting protection level for other services like FTP servers, WEB servers, aso …

Cons according to the original post

YAST uses it’s own configuration files, sometimes it overwrites self-made configurations

SLES is not (anymore) the preferred/development platform for Oracle

Additional notes by Ronny Egner

• SLES10 supports 10gR2, 11g R1 and 11g R2; quite a lot of database versions supported (but not as much as Red Hat 4 does)
• SLES 11 adds xen-based virtualization

• No support of ADVM and ACFS for Oracle 11g R2 on SLES yet. Check here yourself.
• Although OCFS2 OCFS2 “>is shipped with SLES the OCFS2 module need to be updated always
• only virtualization using “Oracle VM” supported by Oracle

Red Hat

Pros according to the original post

On Red Hat 4, Oracle supports 9iR2, 10gR1, 10gR2 and 11gR1, which is not possible on SLES distributions

Satellite Server centralized update and package management tool

Very fast updates or patches in case of any vulnerability. Sometimes RedHat provides daily new updates

Cons according to the original post

Distribution has generally not the most recent available packages versions included

Compared to SuSE, less software is included

Red Hat cluster is not really on the quality level we expected. Some issues in the Oracle start stop procedures have been identified in Red Cluster 5 making the cohabitation of two Oracle instances in parallel on the same cluster impossible

In parallel we had the “opportunity” to test the support for the Red Hat Cluster technology and we had to admit that it was far from our expectations

Unfortunately OCFS2 must be separately loaded and installed (through rpm) on Red Hat (directly included into SLES)

Additional notes by Ronny Egner

• On Red Hat 4 Oracle also supports 11g R2 (in addition to already supported 9iR2,10gR1,10gR2,11gR1) –> most complete database version support in all distributions
• larger installed server base
• packages shipped with Red Hat 5 increased
• most important tools and packages ship with Red Hat 5
• missing packages can be compiled manually or downloaded as pre-compiled packages from the internet
• Virtualization support (xen-based) added in Red Hat 5

• like SuSE ADVM and ACFS shipped with Oracle 11g Release 2 are not supported with Red Hat 4 (but are supported with Red Hat 5)
• OCFS2 needs to be updated anyway…. so no real advantage over SLES from my point of view. I doubt there will be much RAC installations with OCFS2 but i dont have any figures
• only virtualization using “Oracle VM” supported by Oracle

Oracle Enterprise Linux (OEL)

Pros according to the original post

Same advantages than Red Hat, plus :

• Well configured for Oracle, favoured by Oracle and current development platform for Oracle. All in one
• OCFS2 packages missing with Red Hat are already available, this makes Clusterware even more interesting
• ASMLibs also directly available in OEL
• Based on Red Hat (but with Oracle support, which could be a benefit compared to the Red Hat support)
• Clusterware supported by Oracle as soon as OEL has been licensed. Clusterware come a serious competitor to Red Hat Cluster or Veritas cluster or even against pure Open Source solutions like Heartbeat and DRBD

Support/License pricing seems much cheaper

Cons according to the original post

Some extra (hardware provider) components could not be easily installed because this distribution is whether a Red Hat nor a SuSe and therefore unknown for some packages

Additional notes by Ronny Egner

• Same notes as for Red Hat plus:
  • as indicated in the original post if there are problems with oracle in conjunction with linux operating system support progress is far more easy with OEL compared to SLES or Red Hat because oracle will support OEL as well
  • hardware support (drivers) for OEL are now widely available; fast support of newest OEL releases by vendors
  • public available yum server for installing packages over the internet (see here)

Summary given in the original post

For pure Oracle business, Oracle Enterprise Linux (and therefore “Clusterware for Free”) seems to be the most interesting choice and Trivadis is also convinced that this would be the right Linux to deploy for Oracle based business. However do not underestimate the current Linux situation in your environment, never change a running system, if you are fully happy on SLES, use SLES, the same for Red Hat. Another issue could be that other applications beside Oracle are not OEL certified, therefore a withdraw to RHEL could be necessary.

Summary by Ronny Egner as of October 2009

As of writing this review many arguments made by Yann Neuhaus are still valid.

Especially for running a singe database instance without ASM on linux all covered distributions are equally useable.

For running single instance with ASM or running a clustered database i would definitely use either Red Hat 4/5 or – even more preferred by me – OEL. SuSE works as well but installation is not as “smooth” (many small, time consuming problems) as in Red Hat or Oracle Enterprise Linux. In addition to that the “cool” new features shipping with Oracle 11g Release 2 (ADVM and ACFS) are only available with OEL 5 or Red Hat 5. In addition to that support for OEL made by oracle might speed up problem resolution and will decrease the number of different parties involved.

As pointed out by Yann Neuhaus application certification is still important but increased during 2009. But anyway: There might be still application which are only certified on SLES or Red Hat thus fording you to use SLES or Red Hat.

Summarizing my personal :
If you stick with single instance database you can use any Linux distribution. If you want advanced features like ADVM, ACFS or Clustering (“RAC”) use Oracle Enterprise Linux. For oracle database-based applications always use the certified Linux distribution; if this is SuSE – use it; even if it is Ubuntu – use it!

my blog has moved to a different server.

The new address is:

http://blog.ronnyegner-consulting.de

The past week i had a argumentation with a colleague of mine on the behavior of the parameter undo_retention in 10g.

He stated in 10g undo_retention determines the lower number of seconds oracle keeps the undo records. If there is enough space available an ORA-01555 should not be raised.

I knew there are cases in which undo_retention is set to a ridiculous high value (for instance a week), the undo tablespace has a lot of free space but an ORA-01555 is raised nonetheless. So we took a look at the documentation:

For Oracle 10g Release 1 the documentation states:

UNDO_RETENTION specifies (in seconds) the low threshold value of undo retention. The system retains undo for at least the time specified in this parameter and automatically tunes the undo retention period to satisfy the undo requirements of the queries.

But for Oracle 10g Release 2 the documentation says different:

UNDO_RETENTION specifies (in seconds) the low threshold value of undo retention. For AUTOEXTEND undo tablespaces, the system retains undo for at least the time specified in this parameter, and automatically tunes the undo retention period to satisfy the undo requirements of the queries. For fixed- size undo tablespaces, the system automatically tunes for the maximum possible undo retention period, based on undo tablespace size and usage history, and ignores UNDO_RETENTION unless retention guarantee is enabled.

In Oracle 11g Release 1 and Oracle 11g Release 2 the behavior is the same as for Oracle 10g Release 2.

my blog has moved to a different server.

The new address is:

http://blog.ronnyegner-consulting.de

There are rumors on the release dates of 11.2.0.1.0 for different plattforms. According to my information the release dates are:

• Windows: second quarter 2010
• Linux: already released (32-bit and 64-bit intel)
• HP-UX: 4th quarter 2009
• Solaris: 4th quarter 2009
• AIX 5L based: 4th quarter 2009

I personally expect the release of 11g Release 2 for solaris operating system pretty soon.

my blog has moved to a different server.

The new address is:

http://blog.ronnyegner-consulting.de

Yesterday i did a large import using data pump and the network mode feature. I noticed a table containing a large number of rows were reported by data pump as imported but with only one (1) row.

First i checked my import options, checked for errors in source and target database and even cleaned up everything and restarted the import. Surprise surprise: Same error again.

The environment was:

• Oracle Enterprise Edition 10.2.0.4.0
• Solaris (SPARC) 10
• Data Pump Import with NETWORK_MODE
• excluding STATISTICS, GRANTS and ROLE_GRANTS

Extract  from the data pump log file:

Processing object type SCHEMA_EXPORT/CLUSTER/CLUSTER
 Processing object type SCHEMA_EXPORT/CLUSTER/INDEX
 Processing object type SCHEMA_EXPORT/TABLE/TABLE
 . . imported "USER"."TABLE_A"                 55082451 rows
 . . imported "USER"."SOME_LARGE_TABLE"       307126916 rows
 . . imported "USER"."ANOTHER_TABLE"          176936257 rows
 . . imported "USER"."JUST_ANOTHER_TABLE"     215682029 rows
 . . imported "USER"."HUGE_TABLE"                     1 rows        <====
 . . imported "USER"."AND_SO_ON"              133356302 rows

So i started to dig a little bit deeper and noticed the source table has approx 62 million rows with a size of 28 GB. To my surprise the target database also reported the table with 28 gb in DBA_SEGMENTS.

So i did a simple:

SQL> select count(1) from user.huge_table;

COUNT(1)
 ----------
 62485290

Guess what? The table was imported correctly with all rows while data pump reported only one imported row.

I check metalink for related errors but found nothing.

I can only guess if this is a bug or there is some time limit when querying the rows of the just imported tables.

Update: I did some reseach and created a really large table with 1 billion rows the data pump showed:

. . imported "USER"."TABLE_WITH_1_BILLION_ROWS"             -1416726547 rows

There might be some kind of counter overflow…. i will investiage this further.

my blog has moved to a different server.

The new address is:

http://blog.ronnyegner-consulting.de

I just found a new public yum server hosted by oracle. This server can be used to install missing packages over the internet. It does not contain any security updates or bug fixes.

The public yum server can be found here.

For larger installation you can run your own yum server.

my blog has moved to a different server.

The new address is:

http://blog.ronnyegner-consulting.de

A few users asked what to do if there is no DNS server available for configuring the SCAN names when installing oracle grid infrastructure.

Most asked if they can use hosts file entries. The short answer is: Nope. The grid infrastructure will install fine but the cluster verification utility will fail.

So my suggestion is as follows: Create your own small DNS server on your rac nodes. This is quite fast and easy and described in this article.

Installing required packages

Oracle Enterprise Linux as well as RedHat and even SuSE ship with rpm packages for the name server “bind”. Bind is developed by the ISC (Internet systems consortium) and is the most used dns server. More information can be found here.

In this article we will install bind on our rac system with oracle enterprise 5 update 3. If you use an other distribution or release your version numbers and/or package names might be different.

So for OEL 5U3 we need at least the following packages installed:

• bind-libs-9.3.4-10.P1.el5
• bind-9.3.4-10.P1.el5
• bind-utils-9.3.4-10.P1.el5

In addition to that you can also install the “bind-chroot” package which runs bind in a chroot environment. A chroot environment increases the security and is recommended for running in production. For our test system we will do not install bind chrootet to keep the complexity low.

Configuring the first (master) dns server

In the first step we will create the master DNS server which will reside on any node. In our case the master dns server resides on node “rac1.regner.de” with ip ip “>adress 172.23.15.1. The secondary dns server will be on node “rac2.regner.de” with ip address 172.23.15.2.

Later we will add a secondary DNS server on the second node. This is required for a working name resolution if one node is down.

Note that for a “quick and dirty” configuration one configured dns server is enough to complete the cluster verification tests. Make sure that you specified all node names (private, public and vip named) in your /etc/hosts file!

Creating the required files

All you need to do is to create the files in the named locations (attention: on SuSE /var/named will be probably /var/lib/named !). If you want to you can replace the string “regner.de” with your domain and correct the host names and IPs.

/etc/named.conf

[root@rac1 named]# cat /etc/named.conf
options {

 # The directory statement defines the name server's working directory

 directory "/var/named";

 # Write dump and statistics file to the log subdirectory.  The
 # pathenames are relative to the chroot jail.

 dump-file "/var/log/named_dump.db";
 statistics-file "/var/log/named.stats";

 # The forwarders record contains a list of servers to which queries
 # should be forwarded.  Enable this line and modify the IP address to
 # your provider's name server.  Up to three servers may be listed.

 #forwarders { 192.0.2.1; 192.0.2.2; };

 # Enable the next entry to prefer usage of the name server declared in
 # the forwarders section.

 #forward first;

 # The listen-on record contains a list of local network interfaces to
 # listen on.  Optionally the port can be specified.  Default is to
 # listen on all interfaces found on your system.  The default port is
 # 53.

 #listen-on port 53 { 127.0.0.1; };

 # The listen-on-v6 record enables or disables listening on IPv6
 # interfaces.  Allowed values are 'any' and 'none' or a list of
 # addresses.

 listen-on-v6 { none; };

 # The next three statements may be needed if a firewall stands between
 # the local server and the internet.

 #query-source address * port 53;
 #transfer-source * port 53;
 #notify-source * port 53;

 # The allow-query record contains a list of networks or IP addresses
 # to accept and deny queries from. The default is to allow queries
 # from all hosts.

 #allow-query { 127.0.0.1; };

 # If notify notify ">is set to yes (default), notify messages are sent to other
 # name servers when the the zone data is changed.  Instead of setting
 # a global 'notify' statement in the 'options' section, a separate
 # 'notify' can be added to each zone definition.

 notify no;
};

logging {
 # Log queries to a file limited to a size of 100 MB.
 channel query_logging {
 file "/var/named/named_querylog"
 versions 3 size 100M;
 print-time yes;                 // timestamp log entries
 };
 category queries {
 query_logging;
 };

 # Or log this kind alternatively to syslog.
 channel syslog_queries {
 syslog user;
 severity info;
 };
 category queries { syslog_queries; };

 # Log general name server errors to syslog.
 channel syslog_errors {
 syslog user;
 severity error;
 };
 category default { syslog_errors;  };

 # Don't log lame server messages.
 category lame-servers { null; };
};

acl acl_transf { 172.23.15.2; };

# The following zone definitions don't need any modification.  The first one
# is the definition of the root name servers.  The second one defines
# localhost while the third defines the reverse lookup for localhost.

#zone "." in {
# type hint;
# file "root.hint";
#};

zone "localhost" in {
 type master;
 file "localhost.zone";
};

zone "0.0.127.in-addr.arpa" in {
 type master;
 file "127.0.0.zone";
};

zone "regner.de" in {
 type master;
 file "db.regner.de";
 allow-transfer { acl_transf; };
};

Note the following line:

acl acl_transf { 172.23.15.2; };

this line will later allow the secondary dns server to fetch the zone files from the master server. In our test environment “rac1.regner.de” (172.23.15.1) is the master dns server while “rac2.regner.de” (172.23.15.2) is the secondary one which fetched the zone files. In your environemt you must adjust the line according to your used IPs. You cannot use names here!

/var/named/127.0.0.zone

$TTL 1W
@               IN SOA          localhost.   root.localhost. (
 42              ; serial (d. adams)
 2D              ; refresh
 4H              ; retry
 6W              ; expiry
 1W )            ; minimum

 IN NS           localhost.
1               IN PTR          localhost.

/var/named/localhost.zone

$TTL 1W
@               IN SOA  @   root (
 42              ; serial (d. adams)
 2D              ; refresh
 4H              ; retry
 6W              ; expiry
 1W )            ; minimum

 IN NS           @
 IN A            127.0.0.1

/var/named/db.regner.de

$ORIGIN .
regner.de               SOA     rac1.regner.de. rac1.regner.de. (
 22         ; serial
 900        ; refresh (15 minutes)
 600        ; retry (10 minutes)
 86400      ; expire (1 day)
 3600       ; minimum (1 hour)
 )
 NS      rac1.regner.de.
 NS      rac2.regner.de.
$ORIGIN regner.de.
rac1                    A       172.23.15.1
rac2                    A       172.23.15.2
rac1-vip                A       172.23.15.10
rac2-vip                A       172.23.15.20
rac1-priv               A       192.168.181.10
rac2-priv               A       192.168.181.20
rac-scan                A       172.23.15.3
rac-scan                A       172.23.15.4
rac-scan                A       172.23.15.5

Starting for the first time

Before staing for the first time we need to change permissions:

[root@rac1 named]# touch /var/named/named_querylog
[root@rac1 named]# chown -R named:named /var/named/
[root@rac1 named]# chown named:named /etc/named.conf

After that we can start for the first time by entering:

/etc/init.d/named start

You message log should look like this:

Oct 15 09:51:25 rac1 named[22380]: starting BIND 9.3.4-P1 -u named
Oct 15 09:51:25 rac1 named[22380]: found 1 CPU, using 1 worker thread
Oct 15 09:51:25 rac1 named[22380]: loading configuration from '/etc/named.conf'
Oct 15 09:51:25 rac1 named[22380]: listening on IPv4 interface lo, 127.0.0.1#53
Oct 15 09:51:25 rac1 named[22380]: listening on IPv4 interface bond0, 172.23.15.1#53
Oct 15 09:51:25 rac1 named[22380]: listening on IPv4 interface bond0:3, 172.23.15.10#53
Oct 15 09:51:25 rac1 named[22380]: listening on IPv4 interface bond0:4, 172.23.15.3#53
Oct 15 09:51:25 rac1 named[22380]: listening on IPv4 interface bond0:5, 172.23.15.5#53
Oct 15 09:51:25 rac1 named[22380]: listening on IPv4 interface bond1, 192.168.181.10#53
Oct 15 09:51:25 rac1 named[22380]: command channel listening on 127.0.0.1#953
Oct 15 09:51:25 rac1 named[22380]: command channel listening on ::1#953

There must be no error messages. If you have some errors in the configuration files you need to fix.

Checking configuration

If named started without error messages you can check if host name resolution works. For checking we use “dig” which is part of the bind-utils package. The calling syntax is:

dig @<server> <hostname to query>

In our environment this evaluates to:

[root@rac1 named]# dig @172.23.15.1 rac1.regner.de

; <<>> DiG 9.3.4-P1 <<>> @172.23.15.1 rac1.regner.de
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27024
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;rac1.regner.de.                        IN      A

;; ANSWER SECTION:
rac1.regner.de.         3600    IN      A       172.23.15.1

;; AUTHORITY SECTION:
regner.de.              3600    IN      NS      rac1.regner.de.

;; Query time: 31 msec
;; SERVER: 172.23.15.1#53(172.23.15.1)
;; WHEN: Thu Oct 15 09:48:07 2009
;; MSG SIZE  rcvd: 62

The output consists of two parts:

• question section
• answer section

The interesting part is in the answer section. We queried for “rac1.regner.de” (see question section) and got the following answer section:

;; ANSWER SECTION:
rac1.regner.de.         3600    IN      A       172.23.15.1

The answer section states the host named “regner1.regner.de” has the ip address of 172.23.15.1. Just as we configured it in the zone file. If you changed names and/or IPs your adjusted value must appear in the answer section.

The next query will query for the SCAN name we also configured  in the zone file:

[root@rac1 named]# dig @172.23.15.1 rac-scan.regner.de

; <<>> DiG 9.3.4-P1 <<>> @172.23.15.1 rac-scan.regner.de
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33081
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;rac-scan.regner.de.            IN      A

;; ANSWER SECTION:
rac-scan.regner.de.     3600    IN      A       172.23.15.4
rac-scan.regner.de.     3600    IN      A       172.23.15.5
rac-scan.regner.de.     3600    IN      A       172.23.15.3

;; AUTHORITY SECTION:
regner.de.              3600    IN      NS      rac1.regner.de.

;; ADDITIONAL SECTION:
rac1.regner.de.         3600    IN      A       172.23.15.1

;; Query time: 3 msec
;; SERVER: 172.23.15.1#53(172.23.15.1)
;; WHEN: Thu Oct 15 09:48:21 2009
;; MSG SIZE  rcvd: 119

This time the answer section contained three IP addresses:

;; ANSWER SECTION:
rac-scan.regner.de.     3600    IN      A       172.23.15.4
rac-scan.regner.de.     3600    IN      A       172.23.15.5
rac-scan.regner.de.     3600    IN      A       172.23.15.3

Thats completely fine as oracle needs at least one better three IPs for the scan names. As you can see from the output we configured the recommended amount of three IPs. Oracle will use these IPs for configuring the listeners to listen at this IPs. For more information about SCAN refer here.

Using configuration

In order to use our fresh configured name servers replace the file /etc/resolv.conf with a file like this:

nameserver 172.23.15.1
nameserver 172.23.15.2
search regner.de

The /etc/resolv.conf configures the dns server to use for ip adress resolution. The argument “nameserver” defines the dns server which “search” configures one or more domain suffixes to be appended. In the example listed there are two name servers which will be queried. If one server is unreachable there will be a transparent failover to the second one. The second server will be configured now.

Configuring the second (slave) dns server

Creating the required files

/etc/named.conf

[root@rac2 named]# cat /etc/named.conf
options {

 # The directory statement defines the name server's working directory

 directory "/var/named";

 # Write dump and statistics file to the log subdirectory.  The
 # pathenames are relative to the chroot jail.

 dump-file "/var/log/named_dump.db";
 statistics-file "/var/log/named.stats";

 # The forwarders record contains a list of servers to which queries
 # should be forwarded.  Enable this line and modify the IP address to
 # your provider's name server.  Up to three servers may be listed.

 #forwarders { 192.0.2.1; 192.0.2.2; };

 # Enable the next entry to prefer usage of the name server declared in
 # the forwarders section.

 #forward first;

 # The listen-on record contains a list of local network interfaces to
 # listen on.  Optionally the port can be specified.  Default is to
 # listen on all interfaces found on your system.  The default port is
 # 53.

 #listen-on port 53 { 127.0.0.1; };

 # The listen-on-v6 record enables or disables listening on IPv6
 # interfaces.  Allowed values are 'any' and 'none' or a list of
 # addresses.

 listen-on-v6 { none; };

 # The next three statements may be needed if a firewall stands between
 # the local server and the internet.

 #query-source address * port 53;
 #transfer-source * port 53;
 #notify-source * port 53;

 # The allow-query record contains a list of networks or IP addresses
 # to accept and deny queries from. The default is to allow queries
 # from all hosts.

 #allow-query { 127.0.0.1; };

 # If notify is set to yes (default), notify messages are sent to other
 # name servers when the the zone data is changed.  Instead of setting
 # a global 'notify' statement in the 'options' section, a separate
 # 'notify' can be added to each zone definition.

 notify no;
};

logging {
 # Log queries to a file limited to a size of 100 MB.
 channel query_logging {
 file "/var/named/named_querylog"
 versions 3 size 100M;
 print-time yes;                 // timestamp log entries
 };
 category queries {
 query_logging;
 };

 # Or log this kind alternatively to syslog.
 channel syslog_queries {
 syslog user;
 severity info;
 };
 category queries { syslog_queries; };

 # Log general name server errors to syslog.
 channel syslog_errors {
 syslog user;
 severity error;
 };
 category default { syslog_errors;  };

 # Don't log lame server messages.
 category lame-servers { null; };
};

# The following zone definitions don't need any modification.  The first one
# is the definition of the root name servers.  The second one defines
# localhost while the third defines the reverse lookup for localhost.

#zone "." in {
# type hint;
# file "root.hint";
#};

zone "localhost" in {
 type master;
 file "localhost.zone";
};

zone "0.0.127.in-addr.arpa" in {
 type master;
 file "127.0.0.zone";
};

zone "regner.de" in {
 type slave;
 file "db.regner.de";
 masters { 172.23.15.1; };
};

/var/named/127.0.0.zone

$TTL 1W
@               IN SOA          localhost.   root.localhost. (
 42              ; serial (d. adams)
 2D              ; refresh
 4H              ; retry
 6W              ; expiry
 1W )            ; minimum

 IN NS           localhost.
1               IN PTR          localhost.

/var/named/localhost.zone

$TTL 1W
@               IN SOA  @   root (
 42              ; serial (d. adams)
 2D              ; refresh
 4H              ; retry
 6W              ; expiry
 1W )            ; minimum

 IN NS           @
 IN A            127.0.0.1

/var/named/db.regner.de

You do not need to create the zone file because it will be fetched from the master server.

Starting for the first time

Before staring for the first time we need to change permissions:

[root@rac2 named]# touch /var/named/named_querylog
[root@rac2 named]# chown -R named:named /var/named/
[root@rac2 named]# chown named:named /etc/named.conf

After that we can start for the first time by entering:

/etc/init.d/named start

You message log should look like this:

Oct 15 10:30:02 rac2 named[23116]: starting BIND 9.3.4-P1 -u named
Oct 15 10:30:02 rac2 named[23116]: found 1 CPU, using 1 worker thread
Oct 15 10:30:02 rac2 named[23116]: loading configuration from '/etc/named.conf'
Oct 15 10:30:02 rac2 named[23116]: listening on IPv4 interface lo, 127.0.0.1#53
Oct 15 10:30:02 rac2 named[23116]: listening on IPv4 interface bond0, 172.23.15.2#53
Oct 15 10:30:02 rac2 named[23116]: listening on IPv4 interface bond0:1, 172.23.15.4#53
Oct 15 10:30:02 rac2 named[23116]: listening on IPv4 interface bond0:2, 172.23.15.20#53
Oct 15 10:30:02 rac2 named[23116]: listening on IPv4 interface bond1, 192.168.181.20#53
Oct 15 10:30:02 rac2 named[23116]: command channel listening on 127.0.0.1#953
Oct 15 10:30:02 rac2 named[23116]: command channel listening on ::1#953

There must be no error messages. If you have some errors in the configuration files you need to fix.

Checking configuration

The first this to check is the existence of the file /var/named/db.regner.de:

[root@rac2 named]# ll /var/named/
total 20
-rw-r--r-- 1 named named  192 Oct 15 10:28 127.0.0.zone
drwxrwx--- 2 named named 4096 Jan 21  2009 data
-rw-r--r-- 1 named named  498 Oct 15 10:30 db.regner.de
-rw-r--r-- 1 named named  158 Oct 15 10:28 localhost.zone
-rw-r--r-- 1 named named    0 Oct 15 10:30 named_querylog
drwxrwx--- 2 named named 4096 Jan 21  2009 slaves

The file is there and was transfered from the master server.

We will now query the secondary dns server just like we did with the primary server:

[root@rac2 named]# dig @172.23.15.2 rac1.regner.de

; <<>> DiG 9.3.4-P1 <<>> @172.23.15.2 rac1.regner.de
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9789
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;rac1.regner.de.                        IN      A

;; ANSWER SECTION:
rac1.regner.de.         3600    IN      A       172.23.15.1

;; AUTHORITY SECTION:
regner.de.              3600    IN      NS      rac1.regner.de.

;; Query time: 227 msec
;; SERVER: 172.23.15.2#53(172.23.15.2)
;; WHEN: Thu Oct 15 10:32:23 2009
;; MSG SIZE  rcvd: 62

[root@rac2 named]# dig @172.23.15.2 rac-scan.regner.de

; <<>> DiG 9.3.4-P1 <<>> @172.23.15.2 rac-scan.regner.de
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52238
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;rac-scan.regner.de.            IN      A

;; ANSWER SECTION:
rac-scan.regner.de.     3600    IN      A       172.23.15.5
rac-scan.regner.de.     3600    IN      A       172.23.15.3
rac-scan.regner.de.     3600    IN      A       172.23.15.4

;; AUTHORITY SECTION:
regner.de.              3600    IN      NS      rac1.regner.de.

;; ADDITIONAL SECTION:
rac1.regner.de.         3600    IN      A       172.23.15.1

;; Query time: 1 msec
;; SERVER: 172.23.15.2#53(172.23.15.2)
;; WHEN: Thu Oct 15 10:32:34 2009
;; MSG SIZE  rcvd: 119

Troubleshooting

If you experience problems check out the following files for more information:

• /var/log/messages
• /var/named/named_querylog

The second file records all received dns queries and can be extremely helpful in debugging dns-related problems.

More Information

For information how to change/add/delete dns records refer to the bind documentation.

my blog has moved to a different server.

The new address is:

http://blog.ronnyegner-consulting.de

In this and the following posts i will perform some crash and recover scenarios and show how to recover the cluster successfully.

At the moment the following tests are planned and will be published during the next days:

• suddenly turning off the power and restarting the node
• terminating private network connect between the cluster nodes
• recovering an ACFS file system which will not mount automatically
  (was not able to reproduce more than once; sorry guys)
• overwriting the ASM disk header with the disk group being offline
• corrupting an online and active ASM disk by writing chunks of random data to the disk randomly
• simulating disk errors by removing the device from the operating system
• corrupting  the OCR
• corrupting the Voting Disk

The environment used for the posts are explained in detail here.

Useful scripts can be found here.

my blog has moved to a different server.

The new address is:

http://blog.ronnyegner-consulting.de

Test #4: Corrupting the ASM disk with ASM disk group being online and active

After overwriting the ASM disk header while the disk group was offline we will now put some load on the full running cluster and corrupt the asm disk slightly.

Testcase

Put load on the database

Prior starting this test i created a larger table with one column and approx 1.2 GB size and an empty table with the same structure. So for our simple test we will just copy one table into another table. This forces oracle to read the database blocks and do some writes as well.

create table test2_empty as select * from test where rownum<1;

insert into test2_empty select * from test;

Corrupting the disk

According to fdisk the asm lun has a size of 21474836480 bytes:

Disk /dev/sde: 21.4 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

 Device Boot      Start         End      Blocks     Id  System
/dev/sde1               1       20480    20971504   83  Linux

And the disk is indeed the mirror partner of the disk we destroyed and re-added in the earlier part.

[root@rac1 ~]# oracleasm querydisk /dev/sde1
Device "/dev/sde1" is marked an ASM disk with the label "DISK003B"

We will corrupt the disk by writing 512 byte chunks of random data from /dev/urandom to 1000 different locations all over the lun. Given the lun size this makes 41943040 blocks of 512 bytes each.

Our command to corrupt the lun will be:

dd if=/dev/random bs=512 count=1 of=/dev/sde1 seek=nnnn

Where “seek=nnn” is a number from 0 to 41943040-1.

Note that this time we manipulate the second mirror disk of disk group DATA2 – DISK003B. In test no 3 we overwrote the asm disk header of DISK003A and re-added the disk to the data group. In this example we manipulate DISK003B. If there were undetected errors rebalancing the disks we will discover it now.

For this we first of all needed a small little random number generator within the limits of 0 to 41943040; here it is:

#!/bin/bash

function random()
{
 od -d /dev/urandom | sed -e 's/^[0-9]* //' -e 's/ //g' |\
 while read L ; do echo -n $L ; done |\
 dd bs=1 count=${1:-10} 2>/dev/null
}

A=$(random 10) #return random number with 10 numbers
B=$(expr $C % 41943040)  # keep number within range
echo $B

With this random number generator we created a small shell script for corrupting the asm lun which looks like this:

dd if=/dev/urandom of=/dev/sde1 bs=512 count=1 seek=986024
dd if=/dev/urandom of=/dev/sde1 bs=512 count=1 seek=1594423
dd if=/dev/urandom of=/dev/sde1 bs=512 count=1 seek=2236024
[..]

Expected test results

So what are we expecting?

According to oracle documents (note 416046.1) there are two different scenarios:

• If the primary extent is detected to be corrupt (asm only reads from primary extent unless a read preference set) ASM retries the read. If re-read data is also corrupt ASM fails over to the secondary extent. If data in secondary extent is valid data in primary extent will be overwritten with the good data from the secondary extent.
• If data in secondary extent is corrupt it will remain undetected. If new data is written the new data will be written both to the primary and secondary disk. This will most likely overwrite the corrupted data. But if the primary extent fails oracle will read the secondary extent and will discover the corruption as well -  but this time the corruption is not fixable. You will end up with restoring/recovering the corrupt database blocks and probably much more.

In addition to that i expect ASM and/or the database to automatically fix detected corruptions.

Testing

Message no. 1: Database detected and fixed corruption

During our Insert statement running the database alert.log showed:

Fri Oct 02 10:47:21 2009
Hex dump of (file 6, block 156029) in trace file /u01/app/oracle/diag/rdbms
  /ora11p/ora11p1/trace/ora11p1_ora_30023.trc
Corrupt block relative dba: 0x0182617d (file 6, block 156029)
Bad header found during multiblock buffer read
Data in bad block:
 type: 180 format: 3 rdba: 0xb19a95b2
 last change scn: 0x6f92.6c357e5d seq: 0x1 flg: 0x5a
 spare1: 0xfd spare2: 0x80 spare3: 0x4f3c
 consistency value in tail: 0x1591148c
 check value in block header: 0x2abf
 block checksum disabled
Reading datafile '+DATA2/ora11p/users02.dbf' for corruption at rdba:
  0x0182617d (file 6, block 156029)
Read datafile mirror 'DISK003B' (file 6, block 156029) found same corrupt data
Read datafile mirror 'DISK003A' (file 6, block 156029) found valid data
Repaired corruption at (file 6, block 156029)

Message no. 2: Database detected corruption

Corrupt block relative dba: 0x01c12d35 (file 7, block 77109)
Bad check value found during validation
Data in bad block:
 type: 2 format: 2 rdba: 0x01c12d35
 last change scn: 0x0000.00253bd5 seq: 0x73 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x3bd50273
 check value in block header: 0x94cb
 computed block checksum: 0xf0eb
Trying mirror side DISK003B.
Reread of blocknum=77109, file=+DATA2/ora11p/datafile/undotbs01_02.dbf. found
 same corrupt data
Reread of blocknum=77109, file=+DATA2/ora11p/datafile/undotbs01_02.dbf. found
 valid data

Note: This time the database does not say anything about the corruption being fixed.

Message no. 3: ASM detected and fixed corruption

Mon Oct 05 09:32:15 2009
WARNNING: cache read a corrupted block group=DATA2 fn=1
 blk=893 from disk 0
NOTE: a corrupted block from group DATA2 was dumped to /u01/app/oracle/diag/asm
 /+asm/+ASM1/trace/+ASM1_ora_4739.trc
WARNNING: cache read(retry) a corrupted block group=DATA2 fn=1
 blk=893 from disk 0
NOTE: cache repaired a corrupted block group=DATA2 fn=1 blk=893
 from disk 0

Conclusion

First of all: regardless of the caveats described below the insert statement most of the time completed successfully:

SQL> insert into test2 select * from sys.test;
134217728 rows created.

Second: During our tests ASM performed well and stable. We were consistently able to work on the disk groups even if the disk group missed their mirrors.

However i discovered some problems as well:

• ASM has no ability to check data on disk groups completely (i.e. compare data on all mirror partners and check for corruptions); so corruptions on secondary extents will remain undetected until primary extent fails and secondary extent is read
• ASM nor the database does always fix corruptions found in primary extents (see message 2 above… corruption was detected but NOT corrected and remained on disk) as indicated by note 416046
• Checking database with RMAN and “backup validate database” finds and reports errors but does not fix ANY error found. The error message no. 2 originated from a “select * from test….” but rman produces this type of error messages in alert.log as well. From the Metalink document which describes handling of block corruptions in ASM i would have expected ASM or the database to fix these errors automatically.
• There are scenarios where corrupted primary blocks prevent the database from starting at all. During our tests we have seen this two times with different error messages. One sample error message stack is reproduced in error message no. 3 coming from the asm instance and in the following lines below coming from the database instance:

Exception [type: SIGILL, Illegal operand] [ADDR:0x105DC01]
[PC:0x105DC01, kkestRCSBase()+221] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/ora11p/ora11p1/
trace/ora11p1_ora_5206.trc  (incident=48254):
ORA-07445: exception encountered: core dump [kkestRCSBase()+221]
[SIGILL] [ADDR:0x105DC01] [PC:0x105DC01] [Illegal operand] []
Incident details in: /u01/app/oracle/diag/rdbms/ora11p/ora11p1/
incident/incdir_48254/ora11p1_ora_5206_i48254.trc
Mon Oct 05 09:33:47 2009
Trace dumping is performing id=[cdmp_20091005093347]
Mon Oct 05 09:33:49 2009
Exception [type: SIGILL, Illegal operand] [ADDR:0x105DC01]
[PC:0x105DC01, kkestRCSBase()+221] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/ora11p/ora11p1/
trace/ora11p1_ora_5296.trc  (incident=48310):
ORA-07445: exception encountered: core dump [kkestRCSBase()+221]
[SIGILL] [ADDR:0x105DC01] [PC:0x105DC01] [Illegal operand] []
Incident details in: /u01/app/oracle/diag/rdbms/ora11p/ora11p1/
incident/incdir_48310/ora11p1_ora_5296_i48310.trc

Although we met this errors we were able to fix them: In one case dropping the disk containing the corrupted blocks was enough to start the instance again. The second incident reproduced above required the corrupted disk to be dropped, re-added and ASM instance to be restarted before we were able to successfully start the database instance.

• Dropping a corrupted disk and re-adding the disk requires the disk group the disk belonged to be taken offline. Otherwise labeling the former member disk failed with “device or resource busy”. I will investigate this further.

Summarizing my experiences until now i would go with ASM and normal or even high redundancy. This enables far more options for fixing errors without having to rebuild the disk group and restoring everything from tape/disk.