My current job doesn’t offer me at the moment the opportunity to play with RAC or other stuff, so I decided to build my own RAC on my Windows7-desktop (64-bits, 8GB RAM), using 3 VM’s (VM-workstation , 7.1.3 build-324285) :

  • 2 VM’s for two RAC-nodes, based on OEL 5.5, 11.2.0.2 for infra and database.
  • 1 VM as my own storage, based on Openfiler 2.3 (free) and ASM as storage manager.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Powered by rPath Linux, Openfiler is a free browser-based network storage management utility that delivers file-based Network Attached Storage (NAS) and block-based Storage Area Networking (SAN) in a single framework. Openfiler supports CIFS, NFS, HTTP/DAV, FTP, however, we will only be making use of its iSCSI capabilities to implement an inexpensive SAN for the shared storage components required by Oracle RAC 11g. A 500GB internal hard drive will be connected to the network storage server (sometimes referred to in this article as the Openfiler server) through an internal embedded SATA II controller. The Openfiler server will be configured to use this disk for iSCSI based storage and will be used in our Oracle RAC 11g configuration to store the shared files required by Oracle Clusterware as well as all Oracle ASM volumes.

This piece of text is taken from another website: http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_10.shtml

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

My goal: just a little bit hands-on experience with Openfiler, and do some testing with RAC 11gr2, and especially 11.2.0.2. Maybe it’s helpful for somebody else, so I’ll post my experiences. Steps I’m gonna perform.

–          The first post (this post) :

1.       Planning my installation
2.       Create a VM with OpenFiler
3.       Configure Openfiler

–          The second post:

4.       Create a VM as node rac1 and configure ASM
5.       Create a VM as node rac2
6.       Install Oracle RAC Infrastructure
 
7.       Install Oracle RAC Database software

8.       Install Oracle RAC Database

1.       Planning my installation

A very important part of a RAC-installation. Consistency and a structured approach will help a lot during this kind of installations.

For my own project I used  Internet resources of course, and a few of them are listed  below.

A useful blog to setup storage with Openfiler
Build Your Own Oracle RAC 11g Cluster on Oracle Enterprise Linux and iSCI , by Jeffrey Hunter
http://www.appsdba.info/docs/oracle_apps/RAC/Install_Open_Filer.pdf

First of all you need software:

VMware Workstation (not free) or Server (free). I used Workstation here, which has some advantages like cloning and using snapshots. For running the software you can use VMware player.
Openfiler 2.3 – 64-bits iso file (not the appliance!)
Oracle Enterprise Linux, 5.5
Oracle software 11.2.0.2 (Linux) from My Oracle Support (is not a patch, full install!). 11.2.0.1 is still available on otn.

Creating 3 VM’s:

Node (VM) Instance DB-name Memory O.S.
Rac1 Racdb1 Racdb.jobacle.nl 2,5 GB OEL 5.4 – (x86_64)
Rac2 Racdb2 Racdb.jobacle.nl 2,5 GB
Openfiler1 768 MB Openfiler 2.3 64-bit

Network :

Node (VM) Public IP (bridged, eth0) Private IP (host-only, eth1) Virtual IP
Rac1 192.168.178.151 192.168.188.151 192.168.178.251
Rac2 192.168.178.152 192.168.188.152 192.168.178.252
Openfiler1 192.168.178.195 192.168.188.195

Scan:

SCAN name SCAN IP
Rac-scan 192.168.178.187

Oracle Software on local storage (rac1 + rac2):

Soft-ware O.S.-user Primary Group Suppl.group Home dir Oracle base

/ Oracle home

Grid infra Grid Oinstall asmadmin,asmdba,

asmoper

/home/grid /u01/app/grid

/u01/app/11.2.0/grid

Oracle RAC Oracle Oinstall Dba,oper,asmdba /home/oracle /u01/app/oracle

/u01/app/oracle/product/11.2.0/dbhome_1

Shared storage (total of 80GB):

Storage Filesystem Volume size ASM name ASM redundancy Openfiler Volume name
OCR/Voting disk ASM 2GB +CRS External Racdb_crs1
DB-files ASM 39GB +RACDB_DATA External Racdb_data1
Fast Rec.area ASM 39GB +FRA External Racdb_fra1

2.       Create a VM with OpenFiler.

Decided not to use the applicance, but install Openfiler with the iso-file, and later on add a disk of 80 GB (/dev/sdb) for configuring my ASM-based shared storage. On that disk I’ll create a volume group of 80 GB. You can also choose to create more separate disks and then create 1 volume group to span those disks.

Detailed steps:

2.1   Create a VM

2.2   Install Openfiler

2.3   Add disk of 80GB

Download Openfiler iso-file (64-bits) from here.

2.1 Create a VM.

In VM workstation, click on menu-item VM, ‘new’:

welcome-page VMware workstation

In compatibility screen, click next.

Browse and click your Openfiler – iso-file, ‘next’.

Guest Operating system : Linux. Version , I chose OEL.

VM-name: Openfiler1.

Processor-choice

Processor configuration: Default

Memory for Openfiler

Memory: 768 MB (tried 512 MB, going to swap)

Next screens: network bridged, and I/O: LSI logic.

First I’m using  a disk of 8GB for the installation of Openfiler. Later on I’ll add a disk of 80GB for Oracle datafiles.

Creating 8GB disk

Create new virtual disk , in the next screen: ‘Disk type: SCSI’

Disk: 8GB, split in multiple files. Saves space. Less performance.

Disk file

Disk file: default.

Ready? You get the start screen of VM-workstation.

Now we also add an network-adapter, host-only (for private networking).

Click ‘edit virtual machine settings’

Select network adapter, click add.

Click next.

Add private network adapter

Network adapter type: host-only (=vmnet1)

Click on ok twice.

Check if the cd points to the iso-file of Openfiler, then power on this virtual machine. This will lead to:

2.2 Install Openfiler.

Press enter

Skip the test

Skip the test..

Openfiler install screen

Screen: welcome to OpenFiler NSA.  Click next.

Screen: keyboard configuration: U.S. International. Click next

Screen: disk partitioning setup: Automatic Partitioning. Click next.

Warning screen: would you like to initialize … Click o.k.

Automatic partitioning: remove all partitions, review partitions created.

Warning screen: erase all data? Click o.k.

Screen: disk setup. Click next.

Edit eth0: -> Activate on boot, ip: 192.168.178.195, mask: 255.255.255.0

Edit eth1: -> Activate on boot, ip: 192.168.188.195, mask 255.255.255.0

Set the hostname, insert the gateway.

Screen: timezone: appropriate timezone, no UTC.

Screen: set root password.

Screen: about to install. Click next.

Installing…..

Screen: congratulations (I hope..). Click ‘reboot’.

Logging in with ‘root’ and the password you provided.

You get the prompt.

Then log in through the browser: https://192.168.178.195:446 . You may see same warnings about certificates. Accept them.

Login with ‘openfiler’ and ‘password’ to see if you did it allright.

2.3 Adding disk of 80 GB.

Shutdown (prompt: ‘shutdown –h now’ or through the gui) to add the disk for the RAC-shared storage.

After shutting down:

Edit virtual machine settings

Choose add, Create a new Virtual Disk.

Add disk

SCSI, Independent, persistent.

Add disk 80 GB

80 GB, split into multiple files.

Click finish.

Now we’ve got a standard Openfiler, just like the appliance they are offering, but now with an extra disk of 80GB.

3.      Configure Openfiler

After installing Openfiler and adding a disk we have to configure:

3.1  Configure iSCSI storage

3.2  Configure network access

3.3  Partition the physical storage

3.4  Create a new volume group

3.5  Create all logical volumes

3.6  Create new iSCSI targets for each of the logical volumes.

3.1 Configuring iSCSI storage .

Power on VM Openfiler1.

Log in through a browser: https://192.168.178.195:446 (username: ‘openfiler’ /password: ‘password’)

Click tab ‘Services’

enable iscsi

Click link ‘Enable’ at row ‘iSCSI target server’

3.2 Configure network access :

Click tab ‘System’, scroll down:

Network access

Name Network/Host Netmask Type
rac1-public 192.168.178.151 255.255.255.0 Shared
rac1-priv 192.168.188.151 255.255.255.0 Shared
rac2-public 192.168.178.152 255.255.255.0 Shared
rac2-priv 192.168.188.152 255.255.255.0 Shared

3.3 Partition the physical storage

Click tab ‘Volumes’
Click ‘Create new physical volumes’ in link.

Create physical volume

Click link /dev/sdb

Scroll down and click button ‘Create’.

Create partition

Now we’ve created a Linux Physical Volume (Partition /dev/sdb1):

Now we will create a Volume Group. This volume group will be named ‘racdbvg’ and will contan the newly created primary partition of 80GB.

3.4 Create a Volume Group

Click on Volumes – tab, you will get:

Fill in ‘racdbvg’ in volume group name, click on the checkbox of the partition of 80GB and  click on ‘Add Volume Group’.

Next step is to create ‘logical volumes’ in this Volume Group (racdbvg):

3.5 Create logical volumes

Remember the planning of the disks, see the last column :

Storage Filesystem Volume size ASM name ASM redundancy Openfiler Volume name
OCR/Voting disk ASM 2GB (2208M) +CRS External Racdb_crs1
DB-files ASM 39GB (39840M) +RACDB_DATA External Racdb_data1
Fast Rec.area ASM 39GB (39840M) +FRA External Racdb_fra1

Navigate to Volume-tab:

Click ‘Add Volume’

Fill in the name of the volume, size and filesystem (iSCSI) for all three volumes.

Result:

These volumes are the iSCSI disks that can be presented to our iSCSI clients (racnode1 and racnode2) on the network.

Unfortunately we are not finished yet. The iSCSI clients are still not able to access / ‘see’ the drives. We have to create iSCSI targets for each of the three volumes, and map the iSCSI target to the volumes (so-called ‘LUN-mapping’).

Steps for each of the volumes (three times!)

–      Create a unique Target IQN = the name for the new iSCSI target.

–      Map one of the created volumes to that Target IQN

–      Grant both of the Oracle RC nodes access to the new iSCSI target.

3.6 Create a unique Target IQN

Target IQN iSCSI Volume Name Volume description
Iqn.2006-01.com.openfiler:racdb.crs1 Racdb-crs1 ASM CRS Vol1
Iqn.2006-01.com.openfiler:racdb.data1 Racdb-data1 ASM Data Vol1
Iqn.2006-01.com.openfiler:racdb.fra1 Racdb-fra1 ASM FRA Vol1

Navigate in the Volume-tab to iSCSI-targets:

A name is generated , like this: iqn.2006-01.com.openfiler:tsn.4c0aa6d6d32a .

For better understanding, change this to iqn.2006-01.com.openfiler:racdb.crs1 like this:

Click ‘add’. You get a page to modify some settings for this target. Didn’t change anything.

Then click on the LUN-mapping (grey sub-tab). Be sure it’s the right iSCSI-target you’re in.

Click on ‘map’ for the specific target.

Now we need to grant the two-RAC nodes acces to the new iSCSI targets through the storage (private) network.

Click on ‘network ACL’ in the grey sub-tab.

Allow acces for the rac1-priv and rac2-priv. Click ‘update’.

Perform this three times.

Now we are done for the moment. The storage  is ready for the rac-nodes. More in part II.