0. 规划：
OS: RedHat Linux 6.6 64bit
Grid: 12.1.0.1
Oracle: 12.1.0.1
节点定义：
| 名称 | hostname | 公有IP | VIP | 私网IP | scan-ip |
|---|---|---|---|---|---|
| 节点1 | ora-node1 | 10.1.7.81 | 10.1.7.86 | 172.116.0.81 | 10.1.7.239 |
| 节点2 | ora-node2 | 10.1.7.82 | 10.1.7.87 | 172.116.0.82 |
磁盘：
| 名称 | 配置方式 | 容量 | 数量 | 用途 |
|---|---|---|---|---|
| /dev/vdb | LVM | 50G | 1 | grid、oracle软件、BASE目录、HOME目录 |
| DATA | udev | 50G | 3 | 数据磁盘组 |
| FRA | udev | 15G | 2 | 闪回区 |
| OCR_VOTING | udev | 8G | 3 | OCR，voting disk |
1. OS准备过程
1.1 安装系统包
yum install gcc gcc-c++ binutils compat-libcap1 compat-libstdc++-33 \
glibc glibc-devel ksh libaio libaio-devel libgcc libstdc++ libstdc++-devel \
libxcb libX11 libXau libXi libXtst make net-tools nfs-utils smartmontools \
sysstat xterm unixODBC unixODBC-devel -y
1.2 禁用SELINUX
所有节点
sed 's,'SELINUX=enforcing','SELINUX=disabled',g' -i /etc/sysconfig/selinux
sed 's,'SELINUX=enforcing','SELINUX=disabled',g' -i /etc/selinux/config
setenforce 0
测试环境可以停用防火墙，但生产环境不要停用防火墙。
service iptables stop
chkconfig iptables off
1.3 配置时间同步
所有节点
使用CTSS，所以要停用 NTP 服务。
service ntpd stop
chkconfig ntpd off
mv /etc/ntp.conf /etc/ntp.conf.old
rm -rf /var/run/ntpd.pid
1.4 设置系统用户资源限制
所有节点
1.4.1 修改/etc/security/limits.conf
cat >> /etc/security/limits.conf <<EOF
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft memlock 3145728
oracle hard memlock 3145728
EOF
```
######1.4.2 修改/etc/pam.d/login
```
cat >> /etc/pam.d/login <<EOF
session required pam_limits.so
EOF
```
######1.4.3 shell限制
对默认的 shell 启动文件进行以下更改，以便更改所有 Oracle 安装所有者的 ulimit 设置：
```
cat >> /etc/profile <<EOF
if [ \$USER = "oracle" ] || [ \$USER = "grid" ]; then
if [ \$SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
EOF
```
######1.4.4 修改/etc/sysctl.conf
```
cat >> /etc/sysctl.conf <<EOF
kernel.shmmax = 4294967295
kernel.shmall = 2097152
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
kernel.panic_on_oops=1
EOF
sysctl -p
```
######1.4.5 添加"NOZEROCONF=yes"
```
echo "NOZEROCONF=yes" >> /etc/sysconfig/network
```
注：如果不加入该配置，在安装grid之前的检查会出现如下信息：
可以参考http://blog.csdn.net/msdnchina/article/details/45462319。
####1.5 编辑hosts
```
vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.1.7.81 ora-node1
172.116.0.81 ora-node1-priv
10.1.7.86 ora-node1-vip
10.1.7.82 ora-node2
172.116.0.82 ora-node2-priv
10.1.7.87 ora-node2-vip
```
注意：不要在此文件中配置scan地址和名字。
####1.6 创建用户和组
两个节点都执行
```
groupadd -g 1000 oinstall
groupadd -g 1200 asmadmin
groupadd -g 1201 asmdba
groupadd -g 1202 asmoper
groupadd -g 1300 dba
groupadd -g 1301 oper
useradd -m -u 1100 -g oinstall -Gasmadmin,asmdba,asmoper,dba -d \
/home/grid -s /bin/bash grid
useradd -m -u 1101 -g oinstall -Gdba,oper,asmdba -d \
/home/oracle -s /bin/bash oracle
gpasswd -a grid dba #将grid用户加入到dba组
echo "oracle" | passwd --stdin grid
echo "oracle" | passwd --stdin oracle
```
节点用户(grid,oracle)间互信。以下命令分别在各个节点上的oracle用户和grid用户下执行。
```
ssh-keygen -t rsa
ssh-keygen -t dsa
```
下面的命令可以在其中一个节点执行，然后将生成的authorized_keys文件复制到其它节点。
```
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
cat .ssh/id_dsa.pub >> .ssh/authorized_keys
ssh ora-node2 cat ~/.ssh/id_rsa.pub >> .ssh/authorized_keys
ssh ora-node2 cat ~/.ssh/id_dsa.pub >> .ssh/authorized_keys
scp .ssh/authorized_keys ora-node2:~/.ssh/.
```
验证。在各个节点执行如下格式命令。
```
ssh {hostname} date
```
####1.7 创建目录
所有节点
```
mkdir -p /u01/app/soft #安装的clusterware和DB软件包都放在该目录下.
mkdir -p /u01/app/12.1.0/grid #grid的ORACLE_HOME
chown -R grid:oinstall /u01/app/
mkdir -p /u01/app/oracle #oracle的ORACLE_BASE
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/
```
####1.8 编辑grid，oracle用户的环境变量
grid用户的。
```
# su - grid
$ vim .bash_profile
export ORACLE_SID=+ASM1
#export ORACLE_SID=+ASM2
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/12.1.0/grid_1
export TEMP=/tmp
export TMP=/tmp
export TMPDIR=/tmp
export PATH=$ORACLE_HOME/bin:$PATH:$HOME/bin
umask 022
$ . .bash_profile
```
oracle用户的。
```
# su - oracle
$ vim .bash_profile
export ORACLE_SID=ORCL1
#export ORACLE_SID=ORCL2
export ORACLE_UNQNAME=ORCL
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/12.1.0/db_1
LD_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/oracm/lib:$LD_LIBRARY_PATH
export CLASSPATH=$ORACLE_HOME/jre:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp
export PATH=$ORACLE_HOME/bin:$PATH:$HOME/bin
umask 022
#以下可以不加
#export JAVA_HOME=/u01/app/12.1.0/grid_1/jdk
#export ORACLE_TERM=xterm
#export NLS_DATE_FORMAT="YYYY:MM:DDHH24:MI:SS"
#export NLS_LANG=american_america.AL32UTF8
#export TNS_ADMIN=$ORACLE_HOME/network/admin
#export ORA_NLS11=$ORACLE_HOME/nls/data
$ . .bash_profile
```
####1.9 使用udev配置asm使用的磁盘
**仅仅在一个节点处理即可**
######1.9.1 使用fdisk处理磁盘
```
# fdisk -l | grep sd
Disk /dev/sda: 53.7 GB, 53687091200 bytes
Disk /dev/sdb: 8589 MB, 8589934592 bytes
Disk /dev/sdc: 8589 MB, 8589934592 bytes
Disk /dev/sdd: 8589 MB, 8589934592 bytes
Disk /dev/sde: 16.1 GB, 16106127360 bytes
Disk /dev/sdf: 16.1 GB, 16106127360 bytes
Disk /dev/sdg: 53.7 GB, 53687091200 bytes
```
将上面磁盘进行分区，如下：
```
# fdisk /dev/sda
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xb85ca157.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-8192, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-8192, default 8192):
Using default value 8192
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
```
######1.9.2 编写shell脚本，生成udev配置规则文件
```
# vim genasmdisk.sh
#!/bin/bash
cat /dev/null > /etc/udev/rules.d/99-oracle-asmdisk.rules
for i in a b c d e f g;
do
echo "KERNEL==\"sd?1\", BUS==\"scsi\",PROGRAM==\"/sbin/scsi_id -g -u -d /dev/\$parent\", RESULT==\"`/sbin/scsi_id -g -u -d /dev/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\",GROUP=\"asmadmin\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdisk.rules
done
```
执行脚本。
```
# chmod +x genasmdisk.sh
# ./genasmdisk.sh
```
######1.9.3 检查udev规则文件内容
```
# cat /etc/udev/rules.d/99-oracle-asmdisk.rules
KERNEL=="sd?1", BUS=="scsi",PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="0QEMU_QEMU_HARDDISK_244c9f1f-f15d-49b3-a", NAME="asm-diska", OWNER="grid",GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", BUS=="scsi",PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="0QEMU_QEMU_HARDDISK_080377de-ef70-4de5-8", NAME="asm-diskb", OWNER="grid",GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", BUS=="scsi",PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="0QEMU_QEMU_HARDDISK_7b9b90c2-7330-4fe9-9", NAME="asm-diskc", OWNER="grid",GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", BUS=="scsi",PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="0QEMU_QEMU_HARDDISK_45f70178-1dfe-454a-9", NAME="asm-diskd", OWNER="grid",GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", BUS=="scsi",PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="0QEMU_QEMU_HARDDISK_ca7283c4-c521-459f-a", NAME="asm-diske", OWNER="grid",GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", BUS=="scsi",PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="0QEMU_QEMU_HARDDISK_6fdb0e82-8c1d-4694-a", NAME="asm-diskf", OWNER="grid",GROUP="asmadmin", MODE="0660"
KERNEL=="sd?1", BUS=="scsi",PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="0QEMU_QEMU_HARDDISK_65a9f8aa-e13b-4719-a", NAME="asm-diskg", OWNER="grid",GROUP="asmadmin", MODE="0660"
```
注意：将该rules文件复制到其它节点。
######1.9.4 重启udev，使rules生效（所有节点都执行）
```
# partprobe /dev/sda
# partprobe /dev/sdb
......
```
或
```
# start_udev
```
**建议：这时所有节点重启机器。**
######1.9.5 检查asm disk
```
# ls -al /dev/asm*
brw-rw----. 1 grid asmadmin 8, 1 May 17 15:31 /dev/asm-diska
brw-rw----. 1 grid asmadmin 8, 17 May 17 15:31 /dev/asm-diskb
brw-rw----. 1 grid asmadmin 8, 33 May 17 15:31 /dev/asm-diskc
brw-rw----. 1 grid asmadmin 8, 49 May 17 15:31 /dev/asm-diskd
brw-rw----. 1 grid asmadmin 8, 65 May 17 15:31 /dev/asm-diske
brw-rw----. 1 grid asmadmin 8, 81 May 17 15:31 /dev/asm-diskf
brw-rw----. 1 grid asmadmin 8, 97 May 17 15:31 /dev/asm-diskg
```
到此系统层面的准备工作基本完成（注意swap的要求，这里忽略）。开始安装Oracle集群套件。
####2. 安装Grid Infrastructure
######2.1 下载软件到/u01/app/soft/目录下
从[下载地址](http://www.oracle.com/technetwork/database/enterprise-edition/downloads/database12c-linux-download-2240591.html)下载软件包。解压软件包，然后运行`runInstall`。
- linuxamd64_12c_grid_1of2.zip
- linuxamd64_12c_grid_2of2.zip
######2.2 开始OUI界面安装
######解决问题：
- **cvuqdisk**包处理
`rpm -ivh /u01/app/soft/grid/stage/cvu/cv/remenv/cvuqdisk-1.0.9-1.rpm`，所有节点都执行。
- **panic_on_oops**系统内核参数处理，所有节点都执行。
```
echo 'kernel.panic_on_oops = 1' >> /etc/sysctl.conf
sysctl -p
```
- **zeroconf check**处理，所有节点都执行。
```
# cat >> /etc/sysconfig/network << EOF
NOZEROCONF=yes
EOF
```
- **Device Checks for ASM**
网上说可能是udev生成磁盘属组和第13步中的系统组不一致造成的，但经过查对没有不一致，该问题忽略，后面的安装没有出现相应的问题？！
- **其它问题，忽略即可。**
```
# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
```
```
# /u01/app/12.1.0/grid_1/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.1.0/grid_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0/grid_1/crs/install/crsconfig_params
2017/05/18 13:39:15 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/05/18 13:39:44 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/05/18 13:39:45 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
2017/05/18 13:40:33 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'ora-node1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'ora-node1'
CRS-2676: Start of 'ora.mdnsd' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.evmd' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'ora-node1'
CRS-2676: Start of 'ora.gpnpd' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ora-node1'
CRS-2672: Attempting to start 'ora.gipcd' on 'ora-node1'
CRS-2676: Start of 'ora.cssdmonitor' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'ora-node1'
CRS-2672: Attempting to start 'ora.diskmon' on 'ora-node1'
CRS-2676: Start of 'ora.diskmon' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.cssd' on 'ora-node1' succeeded
ASM created and started successfully.
Disk Group OCR_VOTING created successfully.
CRS-2672: Attempting to start 'ora.crf' on 'ora-node1'
CRS-2672: Attempting to start 'ora.storage' on 'ora-node1'
CRS-2676: Start of 'ora.storage' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.crf' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'ora-node1'
CRS-2676: Start of 'ora.crsd' on 'ora-node1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk d280951c2a7d4f8cbff978c13aa956b4.
Successful addition of voting disk 1f43b0992d254fb4bf564cd383254107.
Successful addition of voting disk ff17d288f6ae4f25bf134fcc3a7542ab.
Successfully replaced voting disk group with +OCR_VOTING.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE d280951c2a7d4f8cbff978c13aa956b4 (/dev/asm-diskb) [OCR_VOTING]
2. ONLINE 1f43b0992d254fb4bf564cd383254107 (/dev/asm-diskc) [OCR_VOTING]
3. ONLINE ff17d288f6ae4f25bf134fcc3a7542ab (/dev/asm-diskd) [OCR_VOTING]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ora-node1'
CRS-2673: Attempting to stop 'ora.crsd' on 'ora-node1'
CRS-2677: Stop of 'ora.crsd' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'ora-node1'
CRS-2677: Stop of 'ora.storage' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.evmd' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.asm' on 'ora-node1'
CRS-2677: Stop of 'ora.mdnsd' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.crf' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.asm' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'ora-node1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'ora-node1'
CRS-2677: Stop of 'ora.cssd' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'ora-node1'
CRS-2677: Stop of 'ora.gipcd' on 'ora-node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ora-node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'ora-node1'
CRS-2672: Attempting to start 'ora.evmd' on 'ora-node1'
CRS-2676: Start of 'ora.mdnsd' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.evmd' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'ora-node1'
CRS-2676: Start of 'ora.gpnpd' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'ora-node1'
CRS-2676: Start of 'ora.gipcd' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ora-node1'
CRS-2676: Start of 'ora.cssdmonitor' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'ora-node1'
CRS-2672: Attempting to start 'ora.diskmon' on 'ora-node1'
CRS-2676: Start of 'ora.diskmon' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.cssd' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'ora-node1'
CRS-2672: Attempting to start 'ora.ctssd' on 'ora-node1'
CRS-2676: Start of 'ora.ctssd' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'ora-node1'
CRS-2676: Start of 'ora.asm' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'ora-node1'
CRS-2676: Start of 'ora.storage' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'ora-node1'
CRS-2676: Start of 'ora.crf' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'ora-node1'
CRS-2676: Start of 'ora.crsd' on 'ora-node1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: ora-node1
CRS-6016: Resource auto-start has completed for server ora-node1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/05/18 13:46:58 CLSRSC-343: Successfully started Oracle Clusterware stack
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'ora-node1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'ora-node1'
CRS-2676: Start of 'ora.asm' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.OCR_VOTING.dg' on 'ora-node1'
CRS-2676: Start of 'ora.OCR_VOTING.dg' on 'ora-node1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ora-node1'
CRS-2673: Attempting to stop 'ora.crsd' on 'ora-node1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'ora-node1'
CRS-2673: Attempting to stop 'ora.OCR_VOTING.dg' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.cvu' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.ora-node1.vip' on 'ora-node1'
CRS-2677: Stop of 'ora.cvu' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.OCR_VOTING.dg' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'ora-node1'
CRS-2677: Stop of 'ora.asm' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'ora-node1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'ora-node1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.ora-node1.vip' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'ora-node1'
CRS-2677: Stop of 'ora.ons' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'ora-node1'
CRS-2677: Stop of 'ora.net1.network' on 'ora-node1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'ora-node1' has completed
CRS-2677: Stop of 'ora.crsd' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.crf' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.evmd' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.storage' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'ora-node1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'ora-node1'
CRS-2677: Stop of 'ora.storage' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'ora-node1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.crf' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'ora-node1' succeeded
CRS-2677: Stop of 'ora.asm' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'ora-node1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'ora-node1'
CRS-2677: Stop of 'ora.cssd' on 'ora-node1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'ora-node1'
CRS-2677: Stop of 'ora.gipcd' on 'ora-node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ora-node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'ora-node1'
CRS-2672: Attempting to start 'ora.evmd' on 'ora-node1'
CRS-2676: Start of 'ora.mdnsd' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.evmd' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'ora-node1'
CRS-2676: Start of 'ora.gpnpd' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'ora-node1'
CRS-2676: Start of 'ora.gipcd' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'ora-node1'
CRS-2676: Start of 'ora.cssdmonitor' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'ora-node1'
CRS-2672: Attempting to start 'ora.diskmon' on 'ora-node1'
CRS-2676: Start of 'ora.diskmon' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.cssd' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'ora-node1'
CRS-2672: Attempting to start 'ora.ctssd' on 'ora-node1'
CRS-2676: Start of 'ora.ctssd' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'ora-node1'
CRS-2676: Start of 'ora.asm' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'ora-node1'
CRS-2676: Start of 'ora.storage' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'ora-node1'
CRS-2676: Start of 'ora.crf' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'ora-node1'
CRS-2676: Start of 'ora.crsd' on 'ora-node1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.OCR_VOTING.dg' is already running on 'ora-node1'
CRS-6017: Processing resource auto-start for servers: ora-node1
CRS-2672: Attempting to start 'ora.net1.network' on 'ora-node1'
CRS-2672: Attempting to start 'ora.oc4j' on 'ora-node1'
CRS-2676: Start of 'ora.net1.network' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'ora-node1'
CRS-2672: Attempting to start 'ora.ons' on 'ora-node1'
CRS-2672: Attempting to start 'ora.ora-node1.vip' on 'ora-node1'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'ora-node1'
CRS-2676: Start of 'ora.cvu' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.ora-node1.vip' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'ora-node1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'ora-node1'
CRS-2676: Start of 'ora.ons' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'ora-node1' succeeded
CRS-2676: Start of 'ora.oc4j' on 'ora-node1' succeeded
CRS-6016: Resource auto-start has completed for server ora-node1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/05/18 13:50:33 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... Succeeded
```
######2.3 检查安装情况
```
# su - grid
$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM.lsnr ora....er.type ONLINE ONLINE ora-node1
ora....ER.lsnr ora....er.type ONLINE ONLINE ora-node1
ora....N1.lsnr ora....er.type ONLINE ONLINE ora-node1
ora.MGMTLSNR ora....nr.type ONLINE ONLINE ora-node1
ora....TING.dg ora....up.type ONLINE ONLINE ora-node1
ora.asm ora.asm.type ONLINE ONLINE ora-node1
ora.cvu ora.cvu.type ONLINE ONLINE ora-node1
ora.mgmtdb ora....db.type ONLINE ONLINE ora-node1
ora....network ora....rk.type ONLINE ONLINE ora-node1
ora.oc4j ora.oc4j.type ONLINE ONLINE ora-node1
ora.ons ora.ons.type ONLINE ONLINE ora-node1
ora....E1.lsnr application ONLINE ONLINE ora-node1
ora....de1.ons application ONLINE ONLINE ora-node1
ora....de1.vip ora....t1.type ONLINE ONLINE ora-node1
ora....E2.lsnr application ONLINE ONLINE ora-node2
ora....de2.ons application ONLINE ONLINE ora-node2
ora....de2.vip ora....t1.type ONLINE ONLINE ora-node2
ora.scan1.vip ora....ip.type ONLINE ONLINE ora-node1
```
```
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE ora-node1 STABLE
ONLINE ONLINE ora-node2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE ora-node1 STABLE
ONLINE ONLINE ora-node2 STABLE
ora.OCR_VOTING.dg
ONLINE ONLINE ora-node1 STABLE
ONLINE ONLINE ora-node2 STABLE
ora.net1.network
ONLINE ONLINE ora-node1 STABLE
ONLINE ONLINE ora-node2 STABLE
ora.ons
ONLINE ONLINE ora-node1 STABLE
ONLINE ONLINE ora-node2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ora-node1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE ora-node1 169.254.48.110 172.1
16.0.81,STABLE
ora.asm
1 ONLINE ONLINE ora-node1 Started,STABLE
2 ONLINE ONLINE ora-node2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE ora-node1 STABLE
ora.mgmtdb
1 ONLINE ONLINE ora-node1 Open,STABLE
ora.oc4j
1 ONLINE ONLINE ora-node1 STABLE
ora.ora-node1.vip
1 ONLINE ONLINE ora-node1 STABLE
ora.ora-node2.vip
1 ONLINE ONLINE ora-node2 STABLE
ora.scan1.vip
1 ONLINE ONLINE ora-node1 STABLE
--------------------------------------------------------------------------------
```
####3. 配置ASM磁盘组
```
$ asmca
```
####4. 安装数据库软件
这里只是记录部分关键步骤。下载软件包，解压，运行`runInstall`。
完成数据库软件的安装，下面安装数据库。
####5. 安装数据库
```
$ dbca
```
```
$ crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE ora-node1 STABLE
ONLINE ONLINE ora-node2 STABLE
ora.DATA.dg
ONLINE ONLINE ora-node1 STABLE
ONLINE ONLINE ora-node2 STABLE
ora.FRA.dg
ONLINE ONLINE ora-node1 STABLE
ONLINE ONLINE ora-node2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE ora-node1 STABLE
ONLINE ONLINE ora-node2 STABLE
ora.OCR_VOTING.dg
ONLINE ONLINE ora-node1 STABLE
ONLINE ONLINE ora-node2 STABLE
ora.net1.network
ONLINE ONLINE ora-node1 STABLE
ONLINE ONLINE ora-node2 STABLE
ora.ons
ONLINE ONLINE ora-node1 STABLE
ONLINE ONLINE ora-node2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ora-node1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE ora-node1 169.254.48.110 172.1
16.0.81,STABLE
ora.asm
1 ONLINE ONLINE ora-node1 Started,STABLE
2 ONLINE ONLINE ora-node2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE ora-node1 STABLE
ora.mgmtdb
1 ONLINE ONLINE ora-node1 Open,STABLE
ora.mkdb.db
1 ONLINE ONLINE ora-node1 Open,STABLE
2 ONLINE ONLINE ora-node2 Open,STABLE
ora.oc4j
1 ONLINE ONLINE ora-node1 STABLE
ora.ora-node1.vip
1 ONLINE ONLINE ora-node1 STABLE
ora.ora-node2.vip
1 ONLINE ONLINE ora-node2 STABLE
ora.orcl.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.scan1.vip
1 ONLINE ONLINE ora-node1 STABLE
--------------------------------------------------------------------------------
```
