GPFS配置手册计算机类

江西省农信社GPFS安装手册V1.0目录1.江西省农信社GPFS规划21.1江西省农信社GPFS集群规划21.2集群Quorum方式31.3NSD磁盘组成31.4文件系统基本信息31.5GPFS系统环境示意图52.GPFS安装前准备52.1安装介质及补丁准备52.2Openssl及Openssh62.3Linux节点安装ksh62.4Linux节点修改网络参数63.GPFS安装配置73.1配置ssh信任关系73.2AIX安装GPFS软件103.3Linux安装GPFS软件113.4GPFS集群配置124.GPFS日常维护194.1GPFS启动194.2GPFS停止194.3查看GPFS使用状况201. 江西省农信社GPFS规划1.1 江西省农信社GPFS集群规划IP地址节点名操作系统集群角色10.0.147.98AIX 7100-02-02-1316Quorum-manager10.0.147.97AIX 7100-02-02-1316Quorum-manager10.0.147.49Redhat 6.4 for Powerclient10.0.147.50Redhat 6.4 for Powerclient10.0.147.51Redhat 6.4 for Powerclient10.0.147.52Redhat 6.4 for Powerclient10.0.147.53Redhat 6.4 for Powerclient10.0.147.54Redhat 6.4 for Powerclient10.0.147.55Redhat 6.4 for Powerclient10.0.147.56Redhat 6.4 for Powerclient10.0.147.57Redhat 6.4 for Powerclient10.0.147.58Redhat 6.4 for Powerclient10.0.147.59Redhat 6.4 for Powerclient10.0.147.60Redhat 6.4 for Powerclient10.0.147.61Redhat 6.4 for Powerclient10.0.147.62Redhat 6.4 for Powerclient10.0.147.63Redhat 6.4 for Powerclient10.0.147.64Redhat 6.4 for Powerclient10.0.147.65Redhat 6.4 for Powerclient10.0.147.66Redhat 6.4 for Powerclient10.0.147.67Redhat 6.4 for Powerclient10.0.147.68Redhat 6.4 for Powerclient10.0.147.69Redhat 6.4 for Powerclient10.0.147.70Redhat 6.4 for Powerclient10.0.147.71Redhat 6.4 for Powerclient10.0.147.72Redhat 6.4 for Powerclient10.0.147.73Redhat 6.4 for Powerclient10.0.147.74Redhat 6.4 for Powerclient10.0.147.75Redhat 6.4 for Powerclient10.0.147.76Redhat 6.4 for Powerclient10.0.147.77Redhat 6.4 for Powerclient10.0.147.78Redhat 6.4 for Powerclient10.0.147.79Redhat 6.4 for Powerclient10.0.147.80Redhat 6.4 for Powerclient10.0.147.81Redhat 6.4 for Powerclient10.0.147.82Redhat 6.4 for Powerclient10.0.147.83Redhat 6.4 for Powerclient10.0.147.84Redhat 6.4 for Powerclient10.0.147.85Redhat 6.4 for Powerclient10.0.147.86Redhat 6.4 for Powerclient10.0.147.87Redhat 6.4 for Powerclient10.0.147.88Redhat 6.4 for Powerclient10.0.147.89Redhat 6.4 for Powerclient10.0.147.90Redhat 6.4 for Powerclient10.0.147.91Redhat 6.4 for Powerclient10.0.147.92Redhat 6.4 for Powerclient10.0.147.93Redhat 6.4 for Powerclient10.0.147.94Redhat 6.4 for Powerclient10.0.147.95Redhat 6.4 for Powerclient10.0.147.96Redhat 6.4 for Powerclient1.2 集群Quorum方式集群Quorum方式采用Node Quorum + Tiebreaker disks,两个AIX节点加上一块专用的SAN磁盘,目前只有一个AIX节点,第二个AIX节点和SAN磁盘需要后续添加。1.3 NSD磁盘组成PV_NAMESIZENSD_NAMEFS_NAMENSD SERVERS LISThdisk2500Gbjxrccb_nsd1gpfshdisk3500Gbjxrccb_nsd2gpfs1.4 文件系统基本信息1.5 GPFS系统环境示意图GPFS Server: 2个AIX分区连接V7000SAN Storage: SVCGPFS Client: 48个Power Linux分区GPFS管理网络: 千兆交换机2. GPFS安装前准备2.1 安装介质及补丁准备安装包:GPFS v3.5.0.0补丁：GPFS-3.5.0.112.2 Openssl及Openssh确认AIX上openssl及openssh是否已安装,Linux默认已经安装;lslpp -l |grep open2.3 Linux节点安装ksh#rpm -ivh ksh-20100621-19.el6.ppc64.rpm2.4 Linux节点修改网络参数修改/etc/sysctl.conf文件,需修改的网络参数如下:# increase Linux TCP buffer limitsnet.core.rmem_max = 8388608net.core.wmem_max = 8388608# increase default and maximum Linux TCP buffer sizesnet.ipv4.tcp_rmem = 4096 262144 8388608net.ipv4.tcp_wmem = 4096 262144 8388608# increase max backlog to avoid dropped packetsdev_max_backlog = 2500修改完毕后执行sysctl -p /etc/sysctl.conf让配置生效3. GPFS安装配置3.1 配置ssh信任关系n 在节点上生成并配置ssh密钥对#ssh-keygen -t rsa# cd .ssh# lsid_rsa id_rsa.pub# cat id_rsa.pub authorized_keys# chmod 600 authorized_keys# echo StrictHostKeyChecking no config# chmod 600 confign 修改各节点的/etc/hosts文件,加入对集群中各节点的别名和ip地址10.0.147.49 10.0.147.50 10.0.147.51 10.0.147.52 10.0.147.53 10.0.147.54 10.0.147.55 10.0.147.56 10.0.147.57 10.0.147.58 10.0.147.59 10.0.147.60 10.0.147.61 10.0.147.62 10.0.147.63 10.0.147.64 10.0.147.65 10.0.147.66 10.0.147.67 10.0.147.68 10.0.147.69 10.0.147.70 10.0.147.71 10.0.147.72 10.0.147.73 10.0.147.74 10.0.147.75 10.0.147.76 10.0.147.77 10.0.147.78 10.0.147.79 10.0.147.80 10.0.147.81 10.0.147.82 10.0.147.83 10.0.147.84 10.0.147.85 10.0.147.86 10.0.147.87 10.0.147.88 10.0.147.89 10.0.147.90 10.0.147.91 10.0.147.92 10.0.147.93 10.0.147.94 10.0.147.95 10.0.147.96 10.0.147.97 10.0.147.98 n 验证本机无需密码的ssh登录#ssh hostnamen 将.ssh目录复制到其他各节点#scp -r /.ssh :/root/#scp -r /.ssh :/root/#scp -r /.ssh :/root/#scp -r /.ssh :/root/验证各节点间无密码的ssh登录3.2 AIX安装GPFS软件1） gpfs3.5.0安装包放在/home/gpfsaix/iso目录下2）切换/home/gpfsaix/iso目录下,通过smitty installp安装GPFS软件3) 安装补丁gpfs3.5版本，需要升级到3.5.0.11以上，补丁下载地址：下载后上传到服务器后解压：gzip -d -c filename.tar.gz |tar -xvf -使用smitty update_all安装补丁4）安装后检查#lslpp -l|grep gpfs如果之前已经创建了cluster，在所有node都down的状态下，加载GPFS版本到最新版本：mmchconfig release=LATEST;如果是新装的GPFS,则可以忽略此步。3.3 Linux安装GPFS软件1) 上传安装介质安装文件位于/home/gpfsrpm/，补丁文件位于/home/gpfsrpm/updte；2) 安装GPFS软件在安装介质所在目录下执行：#rpm -ivh gpfs*.rpm3) 升级GPFS软件版本在补丁文件所在目录下执行：#rpm -Uvh gpfs*.rpm4) 检查安装结果#rpm -qa |grep gpfs5) Building the GPFS portability layer#cd /usr/lpp/mmfs/src#make Autoconfig#make World#make InstallImages3.4 GPFS集群配置n 修改系统环境变量AIX平台下修改/etc/environment在PATH后增加/usr/lpp/mmfs/bin#vi /etc/environmentLinux平台下修改/root/.bashprofile,在PATH后增加/usr/lpp/mmfs/binn 创建集群按照规划，本次集群中包含1个AIX节点和48个Linux client；首先在1个AIX节点平台创建GPFS集群创建集群定义文件#cd /home#mkdir gpfs#cd gpfs#vi jxrcc_cluster10.0.147.98:quorum-manager a) 创建GPFS集群#mmcrcluster N jxrccb_cluster p 10.0.147.98 -r /usr/bin/ssh -R /usr/bin/scp -C jxrccb_cluster A# mmcrcluster -N jxrccb_cluster -p 10.0.147.98 -r /usr/bin/ssh -R /usr/bin/scp -C jxrccb_cluster -AWarning: Permanently added (RSA) to the list of known hosts.Tue Nov 19 20:01:38 CST 2013: 6027-1664 mmcrcluster: Processing node mmcrcluster: Command successfully completedmmcrcluster: 6027-1254 Warning: Not all nodes have proper GPFS license designations. Use the mmchlicense command to designate licenses as needed.b) 配置许可# mmchlicense server -accept -N 10.0.147.98# mmchlicense server -accept -N 10.0.147.98The following nodes will be designated as possessing GPFS server licenses: mmchlicense: Command successfully completedc) 添加Linux节点通过如下命令依次添加48个Linux节点：# mmaddnode N # mmchlicense server -accept -N # mmaddnode N # mmchlicense server -accept -N 查看集群状态#mmlsclusterd) 修改集群参数#mmchconfig prefetchThreads=1000 N 10.0.147.98# mmchconfig pagepool=4G N 10.0.147.98# mmchconfig failureDetectionTime=15# mmchconfig MaxMBps=4096e) 启动集群#mmstartup -a启动后，查看集群状态#mmgetstate -a确保所有节点的状态为Activen 创建NSD磁盘a) 修改fscsi参数在所有直接访问SAN盘的节点上执行#lsdev |grep fscsi|awk print chdev -l,$1,-a dyntrk=yes -a fc_err_recov=fast_fail|shb) 识别外置存储磁盘在所有直接访问SAN盘的节点上执行#cfgmgrc) 修改hdisk参数在所有直接访问SAN盘的节点上执行#lsdev -Cc disk|grep MPIO |awk print chdev -l,$1,-a algorithm=round_robin -a queue_depth=8 -a reserve_policy=no_reserve|shd) 创建NSD描述文件#cd /gpfs#vi nsd.confhdisk2: 10.0.147.98:jxrccb_nsd1:hdisk3: 10.0.147.98:jxrccb_nsd2:备注文件格式：DiskName:ServerList:DiskUsage:FailureGroup:DesiredName:StoragePool在本集群中，将AIX节点定义为NSD Server，如果NSD较多可以将每块盘的首选NSD Server设置为不同，以均衡NSD Sever的负载；e) 生成NSD盘#mmcrnsd -F nsd.conff) 查看NSD盘#mmlsnsd# mmlsnsd File system Disk name NSD servers - (free disk) jxrccb_nsd1 (free disk) jxrccb_nsd2 n 定义Tiebreaker disks集群处于Down状态时才能定义；#mmshutdown -a# mmchconfig tiebreakerDisks= 定义后可用如下命令查看；# mmlsconfign 创建GPFS文件系统#mmcrfs gpfs -F nsd.conf -A yes -B 512K -T /gpfs# mmcrfs gpfs -F nsd.conf -A yes -B 512K -T /gpfsGPFS: 6027-531 The following disks of gpfs will be formatted on node ABSGPFS1: jxrccb_nsd1: size 536870912 KB jxrccb_nsd2: size 536870912 KBGPFS: 6027-540 Formatting file system .GPFS: 6027-535 Disks up to size 4.4 TB can be added to storage pool system.Creating Inode FileCreating Allocation MapsCreating Log FilesClearing Inode Allocation MapClearing Block Allocation MapFormatting Allocation Map for storage pool systemGPFS: 6027-572 Completed creation of file system /dev/gpfs.mmcrfs: 6027-1371 Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.通过mmlsconfig或mmlsfs all命令可以看到所创建的文件系统#mmlsconfig挂载集群文件系统#mmmount all -a挂载后查看集群文件系统的mount状态#mmlsmount allFile system gpfs is mounted on 49 nodes.修改文件系统inode最大值#mmchfs gpfs -inode-limit 13568000修改挂载文件系统的最大节点数#mmchfs gpfs n=128查看文件系统的属性# mmlsfs allGPFS文件系统支持的最大inode数，可根据如下公式进行计算：maximum number of files = (total file system space) / (inode size + subblock size)备注: subblock size是创建文件系统时所指定Block size的1/32；例如：按文件系统400G，inode size为512 bytes，Block size为512k来计算maximum number of files = (600*1024*1024*1024) / (512 + 524288/32)= 381300364. GPFS日常维护4.1 GPFS启动 启动所有的节点：mmstartup a 启动单个节点：例如启动节点mmstartup N 4.2 GPFS停止GPFS停止步骤：umount挂载的FS停止所有的node umount挂载的FS：umount所有节点mmumount aumount 某一个节点：mmumount fsname N gpfswas2 停止所有的节点：mmshutdown a停止某一个节点：mmshutdown N gpfswas24.3 查看GPFS使用状况mmdf gpfs 21网络软硬件