Kickstart installing CentOS, installing and configuring Hadoop at the same time

Disclaimer: This article is the front is merely, I am combining http://www.111cn.net/sys/linux/59969.htm and http://www.cnblogs.com/mchina/p/ Centos-pxe-kickstart-auto-install-os.html was written in two blog posts.

The later part about the IP is own

One: System environment

Experimental platform: Win7,vmware 12.0.0 build-2985596

System to install: CentOS 6.7 64-bit

Software to install: Hadoop 1.2.1,java

Assume that each machine's NIC is eth0

Second: Experimental steps

1. Install a CentOS 6.7 64 on VMware, the network uses bridging mode; This machine is a DHCP server, an HTTP server, a TFTP server Here, assuming its IP is 192.168.0.57, and for convenience, I call this machine server, all of the following configuration is done on the server.

2. Shut down the server's firewall and SELinux

Service Iptables Stop #临时关闭防护墙

Setenforce 0 #临时关闭selinux

3. Install and configure the HTTP service

Yum-y Install httpd #安装httpd

/ETC/INIT.D/HTTPD Start #启动http服务

Mkdir-p/var/www/html/#http服务默认的根目录就是这个文件夹, if not, create a new

HTTP service default port is 80, you can put a file in/var/www/html/, such as Test.txt, and then enter Http://192.168.0.57:80/test.txt in the browser to see if the display

4. Mount the CentOS ISO file

Use any method to find a CentOS ISO file, I use Centos-6.7-x86_64-bin-dvd1.iso, put it in a location in the server, I put in the/root folder (because it is the root operation of the server)

Mkdir-p/var/www/html/os #新建一个os文件夹用来存放ISO中的所有文件

Mount-t Iso9660-o Loop/root/centos-6.7-x86_64-bin-dvd1.iso/var/www/html/os #把ISO挂载到os文件夹下

Cd/var/www/html/os & ll #看看是否挂载成功

5. Configuring the TFTP service

Yum-y Install Tftp-server

Modify configuration file:/etc/xinetd.d/tftp

Service TFTP {         Socket_type             = dgram         protocol                = UDP         Wait                    = yes         user                    = root         Server                  =/usr/sbin/in.tftpd         Server_args             =-s/var/lib/tftpboot         Disable                 = no                         #修改的是这行, The original is yes, change to no        per_source              = one         cps                     = 2         flags                   = IPv4}

/etc/init.d/xinetd Restart #因为tftp服务是挂载在超级进程xinetd, so start the TFTP service by starting the xinetd.

6. Configuring the DHCP service

Yum-y Install DHCP

Cp-f/usr/share/doc/dhcp-4.1.1/dhcpd.conf.sample/etc/dhcp/dhcpd.conf #注意这里的4.1.1 is a version of DHCP and may not be the same

Modify the/etc/dhcp/dhcpd.conf configuration file with the following content:

Ddns-update-style Interim; Ignore client-updates; 　　FileName "pxelinux.0";　　      #pxelinux boot file location; Next-server 192.168.0.57; #这里是server的IP地址subnet 192.168.111.0 netmask 255.255.255.0 {        option routers                  192.168.0.57;    #这里还是server的IP地址        option Subnet-mask              255.255.255.0;        Range DYNAMIC-BOOTP 192.168.0.2 192.168.0.200;   #这是分配的IP地址的范围        default-lease-time 21600;         Max-lease-time 43200;}

/ETC/INIT.D/DHCPD Start #启动DHCP服务

7. Configuring PXE-enabled boot programs

cp/usr/share/syslinux/pxelinux.0/var/lib/tftpboot/

cp/var/www/html/cdrom/images/pxeboot/initrd.img/var/lib/tftpboot/

cp/var/www/html/cdrom/images/pxeboot/vmlinuz/var/lib/tftpboot/

cp/var/www/html/cdrom/isolinux/*.msg/var/lib/tftpboot/

Mkdir/var/lib/tftpboot/pxelinux.cfg

Vi/var/lib/tftpboot/pxelinux.cfg/default #default中的所有内容如下,

Default KS #默认启动的是 ' label KS ' in the boot kernel labeled prompt 1          #显示 ' boot: ' Prompt. is not prompted for ' 0 ', the content specified in the ' default ' parameter will be started directly. Timeout 6 #在用户输入之前的超时时间, unit 1/10 seconds. Display Boot.msg  #显示某个文件的内容, note the path to the file. The default is under the/var/lib/tftpboot/directory. You can also refer to a path + file name similar to '/install/boot.msg '. F1 boot.msg #按下 ' F1 ' Such a key is displayed after the file. F2 options.msg F3 general.msg F4 param.msg F5 rescue.msg label Linux       # ' label ' Specify the keywords you entered at the ' Boot: ' prompt, such as Boot:linux[en TER], this will launch ' label Linux ' under the tagged kernel and initrd.img files.  the kernel vmlinuz  #kernel parameter specifies the kernel to start.  Append initrd=initrd.img #append Specify the parameters appended to the kernel, and the parameters appended to the kernel that can be used in grub are also available here. Label Text   kernel vmlinuz   append initrd=initrd.img text label ks   kernel vmlinuz append   ks=http:// 192.168.0.57/ks.cfg initrd=initrd.img    #告诉系统, where to get the Ks.cfg file, where the IP is the Iplabel local   localboot 1 label of the server Memtest86   kernel memtest   Append-

8. Install the kickstart and configure the Ks.cfg file

Yum Install System-config-kickstart

If you already have x Windows and desktop installed, or a server that is operating in a graphical environment, run the command

System-config-kickstart #启动图形界面, configuring Ks.cfg

If there is no graphical interface, it does not matter, you can create a ks.cfg.

Follow the figure below to save the last file-to-/var/www/html/folder with the file name Ks.cfg

　　　　

　　　　

　　　　

　　　　

　　　　

　　　　

　　　　

　　　　

　　　　

　　　　

All of the configuration items for the server are now ready.

10. Experiments

And then use VMware to create a new blank hard disk of the virtual device, network or bridge mode, after starting to see a line of dhcp:/and then entered the installation

Regarding IP, there are more than n machines in the Hadoop cluster, I need to know their IP and set them to the DNS server, then we need a method to collect all the ip! of the machine. In addition, each machine's IP should be fixed, should not be obtained through the dynamic DHCP!

1. With regard to IP collection, I am thinking of configuring NFS on the server, assuming that the/ips folder is shared, mount this folder on each machine, and create a new file in it, the name of the file is the IP of the machine.

2. Set the static IP, this can only be done by modifying the/etc/sysconfig/network-scripts/ifcfg-eth0

Do you remember the last picture of the 9th step above? He is setting post-installation script, then we can write a script to implement the above 1, 22 steps, the following script to copy to the post-installation script, of course, the script also implements other functions, I have comments.

Other features include adding Hadoop users, downloading Hadoop and Java remotely, configuring the public key of the master server for Path,mount Hadoop, configuring DNS, and (if sharing Hadoop's configuration Files with NFS).

#declare-I number= $RANDOM *100/32768 #这两行我注释了, it is best to uncomment if the cluster is large. Because many of the following operations will access the HTTP server, if the cluster is very large simultaneous access, the HTTP server may explode, so let the main thread sleep a random number of seconds before starting to do the following work #sleep ${number} 
          

Nfs_server_id=192.168.0.57 #不是说要用NFS吗? This is the address of the NFS server. nfs_server_path_ips=/IPs #NFS服务器共享出去的文件夹, the role of this folder is to store the name is the file of each machine IP
Nfs_server_path_hadoop_conf=/conf #干脆hadoop的配置文件也有NFS服务器提供吧 ^_^. The purpose of this folder is to store all of Hadoop's configuration files, but doing so actually increases the risk if the NFS server hangs up ... 
Nfs_server_path_ssh=/ssh #要用NFS共享hadoop的master的公钥, the purpose of this folder is to store the master's public key Pxe_url=http://192.168.0.57 #http服务器的地址, I also provide a download of Hadoop and Java, remember that there is a folder/var/www/html it? Keep both Hadoop and Java GZ compressed files under/var/www/html. root_dir=/opt/Modules # Installation address for Hadoop and Java DNS=192.168.0.116 #大型集群肯定用DNS服务器啦, this is the address of the DNS server

#yum-y install rpcbind nfs-utils #每个datanode也是NFS的一个客户端, I have to load the client software, I put this line commented, because when the Ks.cfg file is generated, the PAC Kage selection can choose Servers-NFS file server/etc/init.d/rpcbind start #就算是NFS的client, you have to start both services/etc/ini t.d/nfslock start chkconfig rpcbind on #开机启动, otherwise Hadoop will not have conf files!!! Chkconfig nfslock     on
Useradd Hadoop #添加一个用户 HadoopEcho '123456'|passwd--stdin Hadoop #把hadoop用户的密码设置为 123456mkdir-P ${root_dir} # new installation root for Hadoop and Java CD ${root_dir} #进入安装根目录 #get Hadoop and Java wget "${pxe_url}/jdk-7u79-linux-x64.tar.gz" "${pxe_url}/hadoop-1.2.1.gz" #从http服务器上下载hadoop和java Tar-XF jdk-7u79-linux-x64.Tar. GZ #解压hadoopTar-XF hadoop-1.2.1. GZ #解压javaChown-R Hadoop ${root_dir} #把安装根目录下的所有文件和文件夹的所有者改成hadoop
RM-RF ${root_dir}/hadoop-1.2.1/conf/* #把hadoop现有的配置文件先都删除了
Mount-t NFS ${nfs_server_id}:${nfs_server_path_hadoop_conf} ${root_dir}/hadoop-1.2.1/conf/ # Mount the remote Hadoop configuration file locallyROUTE=$ (route-n|grep "^0.0.0.0"|awk '{print $}') #一直到下面的红色的EOF都是设置静态IP, and also set the Dnsbroadcast=$ (/sbin/ifconfigeth0|grep-I. bcast|awk '{print $}'|awk-F":" '{print $}') HWADDR=$ (/sbin/ifconfigeth0|grep-I. hwaddr|awk '{print $}') ipaddr=$ (/sbin/ifconfigeth0|grep "inet Addr"|awk '{print $}'|awk-F":" '{print $}') NETMASK=$ (/sbin/ifconfigeth0|grep "inet Addr"|awk '{print $4}'|awk-F":" '{print $}')Cat>/etc/sysconfig/network-scripts/ifcfg-eth0 <<Eofdevice=Eth0bootproto=Staticbroadcast=$BROADCASTHWADDR=$HWADDRIPADDR=$IPADDRNETMASK=$NETMASKGATEWAY=$ROUTErooONBOOT=yesDNS1=$DNSEOF mkdir-P/IPsMount-T NFS ${nfs_server_id}:${nfs_server_path_ips}/IPs #把NFS远程文件夹mount到本地Touch/ips/${IPADDR} #在远程文件夹中新建一个文件, the file name is the IP of the machine, then the IP of all the machines in the entire cluster are collected in the NFS server
Echo "Export java_home=${root_dir}/jdk1.7.0.79">>/home/hadoop/. Bash_profile #这三行只是为了操作java和hadoop时方便, or you may notEcho "Export hadoop_home=${root_dir}/hadoop-1.2.1">>/home/hadoop/. Bash_profileEcho "Export Path=\${java_home}/bin:\${hadoop_home}/bin:\${hadoop_home}/sbin:\${path}">>/home/hadoop/. Bash_profileservice network Restart #重新启动网络服务, or here directly reboot can, anyway is in the installation system, install the system restart is not very normal 
    

　　

Kickstart installing CentOS, installing and configuring Hadoop at the same time