Hadoop cluster (phase 1th) _centos installation configuration

What is CentOS?

CentOS is an enterprise-class Linux distribution that is freely available based on Red Hat Enterprise Linux . each version of CentOS is supported for seven years (via security update). The new version of CentOS is released once every two years , and each version of CentOS is updated periodically (approximately every six months) to support new hardware. This creates a secure, low-maintenance, stable, highly predictive, high-repeatability Linux environment.

CentOS (Community Enterprise Operating System) is one of the Linux distributions and is compiled from Red Hat Enterprise Linux based on source code released under open source rules. Because of the same source code, some servers that require a high degree of stability are used with CentOS instead of the commercial version of Red Hat Enterprise Linux. The difference between the two is that CentOS does not contain closed source code software .

CentOS is a community of open source software contributors and users. It compiles the Rhel source code and becomes one of the many communities that release new releases, and in the process of continuous development, the CentOS community is constantly merging with other similar communities, making CentOS Linux the most widely used version of Rhel compatible. The stability of CentOS Linux is no worse than Rhel, and the only shortage is lack of technical support because it is a free version released by the community.

CentOS Linux and RHEL products have a strict version of the relationship , such as the use of RHEL4 source code to recompile the release of the CentOS Linux 4.0,rhel5 corresponds to the CentOS Linux 5.0,rhel6 that corresponds to CentOS Linux 6. Due to the long life cycle of Rhel products (typically with 3-5 years of official support), Red Hat releases the cumulative updates to the released releases, often referred to as Rhel Update, every once in a while after the Rhel series is released.

For example, the 1th update version of RHEL5 is called Rhel 5 Update 1, and users are often referred to as Rhel 5.1. A corresponding update release will be released for each RHEL update CentOS community released by Red Hat, such as the Update 1 Updater source package based on Rhel 5, and CentOS compiles and packages the CentOS Linux 5.1 release. CentOS Linux and its corresponding version of the Rhel release have package-level binary compatibility, that is, if an RPM package can be installed and run in a Rhel product, it can be installed in the corresponding version of CentOS Linux properly. CentOS Linux is becoming more and more widely used because of its compatibility with Rhel and the stability of enterprise applications and the freedom of users.

CentOS Features

• CentOS can be understood as the Red Hat as series! It's all about making improvements to red Hat as! There are no differences between operation, use and red Hat!
• CentOS is completely free and there is no problem with red HAT AS4 requiring serial numbers.
• CentOS 's exclusive yum command supports online upgrades, updates the system instantly, and does not require the money to buy support services like Red Hat!
• CentOS fixes many red HAT as bug!
• CentOS Release Notes: CentOS3.1 is equivalent to Red Hat AS3 Update1 CentOS3.4 equivalent to Red Hat AS3 Update4 CentOS4.0 equivalent to Red Hat AS4.

1.2 Installation version

This installation we choose CetOS6.0 version, the following from a few aspects of this version of the introduction.

• integrates kernel-based virtualization . CentOS 6 integrates kernel-based virtualization to fully integrate the KVM hypervisor into the kernel. This feature helps CentOS 6.0 users easily migrate virtual machines between hosts and more flexibly deploy and manage IT resources, effectively saving resources for the enterprise. Leveraging the hardware abstraction of the kernel enables applications to be independent of the underlying hardware and increases CPU and memory scalability so that each server can accommodate more virtual machines.
• improve system and resource management capabilities . The standard-based Linux manageability specification (SBLIM) uses web-based enterprise Management (WBEM) to manage systems. Replacing MKINITRD with dracut minimizes the impact of underlying hardware changes, makes it easier to maintain, and enables easier support for third-party drivers.
• Enhanced power Management , on-time kernel improvements enables CentOS 6.0 to idle the processor without active tasks to reduce CPU temperature and further reduce power consumption.
• With the help of a new framework called Control group (i.e. cgroups), CentOS 6.0 provides granular control, allocation, and management of hardware resources . Cgroups runs at the process group level and can be used to manage resources from CPU, memory, network, and hard disk I/O for Applications.
• enhanced reliability, availability, and applicability of the system . CentOS 6.0 provides hot-swappable features with new hardware capabilities and can be checked for enhanced error by AER PCIe devices. CentOS 6.0 includes advanced data integrity features (DIF/DIX). This type of feature examines and verifies data from the application through hardware. The introduction of the Automated Defect Reporting Tool (ABRT) provides the identification and reporting of system anomalies, including kernel failures and user space application crashes.
• improved scalability and kernel performance . CentOS 6.0 provides the scalability to adapt to future systems, from the support of optimizations for a large number of CPU and memory configurations to the ability to handle a larger amount of system interconnect buses and peripherals. These capabilities are appropriate for bare-metal environments and virtualized environments as virtualization becomes ubiquitous with bare-metal deployments.
• CentOS 6.0 improves kernel performance by allowing higher-priority processes to allocate compute time between processors more equitably, with less-than-minimal priority handling of interference. At the same time, CentOS 6.0 improves multiple multi-processor lock synchronizations to eliminate unnecessary lock events, replace many rotation (spin) locks with sleep locks, and adopt more efficient locking primitives.
• Stable application development and production platform . CentOS 6.0 is a high-performance, highly scalable, distributed, memory-based object caching system that greatly improves the speed of dynamic Web applications. Improvements in the three areas of Apache, squid, and memcached have been largely improved on the web infrastructure. In Java performance support, the tight integration of CentOS 6.0 and OPENJDK includes support for Java probes in Systemtap to support better Java debugging. At the same time, CentOS 6.0 is also gradually improving the support of Tomcat 6.

Official website and Documentation

Official homepage: http://www.centos.org/

Official wiki:http://wiki.centos.org/

Official Chinese Document: Http://wiki.centos.org/zh/Documentation

Installation Instructions: http://www.centos.org/docs/

1.3 Hardware Information

The Linux distribution now has a strong hardware recognition capability , even if you do not query what hardware information, the installation of the operating system is not a big problem, but for very new Hardware and laptops are such special devices, it is recommended to check the network for information on the relevant hardware to ensure that the hardware can be driven. Here are a few of the hardware that you might want to focus on , and you need to be aware of the following issues for any one distribution.

• motherboard chipset Support . This is a relatively important set of hardware information that may affect whether the hard disk is available, can enable UDMA, and whether ACPI or APM power management can be used. The support program here is always related to the kernel version, which should pay attention to the corresponding kernel version.
• Network device Support . At present, for the general user is mainly Ethernet card and wireless card, some wireless network card driver has not been integrated into the kernel, but there is already a driver support.
• sound card . Linux's support for sound chips has historically been a big hit, but since the birth of ALSA (Advanced Linux Voice Architecture), the problem has been largely mitigated, and now ALSA is part of the 2.6 kernel. However, there are still some problems with the sound card. In particular, the majority of users currently use the motherboard onboard sound card, before installing the system should look at the Linux kernel on the motherboard chip sound function support.
• video card . Mainstream graphics card can now be supported, if necessary, look at the graphics card chip manufacturers, models, memory size basic can be, if you want to save time, you can consider the choice of performance, although not strong, but open source drive stability and performance of the best Intel integrated graphics card, The pursuit of high-performance graphics card users can consider the Nvidia chip and in the future use of the official driver to achieve better 3D performance, Ati/adm graphics card current Linux driver is not ideal, but AMD has opened up the technical specifications of the chip, the future will certainly be well supported by open source drivers.
• Other Peripherals . USB keyboard and mouse can generally be supported, a few of the lowest end of the hard disk box of the mobile hard disk may not be supported, the camera of these small hardware can be supported, and the other is some laptops, need to investigate.

In order to be prepared, the information to be examined is more, in addition to the Internet to find and ask around friends, the use of Live CD experience is also a very good way.

1.4 Hard Disk partitions

Installing a Linux system on a computer, partitioning the hard disk is a very important step, and here are a few of the partitioning schemes.

• Scenario 1 (Desktop)

/ boot: Used to store the Linux system boot-related programs, such as boot loader, etc., the recommended size is 100MB.

/: The root directory of the Linux system, all directories are hung under this directory, the recommended size is more than 5GB.

/home: The data for ordinary users is the host directory of ordinary users, the recommended size is the remaining space.

Swap: Implements virtual memory, the recommended size is one or more times of physical memory.

• Scenario 2 (server)

/ boot: Used to store the Linux system boot-related programs, such as boot loader, etc., the recommended size is 100MB.

/ usr: Used to store applications in Linux systems, with more relevant data, more than 3GB is recommended.

/ var: Used to store frequently changing data and log files in Linux systems, more than 1GB is recommended.

/home: The data for ordinary users is the host directory of ordinary users, the recommended size is the remaining space.

/: The root directory of the Linux system, all directories are hung under this directory, the recommended size is more than 5GB.

/ tmp: The temporary disk in a separate partition, you can avoid the system when the file system is full of stability affected. The recommended size is above 500MB.

Swap: Implements virtual memory, the recommended size is one or more times of physical memory.

2, CD-ROM installation

2.1 Installation Boot

The first step is to set the computer's BIOS boot order for the CD drive to boot, save the Setup CD into the CD drive, and restart the computer.

The interface shown will appear after the computer starts.

You can simply press <Enter> to enter the installation mode of the GUI or wait 60 seconds .

The following are the boot menu options shown below:

• Install or upgrade an existingsystem (installing or upgrading an existing system):

This option is the default. Select this option to install the system to your computer using the CentOS graphics installer.

• Install the system with basic video driver(Installation systems, basic audio drivers):

This option allows you to install the CentOS mode even if the graphics installer fails to load the correct driver for your video card. If there is a distortion or a blank space on your screen when you use the option to install or upgrade an existing system, restart your computer and try this option.

• Rescue installedSystem (Rescue installation):

Select this option to fix your installation of the CentOS system and prevent normal startup of your problem. Although CentOS is a very stable computing platform, it is accidental that the problem may still occur and prevent booting. The rescue environment contains utilities that allow you to solve these problems in a wide variety of applications.

• bootfrom the local drive (starting from localhost):

This option will boot the disk system from the first installation. If you accidentally start this disc, use the Setup program that is not started immediately this is the boot from hard disk option.

2.2 Detecting Disc Media

As shown, if it is a full installation disk, you can simply click the "Skip" button to skip, otherwise click the "OK" button to detect the integrity of the installation disk.

Note : If you are sure that the DVD or CD you downloaded is not a problem, then you can choose ' Skip ', but you can also press ' OK ' to analyze the DVD, because after analyzing the DVD, There are no strange problems with subsequent installations. But if you press ' OK ', the program will start analyzing all the files in the disc and it will take a lot of time!

2.3 Installation Welcome Screen

When the computer hardware information is detected, enter the installation Welcome screen as shown in.

2.4 Select the language during installation

Click on the "Next" button to enter the interface as shown, select the language used during installation, and select "Chinese (Simplified) (Chinese (Simplified)") here.

2.5 Choosing a keyboard layout type

After selecting the language in the installation process, click the Next button to enter the interface as shown, choosing a keyboard type generally defaults to "American English style (u.s.english)", which is the American keyboard, where the default selection is used.

2.6 Selecting a Device

Select a storage device to install. Basic storage is the default choice for installation space and is suitable for users who do not know which storage device should be selected. The specified storage device requires the user to designate the system installation to a specific storage device, a local device, or, of course, a SAN (storage LAN). Once the user has selected this option, they can add the FCOE/ISCSI/ZFCP disk and filter out the devices that the installer should ignore. Select " Basic storage Device " Here and click "Next" button.

Note : basic storage Devices : For desktops and notebooks, etc., specify storage devices : For servers, and so on.

2.7 Initializing the hard drive

If the partition table is not found on the hard disk, the Setup program requires that the hard disk be initialized . This action causes any existing data on the hard disk to be unreadable. If your system has a brand new hard disk that does not have an operating system installed, or deletes all partitions on the hard disk, click Reinitialize.

Note : The installer will provide you with a separate dialog box for each disk, and it cannot read a valid partition table. Click the Ignore All button, or the Reinitialize All button, to apply the same answer to all devices.

2.8 Setting host name and network

The installer prompts you to provide and the domain name for this computer's hostname format, setting the hostname and domain name. Many networks have DHCP (Dynamic Host Configuration Protocol) services that automatically provide a connection to the domain Name system, allowing the user to enter a host name. Unless you have a specific need to customize the hostname and domain name, the default setting of Localhost.localdomain is a good choice for most users. We use the following table to fill in the hostname and domain name, please see "Hadoop Cluster _ 2nd-Machine Information Distribution table ."

Table 2.8-1 Machine Information distribution

Machine name	IP Address
Master.hadoop	192.168.1.2
Salve1.hadoop	192.168.1.3
Salve2.hadoop	192.168.1.4
Salve3.hadoop	192.168.1.5

Set Fixed IP

Select "Configure Network" à "wired" à "eth0" à "edit", pop-up edit window Select "IPv4 Settings", open "method" side of the drop-down menu, select " manual ". Click the Add button, and then enter the IP, subnet mask , and Gateway for this computer. Enter the DNS address at the followingDNS server . Finally, click on the " Apply " button.

Follow the previous table 2.8-1 to configure the IP address.

2.9 Time zone selection

Because the world is divided into 24 time zones, tell the system where the time zone is. As shown, you can choose Beijing, or simply use the mouse to select on the map. To pay special attention to UTC, which is related to "daylight saving time", we do not need to select this option, otherwise it will cause time zone confusion, causing the system to be displayed in a different time than local.

2.10 Setting the Administrator password

The following is the most important "password for the system administrator" setting, as shown in. In Linux, the system administrator's default name is root, please note that this password is important . At least 6 characters or more, contain special symbols, and keep them in mind.

Note : When you are set to go to the next step, because your password may be set too simple, this will pop up a dialog box, if you want to stick to the password you set, you can choose " anyway is used ".

2.11 Disk partition Configuration

To facilitate the partitioning of hard drives, the CentOS presets give us the partitioning mode, respectively:

Note : We are actually directly selected is " use all Space ", then is the diagram convenient , but the actual work in accordance with the actual requirements of the hard disk partition, reasonable use of hard disk. Instead, select Create a custom layout and repartition the hard disk according to the server partitioning scheme given in the 1.4 section.

The following is a detailed description of each option:

• using all spaces (use all space ):

Select this option to delete all partitions on your hard disk (this includes partitions created by NTFS partitions VFAT or other operating systems, such as Windows).

• Replace existing Linux systems (replace Existing Linux system):

Select this option to eliminate partitions created by previous Linux installations. This does not remove other partitions (such as VFAT or FAT32 partitions) that you may have on your hard drive.

• Narrowing existing systems (Shrink current system ):

Select this option to adjust the current data and partition installed in the manually freed space is a default Red Hat Enterprise Linux layout.

• Use the remaining space (using free spaces ):

Select this option to keep your current data and partitions and install the available scientific on unused storage drives. Make sure that there is enough free space on the storage drive, and then select this option.

• Creating custom layouts (create custom layout):

Select this option to manually store the device for partitioning and create a custom layout.

Here are the steps to create a custom layout :

The first step : Select Create a custom layout, press the Next button, as shown in.

Step Two : Create the "/boot"

Select the free space you want to partition, and when you press create, the following screen appears. After selecting " Standard partition ", click "Generate".

mount point : Select "/boot"; file system type : Use the default "EXT4 log file system"; size : Enter the size of the allocation, in megabytes; Other Size options : Select fixed size , click OK button.

Step three : Create "/"

Continue to select free space, when you press "create", the following screen appears. After selecting "Standard Partition", click "Generate".

mount point : Select "/"; file system type : Use the default "EXT4 log file system"; size : Enter the size of the allocation, in megabytes; Other Size options : Select fixed size , click OK button.

Fourth Step : Create swap space

Continue to select free space, click "Create", you will see the following screen. After selecting "Standard Partition", click "Generate".

file System type : Select "swap"; size : Enter the size of the allocation, in megabytes; Other size options : Select fixed size . Click the OK button.

Note : Swap space is used to implement virtual memory, and the recommended size is one or more times of physical memory.

Fifth Step : Create "/usr"

Continue to select free space, when you press "create", the following screen appears. After selecting "Standard Partition", click "Generate".

mount point : select "/usr"; file system type : Use the default "EXT4 log file system"; size : Enter the size of the allocation, in megabytes for units; Other size options : Select fixed size ; Click OK button.

Sixth step : Create "/var"

Continue to select free space, when you press "create", the following screen appears. After selecting "Standard Partition", click "Generate".

mount point : Select "/var"; file system type : Use the default "EXT4 log file system"; size : Enter the size of the allocation, in megabytes; Other Size options : Select fixed Size , click the OK button.

Note : The figure and the fifth step are basically the same, so slightly.

Seventh Step : Create "/tmp"

Continue to select free space, when you press "create", the following screen appears. After selecting "Standard Partition", click "Generate".

mount point : Select "/tmp"; file system type : Use the default "EXT4 log file system"; size : Enter the size of the allocation, in megabytes for units; Other size options : Select fixed Size , click the OK button.

Note : The figure and the fifth step are basically the same, so slightly.

Eighth Step: "/Home" Partition

Continue to select free space, click "Create", you will see the following screen. After selecting "Standard Partition", click "Generate".

mount point : Select "/Home"; file system type : Use the default "EXT4 log file system"; other size options : Select " use all available space ", click " OK button.

At this point, the partition has all been created, if not satisfied, you can also click the "Reset" button to make changes. If you are sure, click the "Next" button, and then click "Format" to select "format".

The installer prompts you to confirm the partition options that you selected. Click "Write modifications to disk" to allow Setup to partition your hard disk and install system changes.

2.12 Boot Loader Setup

As shown in the Grub Boot installation window, you can use the default settings and click the Next button directly.

Note : To set the boot loader password, select "Use boot loader password" and enter the password in the popup window. We have not set this in practice .

2.13 Select the installed package

The optional server type is more, and the default installation is a very small, even incomplete system. Select Customize now, and then click the Next button, as shown in.

The optional types are described below:

• Desktop: A basic desk system, including common desktop software, such as document viewing tools.

• MinimalDesktop: A basic desk system that contains less software.

• Minimal: Basic system, does not contain any optional packages.

• Basic Server: Platform support for the basic system installed, not including the desktop.

• Database Server: Basic system platform, plus MySQL and PostgreSQL databases, no desktop.

• Web Server: Basic system platform, plus Php,web Server, with MySQL and PostgreSQL database clients, no desktop.

• Virtual Host: Basic system plus virtualized platform.

• Software Development Workstation: Includes many packages, basic system, virtualization platform, desktop environment, development tools.

Note : We install Linux here is used as a server, there is no need to install the graphical interface, so the choice is to minimize the installation (Minimal), if you want to installThe graphical interface, you can choose Desktop Installation (desk).

Because the last step we chose is to minimize the installation (Minimal), so in the following package Setup screen, all packages are not selected by default. Here we only select the packages we need to install.

First , check "basic" in " Basic System ".

then , select Development tools in development .

finally , in language support , select Chinese support . If you have a need to support other languages, you can also choose here. Finally click " next " to start the installation.

2.14 Starting the Linux system installation

Start the installation. In the installation screen, you will see how much time is required, the name of each package, and a brief description of the package, as shown in.

When the installation is finished, everything is finished. When the last screen appears, please take out the disc and press the "reboot" button to start, as shown in.

3. Environment setting

After the CentOS 6 has just been installed, the default security settings and service startup and so on may not match our needs, so before we build various system services, we first set the system environment to the most concise state, In the future to open a variety of services, in accordance with the needs of the service to carry out specific customization.

3.1 Creating a general user

The root user has the highest control over the entire system , if we use root in the general operation, it is easy to accidentally delete the system, error changes and other operations, so we first set up a general user, Use the average user to cope with daily system operations.

　　

After the general user establishment success, we can use the general user to log the system to carry on the daily system management. In some cases where root privileges are required, we can log in as the root user directly in the login state of the general user to perform the operation requiring root permission.

As shown above, we can log in directly to the root user using the "Su-" command in the presence of a normal user.

3.2 Shutting down the firewall and SELinux

This site document is assumed "server within the firewall" under the premise of the compilation, in this premise, the server itself in the internal network is not required firewall (in the case of the need to defend against internal security threats, you can also use your knowledge in the iptables, according to the truth set before fire, and ensure that the server-side firewall is in the state of boot). In addition, in general, we do not need the SELinux complex protection, so it is also set to an invalid state.

Shutting down the firewall

Turn off SELinux

Execute with the following command, and modify the contents as follows:

Vim/etc/sysconfig/selinux

Selinux=enforcing

↓

Selinux=disabled

Next, execute the following command:

Setenforce 0

Getenforce

3.3 Turn off services that you do not need

Use the following command to close services that you do not need:

for SERVICES  in Abrtd acpid auditd avahi-daemon cpuspeed haldaemon mdmonitor messagebus udev-post; Do chkconfig ${SERVICES} off; Done

Note : This document was written after a long time, when installing Linux, so the 3.2 and 3.3 of this section did not proceed, so that later encountered a lot of trouble, so here to put these missing to fill. In addition, all of this document is not actually installed at that time, but in order to write this documentation on the virtual machine installed on the graph, so the information on individual graphs reflects the hardware information of the virtual machine, in the actual is not the same, please pay attention to the minor differences during installation.

Hadoop cluster (phase 1th) _centos installation configuration