Blue growth note-chasing DBA (14): unforgettable "Cloud" end, initial hadoop deployment, dbahadoop

Blue growth note-chasing DBA (14): unforgettable "Cloud" end, initial hadoop deployment, dbahadoop

Original Works are from the blog of "Deep Blue blog". You are welcome to repost them. Please indicate the source when you repost them. Otherwise, the copyright will be held legally responsible.

Deep Blue blog:Http://blog.csdn.net/huangyanlong/article/details/44050117

 

[Overview]

Personal Growth records on the oracle Road, expressed in blue, share the growing emotions, vision and technological changes and growth. Sensitive information is removed in other forms, and no company secrets are disclosed. This is purely for technology sharing.

The creative inspiration comes from self-reflection and recording. I'm glad to have some help or resonance with the database friends who just started out.

Please leave a message or email (hyldba@163.com) indicating that the technical details are incorrect. Thank you very much.

[Preface]

This is an accumulation of personal records. As we enter the blue ocean of oracle, we can't go all the way and continue to test. Share the growth history of blue with database friends.

I don't know when I began to become obsessed with blue. I was obsessed with its wide range, its depth, and its close proximity.

But it cannot be clear from when, watching the oracle red dazzling, illuminating a light in front of the eyes, unknown and confused under their own feet began to reveal some of the enrichment of life and the feedback of youth.

Step by step on the path to pursue DBA's dream.

 

Chapter 1: Exploring the cloud

People with no goals feel wandering wherever they go. On the contrary, if they establish their goals, they will also feel the enrichment of approaching their dreams.

-- Deep Blue

Recalling in Zibo in 2014

Quiet and pleasant, depending on the plan,

Platform transformation, cloud exploration,

Simple question, experience cost.

 

When I first came to Zibo, I was impressed by Ning Xiang. There were no mediocre people, no noisy quarrels, a slow pace of life attitude and a peaceful life principle.

This task is relatively simple. It is to migrate the WAS application service to a server on the cloud platform. Based on previous implementation experience, it is still smooth. But I have no idea about the concept of "Cloud. When I completed my job and communicated with the developers of another Development Company, I first got in touch with hadoop. Although a little confused, a little busy, it sounds like a cloud, so it's a little bit of something to do ~~ As described above, it seems hard to touch.

I am confused, but I still have some gains. I have learned a few terms and what cloud strategic deployment is ".

When it comes to "Cloud", the goal it has achieved has been recognized for the first time. It briefly summarizes the following four points:

1. Share resources, centralize and distribute hardware resources, and improve the utilization rate of software resources and hardware resources;

2. Shared data and centralized data storage improve information sharing, statistical analysis, and deep data mining capabilities;

3. Shared Services: ultra-large data computing, ultra-powerful computing, and ultra-large storage capabilities;

4. Shared applications: develop and upgrade Application Systems Based on the cloud platform.

Next is another migration task in Jinan. With the "Cloud" Thoughts of Yuwen, I set foot on the bus to Jinan.

The task of Jinan is still to migrate to the cloud platform. However, this time I did not know much about the "Cloud", and I am a little sorry. As a pure visitor, I hurried to go.

Chapter 2: Be lucky to touch the distributed architecture and learn hadoop implementation

Embrace big data and distributed databases.

-- Deep Blue

In the young IT industry, oracle emerged in this era because the era has given IT a mission. In the future, technologies around data development may be overwhelming. Just like hadoop, we may not be able to predict it. But before all this comes, we need to keep up with the boosting agents of this wave of "Big Data" as soon as possible. In the future, the data format will inevitably follow the historical development context. The development begins with theory, but will eventually be applied to practicality. On the other hand, I believe that for technical development, one day IT will be "unified" by perfect and efficient technologies. At that time, IT must be a sign that the IT field has entered its heyday. Along with the database technology, we believe it will also be caused by the endless technical disputes, which gives us a clear, clear, and efficient background data architecture.

We are not sure about the future. We can only be based on the present and make a lot of effort.

Thanks to the development and transformation of the company's system against hadoop, we were lucky to have first access to hadoop. Next, we will give a simple summary of the mistakes made.

For specific deployment details, we will not list them for the moment. We will only list the general ideas for building them. I hope I can have a preliminary understanding of hadoop deployment:

1. Install CentOS 6.4 Operating System (64bit); 2. Configure the network and connect to the operating system remotely. 3. Install JDK 4, modify the host name, and configure the host file (multi-node IP resolution) 5. Disable firewall 6. Disable SElinux7, configure JAVA_HOME8, configure yum Source 9, add cloud users 10, install cloud Management Console 11, and install nameNode nodes (master node) through the Management Console) 12. Install dataNode nodes (distributed multi-node) on the console)

Looking at this process, can you feel like "Cloud" is not that mysterious and unpredictable. However, in this installation process, I still made several low-level mistakes and made a summary:

First hurdle: HTTP service

Incorrect yum Configuration

During hadoop deployment, you must configure the yum source to complete subsequent installation. Yum source configuration is very simple for old birds familiar with linux. However, the mistake I made this time is due to incomplete understanding of yum configuration. In the past, yum was the required package for yum tool installation during oracle installation. The path file starting with file is "file: // cdrom1. This is of course simple. You only need to mount the disc to the cdrom1 directory. The repo source file obtained from R & D this time uses the path "http. This is actually not difficult. You need to build an http server and map the IP address of the http server to the hosts parsing file.

Here we use apache to build an http server and download the following files from the website:

Click Next To Go To The following configuration page for local configuration.

After the installation is complete, a cmd error prompt box is displayed. Press esc to exit. The error is displayed because the default port 80 of the http server is occupied by other devices on the computer. After the installation is complete, modify the port number in the configuration file.

For example, the apache service cannot be started after the installation is complete.

Find the configuration file and open it in the editor, for example:

Change the port number from 80 to 8080, for example:

After modification, start the apache service again, for example:

Enter the http server address in the browser. If the server is successfully installed, enter localhost to verify the local server. Enter the IP address when the server is remote. For example:

In this case, you only need to replace the file in the default path with the file pointed to by the yum source.

For example:

Try to see the effect, such:

We recommend that you use win2008 to configure the http server and configure it to use port 80.

After configuring the http server, you can configure path addresses in the "http: //" format in the yum source, for example:

Baseurl = Signature:

<span style="font-size:14px;"># vi /etc/hosts192.168.56.1  archive.cloudera.com</span>

Second hurdle: cloudera console installation error-Incorrect http path

After configuring the yum source and executing the binfile on the cloudera console, the following error is returned:

 

View the error log as prompted:

[Root @ master opt] # cat/var/log/cloudera-manager-installer/3. install-cloudera-manager-server.log

Loaded plugins: fastestmirror, refresh-packagekit, securityLoading mirror speeds from cached hostfileSetting up Install ProcessResolving Dependencies--> Running transaction check---> Package cloudera-manager-server.x86_64 0:4.8.0-1.cm480.p0.50.el6 will be installed--> Processing Dependency: cloudera-manager-daemons = 4.8.0 for package: cloudera-manager-server-4.8.0-1.cm480.p0.50.el6.x86_64--> Running transaction check---> Package cloudera-manager-daemons.x86_64 0:4.8.0-1.cm480.p0.50.el6 will be installed--> Finished Dependency ResolutionDependencies Resolved================================================================================ Package                  Arch   Version                 Repository        Size================================================================================Installing: cloudera-manager-server  x86_64 4.8.0-1.cm480.p0.50.el6 cloudera-manager 7.7 kInstalling for dependencies: cloudera-manager-daemons x86_64 4.8.0-1.cm480.p0.50.el6 cloudera-manager 147 MTransaction Summary================================================================================Install       2 Package(s)Total download size: 147 MInstalled size: 201 MDownloading Packages:http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4.8.0/RPMS/x86_64/cloudera-manager-daemons-4.8.0-1.cm480.p0.50.el6.x86_64.rpm: [Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=cloudera-manager clean metadataTrying other mirror.http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4.8.0/RPMS/x86_64/cloudera-manager-server-4.8.0-1.cm480.p0.50.el6.x86_64.rpm: [Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=cloudera-manager clean metadataTrying other mirror.Error Downloading Packages:  cloudera-manager-daemons-4.8.0-1.cm480.p0.50.el6.x86_64: failure: RPMS/x86_64/cloudera-manager-daemons-4.8.0-1.cm480.p0.50.el6.x86_64.rpm from cloudera-manager: [Errno 256] No more mirrors to try.  cloudera-manager-server-4.8.0-1.cm480.p0.50.el6.x86_64: failure: RPMS/x86_64/cloudera-manager-server-4.8.0-1.cm480.p0.50.el6.x86_64.rpm from cloudera-manager: [Errno 256] No more mirrors to try.

Solution:

(1) Finally, lock the problem to the directory as follows:

Http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4.8.0/RPMS/x86_64/ cloudera-manager-server-4.8.0-1.cm480.p0.50.el6.x86_64.rpm

(2) Go to the http server to view the discovery path, as shown below:

Http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4.8.0/RPMS/ cloudera-manager-server-4.8.0-1.cm480.p0.50.el6.x86_64.rpm

Comparison (1) and (2) can be found that the second x86_64 directory is missing in the http server, so the x86_64 directory is added to the http server.

Run the command again and no longer report this error.

 

Supplement:

When adding a directory to an http server, if an access directory contains a bootstrap "index file (html directory file)", if you need to delete it, otherwise, a new directory cannot be created.

Supplemented.

 

Third hurdle: cloudera console installation error-the postgresql package is missing

When the path problem is solved and the binfile of cloudera is installed again, an error is reported again, as shown below:

The system prompts you to view the log file, so you can view the configuration file again as follows:

[Root @ master opt] # cat/var/log/cloudera-manager-installer/4. install-cloudera-manager-server-db.log

Loaded plugins: fastestmirror, refresh-packagekit, securityLoading mirror speeds from cached hostfileSetting up Install ProcessResolving Dependencies--> Running transaction check---> Package cloudera-manager-server-db.x86_64 0:4.8.0-1.cm480.p0.50.el6 will be installed--> Processing Dependency: postgresql-server >= 8.0 for package: cloudera-manager-server-db-4.8.0-1.cm480.p0.50.el6.x86_64--> Finished Dependency ResolutionError: Package: cloudera-manager-server-db-4.8.0-1.cm480.p0.50.el6.x86_64 (cloudera-manager)           Requires: postgresql-server >= 8.0You could try using --skip-broken to work around the problemYou could try running: rpm -Va --nofiles –nodigest

Solution:

Check the log and find that the postgresql-server version 8.0 or later is required for installation. Therefore, use the yum method (the postgresql-server package has dependency and needs to be installed using yum) for installation. Follow these steps:

[Root @ master yum. repos. d] # yum install-y postgresql-server *

Loaded plugins: fastestmirror, refresh-packagekit, securityLoading mirror speeds from cached hostfileSetting up Install ProcessResolving Dependencies--> Running transaction check---> Package postgresql-server.x86_64 0:8.4.13-1.el6_3 will be installed--> Finished Dependency ResolutionDependencies Resolved================================================================================================================ Package                         Arch                 Version                         Repository           Size================================================================================================================Installing: postgresql-server               x86_64               8.4.13-1.el6_3                  debug               3.4 MTransaction Summary================================================================================================================Install       1 Package(s)Total download size: 3.4 MInstalled size: 14 MDownloading Packages:Running rpm_check_debugRunning Transaction TestTransaction Test SucceededRunning TransactionWarning: RPMDB altered outside of yum.  Installing : postgresql-server-8.4.13-1.el6_3.x86_64                                                      1/1   Verifying  : postgresql-server-8.4.13-1.el6_3.x86_64                                                      1/1 Installed:  postgresql-server.x86_64 0:8.4.13-1.el6_3                                                                     Complete!

After three twists and turns, the console has been successfully installed.

Next, complete the configuration (Procedure omitted). After completing the hadoop configuration, enter the management interface, as shown in:

This is a start for hadoop learning. In the face of the booming "Database" field, it seems that apart from oracle, we should also add some new technologies.

 

 

Series links:

Blue growth note-chasing DBA (1): traveling on the road to Shandong 

Blue growth notes-Chase DBA (2): Install! Install! Long-lost memories have aroused my new awareness of DBAs.

Blue growth note-chasing DBA (3): importing and exporting data on antiques becomes a problem 

Blue growth note-chasing DBA (4): recalling the sorrow of teenagers, and exploring oracle Installation (10g and 11g in Linux) 

Blue growth note-chasing DBA (5): Not talking about technology or business, annoying Application Systems

Blue growth note-chasing DBA (6): doing things and being human: Small technology, great human

Blue growth note-Chase DBA (7): Basic commands, foundation stone

Blue growth notes-chasing DBA (8): repicking SP reports and recalling oracle's STATSPACK Experiment

Blue growth-chasing DBA (9): Chasing DBA, new planning, new departure

Blue growth note-chasing DBA (10): Flying knife defense, familiarity rather than expertise: Playing with middleware Websphere

Blue growth note-chasing DBA (11): It's easy to go home and wake up.

Blue growth notes-Chase DBA (12): seven days and seven gains of SQL

BlueGrowth story-chasing DBA (13): Coordinating hardware manufacturers. Six Stories: What you see: servers, storage, switches ......"

 

 

Original Works are from the blog of "Deep Blue blog". You are welcome to repost them. Please indicate the source when you repost them. Otherwise, the copyright will be held legally responsible.

Deep Blue blog:Http://blog.csdn.net/huangyanlong/article/details/44050117