Big Data System Toolset

bootstrapping boot:Kickstart, Cobbler, Rpmbuild/xen, KVM, LXC, Openstack, Cloudstack, Opennebula, Eucalyplus, RHEV
Configuration class Tools:Capistrano, Chef, puppet, Func, Salstack, Ansible, Rundeck
Monitoring class Tools:Cacti, Nagios (Icinga), Zabbix, time-based monitoring front-end Grafana, mtop
Log System:Logstash, Scribe
Drawing tools:RRDtool, Gnuplot
Flow control System:Panabit
PAAs platform as a service:Cloudify, Cloudfoundry, Openshift, Deis (Docker, CoreOS, Atomic)
Troubleshooting Troubleshooting:Sysdig, Systemtap, Perf
Disk pressure test:Fio, IOzone, IOMeter (Win)
MySQL HA:Mmm/mha/heartbeat+drbd/heartbeat+san/ndb cluster/rhcs+drbd/galera+xtradb Cluster/mmm+amoeba
MySQL Monitoring:Mytop, Orzdba, Percona-toolkit, Maatkit, Innotop, Myawr, SQL-level monitoring MYSQLPCAP
MySQL benchmark test:Mysqlsla, Sql-bench, Super Smack, Percona ' s tpcc-mysql Tool, Sysbench
MongoDB pressure measurement:Iibench&sysbench

Capistrano

Capistrano is an open source tool that runs scripts on multiple servers and is primarily used to deploy web apps. It automates the synchronization update of new versions on multiple servers, including changes to the database. Capistrano was originally developed by Jamis Buck in Ruby and deployed with RubyGems deployment channels. Now Capistrano is not limited to applying the Ruby on Rails Web application framework, but it can also be used to deploy Web applications with other frameworks, such as those developed with PHP. (Project details)

Code Escrow Address : Https://github.com/capistrano/capistrano

recommend related documents :

Capistrano Development Diary (1) (2)
Automated deployment practices Capistrano
Write Ruby edge with Capistrano to deploy iterations
Write a simple deploy script with Capistrano
Using the Capistrano--rails App Quick Deploy tool
Deploy Apache+mongrel cluster with Capistrano

Chef

A system Integration framework that provides configuration management for your entire base device. With chef, you can:

Write code to manage your server instead of running commands (via cookbooks)
Integrate tightly into your application, database, LDAP directory, etc... (through the class library);
Easily configure your application, but need to know what system your infrastructure is running on? What is the current primary database server? ）
Basically, chef is a Ruby configuration management engine. You provide the recipe, how you want your system to be configured, and then hand it over to chef chef, which will configure everything you want. You can write cute ruby code to manage your servers without the need to use commands to execute them. (Project details)

Code Escrow Address : Https://github.com/opscode/chef

Recommended Download Resources :

Chef detailed Configuration
Chef Fundamental
Chef-infrastructure-automation-cookbook-ebook.pdf
Automated Chef Cookbook testing with Drone.io and GitHub

Docker

Docker is DotCloud open source, a tool that can be packaged in Linux container for any application, and released its first version in March 2013. When the app is packaged into Docker image, deployment and operations become extremely simple. You can download, launch, expand, Delete, and migrate in a unified manner. Docker can be used to: automate the packaging and deployment of any application, create a lightweight, private PAAs cloud, build a development test environment, and deploy scalable Web applications. (Project details)

Project home: Http://docker.io

Code Escrow Address: Https://github.com/dotcloud/docker

Recommend related documents:

From CoreOS to Docker to Golang
[Docker] First Contact Docker introduction: Related Technologies (LXC)
Docker create MySQL container some tips and tricks for Docker
Easily deploy Docker services on CentOS

Recommended Download Resources:

Docker on Google App Engine
Docker Getting Started Tutorial
Docker_mongodb
Docker the road ahead
Docker Chinese community: Docker with Openstack.pdf
Seven Qiniu storage chief Evangelist Xu Li: the Docker
Guiyang: Automates deployment of Docker in CoreOS with workflow

Logstash

Logstash is a platform for application logging, event transfer, processing, management, and search. (Project details)

Project home: http://logstash.net/

Code Escrow Address: Https://github.com/elasticsearch/logstash

Recommend related documents:

Analyzing Apache logs with Logstash
Log file monitoring Tool-Logstash
Integration of Logback and Logstash
Services for billions of customer searches with Elasticsearch and Logstash
Use Logstash+elasticsearch+kibana to quickly build a log platform
Logstash Open Source Log management System -2-logstash Configuration language Introduction

OpenStack

OpenStack is a cloud-based open source project jointly launched by Rackspace and NASA in July 2010 to provide a large-scale cloud operating system that supports an IAAS platform similar to AWS capabilities. It is now the largest open-source community behind Linux, with members covering almost all major IT vendors. OpenStack is widely deployed between internet companies and traditional enterprises, and many startups have been born. OpenStack has a very good architecture, which is reflected in all the functions of the module and the API, the module loosely coupled. (Project details)

Project home: http://www.openstack.org/

Code hosting Address: Https://github.com/openstack/openstack Recommended Documentation:

How to learn OpenStack and how to become an OpenStack engineer?
How far Openstack can go--openstack, vmware analysis
A probe into the cinder service of "OpenStack" OpenStack
"OpenStack" builds OpenStack UT environment on OpenStack
OpenStack Learning notes--openstack Nova architecture
Recommended Download Resources:

OpenStack Fast Advanced
OpenStack Operations Guide
OpenStack Basics Explained
OpenStack Installation and Configuration tutorial Super detailed
OpenStack Cloud computing Platform Management tutorial Download OpenStack Getting Started Tutorial

Puppet

You can use Puppet to centrally manage every important aspect of your system using a cross-platform specification language, which manages the separate elements that are typically clustered in different files, such as users, cron jobs, and hosts, along with apparently discrete elements such as packaging, services, and files. Puppet's simple statement of the ability to standardize languages provides a powerful classing to develop similarities between hosts, while enabling them to provide as specific a necessary as possible, and it relies on a clear and unambiguous relationship between the prerequisites and the objects. (Project details)

Code Escrow Address : Https://github.com/puppetlabs/puppet

recommend related documents :

Puppet configuration of the puppet.conf detailed in English and Chinese control
Open Source Automation Deployment management Tool Puppet Installation
Centralized operation and maintenance management--puppet Management Road
Puppet core resource types and their common attributes learning notes
Automated Operation Koriyuki Puppet

Recommended Download Resources:

Managing infrastructure using the puppet framework
Puppet the most classic Chinese manual materials
[Proficient in Puppet configuration management Tools]. Shou. Scanned version
Puppet service-side installation process
Puppet installation and testing on the cluster

Statsd

STATSD is a network application running on the node. JS platform that can be used to listen to UDP port information and generate real-time graphs of the monitored data. STATSD 0.1. Version 0 was released by Etsy on February 16, 2012. (Project details)

Code Escrow Address : HTTPS://GITHUB.COM/ETSY/STATSD

recommend related documents :

STATSD Learning
STATSD and Graphite Joint operation
Web Monitoring Series Fourth: STATSD guide
On CENTOS6, the deployment process for STATSD and graphite

Vagrant

Vagrant is a tool for building and deploying a virtual development environment that is ideal for developing Web applications in Php/python/ruby/java languages, and can be distributed to team members through Vagrant encapsulation of a Linux development environment. Members can develop programs on their favorite desktop systems (Mac/windows/linux), but the code can be run uniformly in a packaged environment. It uses the VirtualBox virtualization system to create an automated virtual environment using chef. (Project details)

Code Escrow Address : https://github.com/mitchellh/vagrant

recommend related documents :

Vagrant Practice
Vagrant:up and Running
Build a cross-platform development environment with vagrant
Vagrant: Programmer's VirtualBox (i)

Ansible

Ansible is a model-driven Configuration manager that supports multi-node publishing and remote task execution. SSH is used by default for remote connections. There is no need to install additional software on managed nodes, which can be extended using a variety of programming languages. Ansible provides one of the simplest ways to publish, manage, and orchestrate a computer system, and you can do it in a few minutes. (Project details) code managed address : https://github.com/ansible/ansible

recommend related documents :

Development of TORNADO+ANSIBLE+TWISTED+MONGODB operation and maintenance Automation system (i) (ii) (iii)
Using the PPA in puppet/ansible
[email protected] An efficient configuration management tool (series articles)

Salt

Salt is a large distributed configuration Management system (Install upgrade uninstall software, detection environment) and also a remote command execution system. As a powerful remote execution manager, Salt is used for fast and efficient server management. More powerful than Func. Extension is more convenient. (Project details) code managed address : Https://github.com/saltstack/salt

recommend related documents :

Quick start of salt
Installation and configuration of salt
The grains of Salt translation
Salt Combat automatic Installation Deployment Moosefs
Use Salt + Hash to encrypt passwords

Graphite-web

Graphite-web is one of the Graphite components that provides a Django-capable, highly scalable, real-time drawing system. (Project details)

Code Escrow Address : Https://github.com/graphite-project/graphite-web

Official document : http://graphite.readthedocs.org/en/latest/

Fabric

Fabric is a Python (2.5 or higher) library and command-line tool for connecting to an SSH server and executing commands. (Project details)
Code Escrow Address : https://github.com/fabric/fabric

recommend related documents :

Python fabric for remote operation and deployment
MySQL native HA solution –fabric Experience Tour
MySQL fabric deployment uses fabric to deploy Hadoop and HBase
Python fabric for remote operation and deployment

This article is from the "Guo Weijun work Learning record" blog, so be sure to keep this source http://visonguo.blog.51cto.com/510379/1583052

Big Data System Toolset