GitLab construction and maintenance (based on Docker image sameersbn/docker-gitlab)

GitLab construction and maintenance (based on Docker image sameersbn/docker-gitlab)
1. Read the basics of this Article

• Familiar with git
• Familiar with docker

2. GitLab introduction 2.1. Overview

• GitLab is an open-source project used for warehouse management systems. A web service built on Git as a code management tool. Github is a public git repository, and Gitlab is suitable for building private git repositories within an enterprise.
• Official Website:

  https://about.gitlab.com/https://github.com/gitlabhq/gitlabhq

• The creation time of this article. The latest GitLab version is v7.4.3.

2.2. Architecture

• :

2.3. Components

• Front-end: Nginx, used for http or https protocols for pages and Git tools
• Backend: Gitlab service, which adopts the Ruby on Rails framework and uses unicorn to implement background services and multi-process
• SSHD: enables the sshd service for users to upload ssh keys for version cloning and uploading. Note: The ssh key uploaded by the user is saved to the git account.
• Database: currently, only MySQL and PostgreSQL are supported.
• Redis: used to store user sessions and tasks, including creating a warehouse and sending emails.
• Sidekiq: a built-in Rails framework that subscribes to and executes redis tasks.

3. GitLab installation and deployment 3.1. Official support

• RPM package containing everything: https://about.gitlab.com/downloads/ (officially recommended)

  I personally don't recommend it, because the RPM package capacity is too large, more than 200 M, suitable for Linux beginners, not tried

• Manual installation: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md (learn more)

  You can have a better understanding of the architecture between components of GitLab, but it is not suitable for getting started.

• Third-party docker image: https://github.com/sameersbn/docker-gitlab (recommended by the author)

  Images can be deployed and used quickly. They are suitable for those who are familiar with Docker and get started quickly. In addition, the use of Docker images frees you from having to learn too much about internal details. You can configure GitLab parameters by starting the container with environment variables.

3.2. Install GitLab3.2.1. Introduction

• Official Website: https://github.com/sameersbn/docker-gitlab
• In versions earlier than 7.4.3, the image contains all components. In 7.4.3, the image only contains core components: nginx, sshd, ruby on rails, and sidekiq.

3.2.2. Architecture Diagram

3.2.3. Download the image

• Docker pull sameersbn/gitlab: 7.4.3 # download the gitlab Image
• Docker pull sameersbn/mysql: latest # download the mysql image used by gitlab
• Docker pull sameersbn/redis: latest # download the redis image used by gitlab

3.2.4. Install 3.2.4.1. Start redis

• Command:

  docker run \--name=gitlab_redis \-tid \sameersbn/redis:latest

3.2.4.2. Start mysql

• Mkdir-p/opt/gitlab/mysql
• Command:

  docker run \--name=gitlab_mysql \-tid \-e 'DB_NAME=gitlabhq_production' \-e 'DB_USER=gitlab' \-e 'DB_PASS=password' \-v /opt/gitlab/mysql:/var/lib/mysql \sameersbn/mysql:latest

3.2.4.3. Start gitlab

• Mkdir-p/opt/gitlab/data/opt/gitlab/log
• Command:

  Docker run \ -- name = 'gitlab '\-itd \ -- link gitlab_mysql: mysql \ -- link gitlab_redis: redisio \-e 'gitlab _ PORT = 80' \-e 'gitlab _ SSH_PORT = 22' \-e' GITLAB _ HOST = gitlab.example.com '\-v/var/run/docker. sock:/run/docker. sock \-v $ (which docker):/bin/docker \-v/opt/gitlab/data:/home/git/data \-v/opt/gitlab/log: /var/log/gitlab \ sameersbn/gitlab: 7.4.3 The above is a basic gitlab. Complete (including LDAP and EMAIL): docker run \ -- name = 'gitlab '\-itd \ -- link gitlab_mysql: mysql \ -- link gitlab_redis: redisio \-e 'gitlab _ PORT = 80' \-e 'gitlab _ SSH_PORT = 22' \-e 'ldap _ ENABLED = true' \-e' LDAP _ HOST = 192.168.1.1 '\-e' LDAP _ PORT = 8080' \-e' LDAP _ UID = samaccountname' \-e' LDAP _ METHOD = plain' \-e' LDAP _ BIND_DN = test@example.com '\-e' LDAP _ PASS = passwd' \-e' LDAP _ BASE = OU = example_users, DC = example-family, DC = com '\-e 'ldap _ ACTIVE_DIRECTORY = true' \-e 'ldap _ ALLOW_USERNAME_OR_EMAIL_LOGIN = false' \-e 'gitlab _ HOST = gitlab.example.com' \-e' SMTP _ ENABLED = true' \-e' SMTP _ DOMAIN = example.com '\-e 'smtp _ HOST = 192.168.1.2' \-e' SMTP _ PORT = 25' \-e' SMTP _ STARTTLS = false' \-v/var/run/docker. sock:/run/docker. sock \-v $ (which docker):/bin/docker \-v/opt/gitlab/data:/home/git/data \-v/opt/gitlab/log: /var/log/gitlab \ sameersbn/gitlab: 7.4.3

 

This step takes several minutes, because it will perform initialization operations, such as importing the data table structure. You can view the installation process through docker logs gitlab. Similarly, mysql and redis containers can also view startup information through docker logs gitlab_mysql and docker logs gitlab_redis.

 

Note: The three containers created above must be located on the same host, because-link gitlab_mysql: mysql-link gitlab_redis: redisio is used to link the containers on the same host.

• Of course, redis and mysql also support the use of ip + port, without-link, with environment variables, the method is:

  Complete: docker run \ -- name = 'gitlab '\-itd \ -- net = none \ -- hostname = 'gitlab .example.com' \-e 'db _ TYPE = mysql' \-e 'db _ HOST = 192.168.3.1 '\-e 'db _ PORT = 100' \-e 'db _ NAME = gitlabhq_production' \-e 'db _ USER = gitlab '\-e' DB _ PASS = passwd' \-e 'redis _ HOST = 192.168.3.2 '\-e' REDIS _ PORT = 8080' \-e' UNICORN _ WORKERS = 20' \-e' GITLAB _ PORT = 80' \-e 'gitlab _ SSH_PORT = 22' \-e 'ldap _ ENABLED = true' \-e 'ldap _ HOST = 192.168.3.3 '\-e' LDAP _ PORT = 389 '\-e 'ldap _ UID = samaccountname' \-e 'ldap _ METHOD = plain' \-e 'ldap _ BIND_DN = test@example.com '\-e' LDAP _ PASS = passwd' \-e 'ldap _ BASE = OU = example_users, DC = example-family, DC = com '\-e 'ldap _ ACTIVE_DIRECTORY = true' \-e 'ldap _ ALLOW_USERNAME_OR_EMAIL_LOGIN = false' \-e 'gitlab _ HOST = gitlab.example.com' \-e' SMTP _ ENABLED = true' \-e' SMTP _ DOMAIN = example.com '\-e 'smtp _ HOST = 192.168.3.4' \-e' SMTP _ PORT = 25' \-e' SMTP _ STARTTLS = false' \-v/var/run/docker. sock:/run/docker. sock \-v $ (which docker):/bin/docker \-v/opt/gitlab/data:/home/git/data \-v/opt/gitlab/log: /var/log/gitlab \ sameersbn/gitlab: 7.4.3sameersbn/gitlab: 7.4.3 redis does not support any verification and can only be used without a password

 

Currently, sameersbn/gitlab: 7.4.3 has a very bad location: the container may not be able to connect to the database at startup because the container will detect whether the database is available at startup, the test method is to connect mysqladmin to port 3306 of DB_HOST, instead of the port specified by DB_PORT. The issue has been submitted to the author.

3.2.4.4. Configure the IP address for the gitlab container

• Pipework br1 gitlab 192.168.1.1/24@192.168.1.254

 

When the gitlab container starts (docker run), you can add the-p parameter to map the port in the container to the host. However, I prefer to configure an independent IP address for the container, therefore, the preceding command does not use-p for port ing, however,-e 'gitlab _ PORT = 80'-e' GITLAB _ SSH_PORT = 22'-e' GITLAB _ HOST = gitlab.example.com 'still needs to be specified, otherwise GITLAB cannot be used

3.2.4.5. The installation is complete. You can open the page

• Url: gitlab.example.com

  Account: root Password: 5 iveL! Fe

• Effect:

3.2.4.6. Add to start

• Add/etc/rc. local

  echo 'docker start gitlab_redis' >> /etc/rc.localecho 'docker start gitlab_mysql' >> /etc/rc.localecho 'docker start gitlab' >> /etc/rc.localecho 'pipework br1 gitlab 192.168.1.1/24@192.168.1.254' >> /etc/rc.local

4. GitLab API wrappers

• Official recommendation: https://about.gitlab.com/applications/
• Python: pyapi-gitlab

  Basic usage: pip install pyapi-gitlabimport gitlabgit = gitlab. gitlab ("http://gitlab.example.com", token = "EHBLkwhr_WYzn-sXNnNs") # token shows automatically generated Private tokengit in Profile settings-> Account on the page. getusers ()

• By the way, we recommend a python api module for Git itself: https://github.com/FriendCode/gittle

 

The implementation of many other git libraries is too low-level. It is totally different from the linux git commands and is not friendly at all. However, I found that this module only supports SSH and does not support HTTP. Therefore, I prefer linux git commands.

5. The FAQ5.1. git Code released by GitLab supports two methods: ssh and http (s)

• Ssh: you must add an ssh key before publishing.
• Http: the user name and password are used. If LDAP is connected, the account password in LDAP is used.
• Https: not tested yet

5.2. restrict the size of a single File Uploaded By git

• If gitlab is created through the sameersbn/gitlab: 7.4.3 image, you can use the environment variable NGINX_MAX_UPLOAD_SIZE to restrict

5.3. Perform some operations through the API, such as creating a Project, to get a successful response, but it does not take effect immediately. It takes several seconds to take effect.

• Due to GitLab's asynchronous architecture, Ruby on Rails pushed the task to Redis after receiving the Project Creation request, but did not wait for the execution to complete, but directly returned success. The background Sidekiq subscribes to the task from Redis and executes it in real time. However, because the execution takes time and the API returns fast, this may happen. Currently, there is no other solution. You can only retry the program by sleep for several seconds in your program logic.

5.4. Differences between "Profile settings-> SSH Keys" and "Deploy Keys" in "Project Settings" on the page

• Profile settings-> SSH Keys: The global Key of the user. It has the permission to upload and download (push and clone) All project repositories of the user.
• Deploy Keys in Project Settings: For a Project, it has the permission to download (clone), but not to upload (push ).

5.5. The default time zone of gitlab is UTC.

• Sameersbn/gitlab: 7.4.3 The image cannot modify the time zone. If you have manually installed gitlab, you can modify one. rb file to take effect, specific has not been tested, if necessary, you can search Baidu.

 

The time zone is UTC, which mainly affects the value of the time field in the database (created_at, updated_at, etc.), and does not affect the git repository, because the git repository is created and code updated by the user through the git tool, it is the user's own time zone.

6. GitLab maintenance 6.1. data path and Log Path

• The gitlab container has mounted the/opt/gitlab/data and/opt/gitlab/log directories of the host to the container at startup, therefore, you can go to these two directories on the host and view them through docker logs.

6.2. High Availability

• There is an official article on GitLab High Availability: https://about.gitlab.com/high-availability/

  1/3 of users are talking about the advantages and disadvantages of high availability, the benefits of high availability to varying degrees, and more problems that may be caused. Since I have just started using GitLab, I will not test the multi-host load balance or failover.

• Two Backup Recovery Methods

  1. Backup configuration, warehouse, and database. 2. Create a snapshot for the file system. Officially, the second method is faster than the first one, because it can save the trouble of manual intervention in restore. XFS file systems support snapshots in two articles: http://www.icicletech.com/blog/gitlab-backup-made-easy database data backup to local http://doc.gitlab.com/ce/raketasks/backup_restore.html official, and the above is actually the same

• My backup solution:

  I. Introduction: Back up data at regular intervals every day (only important files in the MySQL and GitLab repositories are backed up, and others are not backed up ). If data is lost and needs to be restored, the latest backup data is used to restore all data, including local files (warehouses and other files) and databases, to the backup time. II. specific Operation: 1. backup: docker run \ -- name = 'gitlab _ backup' \-it \ -- rm \ -- link gitlab_mysql: mysql \ -- link gitlab_redis: redisio \-v/var/run/docker. sock:/run/docker. sock \-v $ (which docker):/bin/docker \-v/opt/gitlab/data:/home/git/data \-v/opt/gitlab/log: /var/log/gitlab \ sameersbn/gitlab: 7.4.3 app: rake gitlab: backup: The create process can be viewed on the screen, the backup is automatically packed into a tar package in/opt/gitlab/data/backups. You can compress the tar package by yourself, for example, gzip time stamp _gitlab_backup.tar. If $? Then, the generated file is compressed to .tar.gz using gzip. then, rsync is pushed to the storage and uploaded to a backup storage using the rsync command. recovery: docker run \ -- name = 'gitlab _ restore' \-it \ -- rm \ -- link gitlab_mysql: mysql \ -- link gitlab_redis: redisio \-v/var/run/docker. sock:/run/docker. sock \-v $ (which docker):/bin/docker \-v/opt/gitlab/data:/home/git/data \-v/opt/gitlab/log: /var/log/gitlab \ sameersbn/gitlab: 7.4.3 app: rake gitlab: backup: restore the/opt/gitlab/data/backup All files and directories in s are listed. copy and paste the file name of the tar package to be restored, and press enter to start recovery. Note: during recovery, all tables in the current database will be deleted first and then imported into the SQL file in the backup tar package. Therefore, be careful in this step. If redis and mysql are imported into the gitlab container using environment variables, the backup and recovery commands are similar. Copy the command to start gitlab and modify -- name to add a -- rm, change CMD to gitlab: backup: create or gitlab: backup: restore. note: 1. the user-configured key (that is, data /. ssh) is saved in the database. When the gitlab service is started, it will be loaded from the database and imported. in ssh, managed by gitlab-shell 2. because the ssh directory (which saves the ssh_key generated by the server when the sshd service is started) is not backed up, if there is a file in the directory during restoration, ssh_key will not be regenerated. If it does not exist, then generate a new one. If you use the ssh protocol for git push again, an error is reported to indicate that the authentication is invalid. backup only backs up three directories: repositories, db, and uploads, and then generates an additional backup_information.yml file 4. gitlab-satellites and tmp are both temporary directories, so they are not involved in backup 5. docker run -- rm does not affect the output of container exit status 6. during restoration, the tables in the database are first dropped and then created, because the SQL file will DROP table if exists 7 before INSERT. if the directory to be restored exists during restoration, the original directory will be mv. old. timestamp

For more GitLab tutorials, see the following:

GitLab Installation Guide in Ubuntu 14.04

How to install Gitlab in Ubuntu Server 14.04

Install GitLab in Chinese with CentOS source code

Install GitLab on Ubuntu 12.04

GitLab 5.3 upgrade considerations

Deploy GitLab on CentOS (self-managed Git project repository)

Install GitLab 6.0.2 on RHEL6/CentOS6/ScientificLinux6

CentOS 6.5 GitLab installation tutorial and Related Problems

GitLab details: click here
GitLab: click here

This article permanently updates the link address: