Download the latest Android source from GitHub

At the end of May this year, Google was completely blocked by the wall, all Google's website can not visit, this time including the android.org,googlesource.com,code.google.com. Android's official resources are inaccessible, and it's certainly difficult to download the Android code.

This article is to solve this problem--to explain how to make download from Github.com/andorid andorid the latest source code. As the saying goes, "to give the fish as a fishing," this article is not just a package of the source code is a link, but to reproduce how I downloaded, I hope that the similar problems you encounter can help.

github.com/android Introduction

Open github.com/android to see:

A link to each item is listed in bold in the middle. The Android source hosted on Github.com/android is to manage each project separately with Git, instead of managing all projects like googlesource.com using repo (a Google-developed, git-based command-line tool). So, if you just want to download a project's source code, you can download it separately (such as the development of the app want to follow the SDK some of the source code can download Platform_frameworks_base project).

How to download the code for a project.

The classmate who used Git is sure, you can use the following command to get the code for the Remote warehouse:

The right column of the GitHub project page gives the warehouse URL, as shown below (the text box below the clone URL):

Of course, if you only want to configure the "app source debugging" Environment, you can click on the "Download zip" button, download the ZIP format source package, and then choose to download the good source package when you debug the eclipse.
How to bulk download the Android source on GitHub

Here is the focus of this article--Bulk download GitHub on the Android source.

Depending on the process of downloading a single item individually, there are two ways to download the source code for all projects: Get all the project's git repository URL to get all the items in the zip-package download link

The next question is how to get a URL like this?

Of course, from the page (that is, the HTML file on the page).

Now the question is how to get these pages.

The only way I could think of was--with curl.

Ps:curl is a command-line upload download tool that supports a variety of protocols including HTTP, HTTPS, FTP, and more.

Students without curl can be downloaded with the following command:

sudo apt-get Install Curl

With the curl used to capture the page, there is no worry about the warehouse URL or zip package URL:

First of all, to crawl Github.com/android page, and then "further" to crawl the various items of the page, and then from the items on the page to grab the warehouse URL or zip package URL.

Crawl Page

At the bottom of the Github.com/android page you can see github.com/andoid altogether 5 pages, click the following several pages, you can find that these several pages of the same URL form:

Https://github.com/android?page=N

where n means "page", like the second page, is https://github.com/android?page=2

As a result, you can use a bash script to easily write code that crawls the 5 pages (you can execute the following commands directly in the terminal):

for ((I=1; i<=5; i++)); Do
	curl https://github.com/android?page= $i > android.github.com. $i. html;
Done

there's a shortcut.

Looking at the project's first page URL and the warehouse URL can be found:

Warehouse url = home URL +. Git

For example, Platform_frameworks_base's homepage URL is:

Https://github.com/android/platform_frameworks_base

The Warehouse URL is:

Https://github.com/android/platform_frameworks_base.git

Of course, if you have a github.com account, and the current environment of the PubKey has been added to the profile, you can also use the SSH protocol to download (instead of https), ssh download URL is:

Git://github.com/android/platform_frameworks_base.git

So, getting the warehouse URL doesn't have to grab the project's home page any further, it's simpler (just get all the warehouse URLs below).

parsing pages

The question now is-how do I get the URL of the warehouse from these pages?

You can quickly find the HTML code block for the previous link in the page by using the browser's "Review element" menu, as shown in the following figure:

You can see the hyperlink for the Platform_frameworks_base item (<a href= "xxx" >) is surrounded by a

According to the two laws just found: The project first page URL always appears on the next line of the

Warehouse url = home URL +. Git

You can write Python code (getgitrepos.py) that resolves each project's home page (warehouse URL):

#!/usr/bin/python
Import sys

flag = False
link_prefix = ' https://github.com ' # If you are downloading using the SSH protocol, you can change HTTPS to git

alllines = Sys.stdin.readlines () for
curline in Alllines:
	if Curline.find (' repo-list-name ') >= 0:
		Flag = True
	if flag:
		pos = curline.find (' href= ')
		if POS >= 0: pos = =
			len (' href= ') Last
			= Curlin E[pos:] End
			= Last.find (' ")
			link = last[:end]
			
			# Name to path.
			Name = Link[link.rfind ('/') + 1:]
			prefix = name.find (' platform_ ')
			if prefix >= 0:
				name = Name[len (' Platform_ '):] # ignore platform_
			path = Name.replace (' _ ', '/')
			
			link = link_prefix + link
			# print ' Curl ', Li NK, ' > ', name
			# https://github.com/android/platform_external_qemu.git
			print ' git clone ', link + '. Git ', Path # output git clone command
			flag = False

The reason for parsing in Python is that Python is simple to write and can be used in other languages.

This Python program reads the text from the standard input and parses it, which can be implemented using the redirection operator, which replaces the standard input with several github.com/android of the previously crawled page.

The path parameter of the git clone command for the program output ignores the PLATFORM_ prefix in front of the warehouse name, which makes the final downloaded source structure similar to the Repo sync.

You can first test a page to see if the URL is resolved (the Python code above is correct), the terminal input the following command:

./getgitrepos.py < android.github.com.1.html

The output of this command is:

git clone https://github.com/android/platform_dalvik.git Dalvik git clone https://github.com/android/platform_ Bionic.git bionic git clone https://github.com/android/platform_system_core.git system/core git clone https:// Github.com/android/kernel_common.git Kernel/common git clone https://github.com/android/platform_external_qemu.git External/qemu git clone https://github.com/android/platform_build.git build git clone Https://github.com/android/ Platform_development.git development git clone https://github.com/android/platform_frameworks_base.git frameworks/ Base git clone https://github.com/android/platform_manifest.git manifest git clone Https://github.com/android/ Platform_frameworks_support.git frameworks/support git clone https://github.com/android/platform_packages_apps_ Settings.git packages/apps/settings git clone https://github.com/android/platform_external_dhcpcd.git external/ DHCPCD git clone https://github.com/android/platform_external_webkit.git external/webkit git clone https://github.com/android/platform_external_protobuf.git external/protobuf git clone https://github.com/android/platform _packages_providers_mediaprovider.git packages/providers/mediaprovider git clone Https://github.com/android/ Platform_external_elfutils.git external/elfutils git clone https://github.com/android/platform_external_strace.git External/strace git clone https://github.com/android/platform_hardware_libhardware.git hardware/libhardware git Clone https://github.com/android/platform_external_tinyxml.git external/tinyxml git clone https://github.com/ Android/platform_external_oprofile.git External/oprofile

The test is basically problematic because the getgitrepos.py output is a git clone command that can perform the actual download action after execution, so you need to save the output;

Use the following command to save the git clone command that is parsed for each subpage into a text file:

For page in ' LS android.github.com.* '; Do
	Echo Parse $page
	./getgitrepos.py < $page >> GitRepos.txt;
Done

Once you have the gitRepos.txt that fills the git clone command, you can do so by using the following command:

SH gitRepos.txt

One-click Download

In order to facilitate the use of the vast number of netizens, I have to use the above command to write the following clonerepos.sh:

#!/bin/bash
# clone Aosp each repo to https://github.com/android
# This shell depends on git, curl and Python 2.
# INSTALL depends:
#	sudo apt-get INSTALL git curl python
# by
Xu (xusiwei1236@163.com).

# download GitHub aosp sub pages for
((i=1; i<=5; i++)); does
	echo get AOSP subpage ge= $i
	Curl https://github.com/android?page= $i > android.github.com. $i. html;
Done

# parse git repo URLs from all sub pages
mkdir repos
cat/dev/null > GitRepos.txt for
page in ' Ls android.github.com.* '; Do
	Echo Parse $page
	./getgitrepos.py < $page >> GitRepos.txt;
Done

# clone repos
sh gitRepos.txt

In addition, I have packed the above clonerepos.sh and getgitrepos.py together, can be used to download http://download.csdn.net/detail/xusiwei1236/7960925.

Of course, if you feel yourself running trouble, you can directly copy the generated gitRepos.txt, paste directly into the terminal to execute, you can download all the Android source code:

git clone https://github.com/android/platform_dalvik.git dalvik
git clone https://github.com/android/platform_bionic.git bionic
git clone https://github.com/android/platform_system_core.git system/core
git clone https://github.com/android/kernel_common.git kernel/common
git clone https://github.com/android/platform_external_qemu.git external/qemu
git clone https://github.com/android/platform_build.git build
git clone https://github.com/android/platform_development.git development
git clone https://github.com/android/platform_frameworks_base.git frameworks/base
git clone https://github.com/android/platform_manifest.git manifest
git clone https://github.com/android/platform_frameworks_support.git frameworks/support
git clone https://github.com/android/platform_packages_apps_settings.git packages/apps/settings
git clone https://github.com/android/platform_external_dhcpcd.git external/dhcpcd
git clone https://github.com/android/platform_external_webkit.git external/webkit
git clone https://github.com/android/platform_external_protobuf.git external/protobuf
git clone https://github.com/android/platform_packages_providers_mediaprovider.git packages/providers/mediaprovider
git clone https://github.com/android/platform_external_elfutils.git external/elfutils
git clone https://github.com/android/platform_external_strace.git external/strace
git clone https://github.com/android/platform_hardware_libhardware.git hardware/libhardware
git clone https://github.com/android/platform_external_tinyxml.git external/tinyxml
git clone https://github.com/android/platform_external_oprofile.git external/oprofile
git clone https://github.com/android/platform_external_neven.git external/neven
git clone https://github.com/android/platform_external_netperf.git external/netperf
git clone https://github.com/android/platform_external_netcat.git external/netcat
git clone https://github.com/android/platform_external_libxml2.git external/libxml2
git clone https://github.com/android/platform_external_libpng.git external/libpng
git clone https://github.com/android/platform_external_libpcap.git external/libpcap
git clone https://github.com/android/platform_external_libffi.git external/libffi
git clone https://github.com/android/platform_external_jpeg.git external/jpeg
git clone https://github.com/android/platform_external_jhead.git external/jhead
git clone https://github.com/android/platform_external_jdiff.git external/jdiff
git clone https://github.com/android/platform_external_iptables.git external/iptables
git clone https://github.com/android/platform_external_icu4c.git external/icu4c
git clone https://github.com/android/platform_external_giflib.git external/giflib
git clone https://github.com/android/platform_external_freetype.git external/freetype
git clone https://github.com/android/platform_external_fdlibm.git external/fdlibm
git clone https://github.com/android/platform_external_expat.git external/expat
git clone https://github.com/android/platform_external_esd.git external/esd
git clone https://github.com/android/platform_external_emma.git external/emma
git clone https://github.com/android/platform_external_dropbear.git external/dropbear
git clone https://github.com/android/platform_external_apache-http.git external/apache-http
git clone https://github.com/android/platform_external_openssl.git external/openssl
git clone https://github.com/android/platform_external_sonivox.git external/sonivox
git clone https://github.com/android/platform_packages_providers_telephonyprovider.git packages/providers/telephonyprovider
git clone https://github.com/android/platform_packages_providers_downloadprovider.git packages/providers/downloadprovider
git clone https://github.com/android/platform_packages_providers_contactsprovider.git packages/providers/contactsprovider
git clone https://github.com/android/platform_packages_providers_calendarprovider.git packages/providers/calendarprovider
git clone https://github.com/android/platform_packages_apps_voicedialer.git packages/apps/voicedialer
git clone https://github.com/android/platform_packages_apps_stk.git packages/apps/stk
git clone https://github.com/android/platform_packages_apps_soundrecorder.git packages/apps/soundrecorder
git clone https://github.com/android/platform_packages_apps_phone.git packages/apps/phone
git clone https://github.com/android/platform_packages_apps_packageinstaller.git packages/apps/packageinstaller
git clone https://github.com/android/platform_packages_apps_music.git packages/apps/music
git clone https://github.com/android/platform_packages_apps_mms.git packages/apps/mms
git clone https://github.com/android/platform_packages_apps_htmlviewer.git packages/apps/htmlviewer
git clone https://github.com/android/platform_packages_apps_email.git packages/apps/email
git clone https://github.com/android/platform_packages_apps_contacts.git packages/apps/contacts
git clone https://github.com/android/platform_packages_apps_camera.git packages/apps/camera
git clone https://github.com/android/platform_packages_apps_calendar.git packages/apps/calendar
git clone https://github.com/android/platform_packages_apps_calculator.git packages/apps/calculator
git clone https://github.com/android/platform_packages_apps_browser.git packages/apps/browser
git clone https://github.com/android/platform_hardware_ril.git hardware/ril
git clone https://github.com/android/platform_external_zlib.git external/zlib
git clone https://github.com/android/platform_external_yaffs2.git external/yaffs2
git clone https://github.com/android/platform_external_tcpdump.git external/tcpdump
git clone https://github.com/android/platform_external_tagsoup.git external/tagsoup
git clone https://github.com/android/platform_external_srec.git external/srec
git clone https://github.com/android/platform_external_sqlite.git external/sqlite
git clone https://github.com/android/platform_external_skia.git external/skia
git clone https://github.com/android/platform_external_safe-iop.git external/safe-iop
git clone https://github.com/android/platform_external_ppp.git external/ppp
git clone https://github.com/android/kernel_msm.git kernel/msm
git clone https://github.com/android/platform_system_bluetooth.git system/bluetooth
git clone https://github.com/android/platform_prebuilt.git prebuilt
git clone https://github.com/android/platform_packages_providers_drmprovider.git packages/providers/drmprovider
git clone https://github.com/android/platform_external_ping.git external/ping
git clone https://github.com/android/platform_external_dbus.git external/dbus
git clone https://github.com/android/tools_repo.git tools/repo
git clone https://github.com/android/android.github.io.git android.github.io
git clone https://github.com/android/platform_external_wpa_supplicant.git external/wpa/supplicant
git clone https://github.com/android/platform_packages_apps_googlesearch.git packages/apps/googlesearch
git clone https://github.com/android/platform_system_wlan_ti.git system/wlan/ti
git clone https://github.com/android/platform_packages_providers_googlecontactsprovider.git packages/providers/googlecontactsprovider
git clone https://github.com/android/platform_packages_apps_im.git packages/apps/im
git clone https://github.com/android/platform_packages_providers_improvider.git packages/providers/improvider
git clone https://github.com/android/platform_external_tremor.git external/tremor
git clone https://github.com/android/platform_packages_apps_sync.git packages/apps/sync
git clone https://github.com/android/platform_packages_apps_alarmclock.git packages/apps/alarmclock
git clone https://github.com/android/platform_external_aes.git external/aes
git clone https://github.com/android/platform_external_clearsilver.git external/clearsilver
git clone https://github.com/android/platform_external_googleclient.git external/googleclient
git clone https://github.com/android/platform_packages_apps_launcher.git packages/apps/launcher
git clone https://github.com/android/platform_external_bluez.git external/bluez
git clone https://github.com/android/platform_external_gdata.git external/gdata
git clone https://github.com/android/platform_frameworks_opt_com.google.android.git frameworks/opt/com.google.android
git clone https://github.com/android/platform_external_elfcopy.git external/elfcopy
git clone https://github.com/android/platform_frameworks_policies_base.git frameworks/policies/base
git clone https://github.com/android/platform_packages_apps_updater.git packages/apps/updater
git clone https://github.com/android/platform_external_opencore.git external/opencore