AOSP Creating and managing your own repo(repository) mirror

Hello Guys!!!
Hope you all are doing well !!!!

Toady I am going to discuss a very important topic in android system AOSP development.
As you people know that for Android System development, we all need to download AOSP source code and maintain this source code for our development. 

It is very common to development over the time that we need different different implementation/modification in our AOSP source code. As we all know that downloading the new AOSP source code is not easy task. It is time consuming and huge file(more than 30 gb).
So, We can save a lot of time to create and manage a local repo mirror instead of downloading source code from remote repositories for all configuration changes /or modification.
  Now I am going to show you it step by step approach 

1. Repo and manifest
To create and manage repository mirrors, First we have to understand the :- 

• repo command and 
• the directory structure managed by repo

 The repo command deals with a XML file manifest and it stores everything in a folder called .repo. When we run the repo init command a .repo folder is created under the current folder. we can see the following contents inside .repo folder:-
$ ls -F .repo 

1. manifests/   This is a working copy of the Git repository of the manifest itself.
2. manifests.git/  This is the Git repository of the manifest. The manifest itself isunder the version control using Git.
3. manifest.xml@  This is a symbolic link to thefile .repo/manifests/default.xml. This file is the main configuration file
   used by repo. I shall discuss it soon.
4. repo/  The repo tool itself is written in the Python language. Python scripts arestored in this folder

After running the repo init command to initialize the repo data structure, we run the repo sync command to retrieve/pull a working copy. Now look at the .repo folder again we can see that there are three project related folders created by the repo sync command.

1. project.list: This is a list of all projects downloaded.
2. project-objects: This is a copy of the remote repository.
3. projects: This is the repository hierarchy matching the working copy. The contents in this folder are symbolic links to the items in project-objects

The most important file in the .repo folder is .repo/manifests/default.xml or its symbolic link manifest.xml. You can find detailed specification of this file in the document under the .repo folder at .repo/repo/docs/manifest-format.txt. 

I am not going into any details, but let's take look at the most commonly used elements.

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<remote name="aosp"
fetch=".." />
<default revision="refs/tags/android-7.1.1_r4"
remote="aosp"
sync-j="4" />
<project path="build" name="platform/build" groups="pdk" >
<copyfile src="core/root.mk" dest="Makefile" />
</project>
<project path="abi/cpp" name="platform/abi/cpp" groups="pdk" />
<project path="art" name="platform/art" groups="pdk" />
<project path="bionic" name="platform/bionic" groups="pdk" />
...</manifest>

Explanation of above mention xml tag:- 

• remote: The remote element provides the details about remote repository. We can give it a name such as aosp. Fetch field provide URL of the remote repository. It can be a relative path or a full path. 
• default: There are multiple remote elements that can be specified in manifest. The default element defines which remote is the default one.
•  project: Each project element defines a Git repository. The path field supplies the local path after it is downloaded. The name field supplies the remote path of the Git repository. The revision field supplies the branch that we want to get and the remote field tells us which remote server we use to get the Git repository. 

There are other XML elements that can be used in manifest as well. You can check more detail by looking at the preceding specification yourself in documents.

Using a local mirror for AOSP
If you guys refer to the article from the Google website about downloading the source code for AOSp, you can find a section called Using a local mirror. It reveals that if you need two different configurations of the AOSP build environment. It is very simple to set up a mirror as follows:

$ mkdir -p /usr/local/mirror/aosp
$ cd /usr/local/mirror/aosp
$ repo init -u https://android.googlesource.com/mirror/manifest --mirror
$ repo sync

we can see that we actually use a different manifest to create a mirror. If we look at the content of the manifest for a mirror, we can see the following XML code:

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<remote name="aosp"
fetch=".." />
<default revision="master"
remote="aosp"
sync-j="4" />
<project name="accessories/manifest" />
<project name="brillo/manifest" />
...</manifest>

We can see that in above xml file, for all projects, there are only the project names without other information in each project item. This is because we actually copy each Git repository to the local as a bare Git repository. We won't check out a working copy, so we don't need to worry about the version.
If we look at the manifest to check out a working copy, we will see the following:

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<remote name="aosp"
fetch=".." />
<default revision="refs/tags/android-6.0.1_r61"
remote="aosp"
sync-j="4" />
<project path="build" name="platform/build" groups="pdk" >
<copyfile src="core/root.mk" dest="Makefile" />
</project>
<project path="abi/cpp" name="platform/abi/cpp" groups="pdk" />
<project path="art" name="platform/art" groups="pdk" />
...</manifest>

It includes more items than the one to create a mirror. The name field specifies the path at the remote repository and the path field specifies the local path after the repository is downloaded to the local. We also need to specify revision that we want to retrieve. After we have a mirror, we can check out a copy of the AOSP source from that mirror as follows:
$ mkdir -p $HOME/aosp/master
$ cd $HOME/aosp/master
$ repo init -u /usr/local/mirror/aosp/platform/manifest.git
$ repo sync
If you need, you can check out multiple copies from the local mirror. No matter if you check out multiple copies or you change to a different version, you can save a lot of time compared to checking out from a remote repository.

This is how we can download AOSP and mange a local mirror for our need. In next blog, I shall discus following repo mirror topic:-

• Creating your own mirror of GitHub
• Fetching Git repositories outside GitHub
• Creating your own manifest for client download

Happy Coding!!!