Merge multiple gradle projects including git history

As we started 2 years ago with a micro-service architecture we faced some issues in the productivity of the teams a while ago. This made me think what we could do about that from an architecture and development point of view. As we used different Git repos for the various components of our micro services in the long run it became harder and harder for developers to work with those components. At that point in time I decided to simplify the development workflow to merge the different Git repos of one functional bounded context into one Git repository.

Finally I found some time to document this process in the form of a blogpost for my own reference, but also to share this with the community. In case you are thinking, I'm not using Gradle and are at the point to quite reading, please don't! Only last bit goes Gradle specific. The first 75% you can reuse for whatever kind of project you are running.

To keep impact for development team low to non-existing I wrote a migration script in bash to be able to do the migration within a couple of minutes and do some dry runs to verify if everything was still working. By everything I mean the following parts:

Do the projects still compile?
Do the unit and integration test still succeed?
Does the continuous integration still work on Jenkins?
Does the continuous deployment still work on Kubernetes?

Lets start with a folder structure example which represents an similar situation we faced. In every folder we had a Git repository containing the Gradle project including a Jenkinsfile and some other common files for CI and CD purposes.

terminal

$ tree code-folder
code-folder
├── domain-a-api
├── domain-a-commands
├── domain-a-consumer
├── domain-a-events
├── domain-b-api
├── domain-b-commands
├── domain-b-consumer
└── domain-b-events
 
8 directories, 0 files

I'll zoom in for the remainder of this article on domain-a. The goal is to end up with a structure like the following.

terminal

$ tree -a code-folder/domain-a
code-folder/domain-a
├── .git
├── Jenkinsfile
├── api
│   └── src
├── build.gradle
├── commands
│   └── src
├── consumer
│   └── src
└── events
    └── src
 
9 directories, 2 files

The first step is to move all files in the existing repositories into a subfolder. For that I wrote a bash function to be reused in all the projects I had to migrate.

function move_into_subfolder () {
    local subfolder=$1
    if [ ! -z "$2" ] ; then
        subfolder=$2
    fi
 
    pushd $1
    mkdir $subfolder
    git mv `ls -1 | grep -v $subfolder` $subfolder
    git mv .gitignore $subfolder
    git commit -a -m "Moving files into subfolder $subfolder"
 
    # navigate back to previous folder
    popd
}

This function can be used as following, move_into_subfolder "domain-a-api" "api", which will result in all files within the folder domain-a-api to be moved into domain-a-api/api. The move of files will also be committed in the existing repository.

As I was doing this for many repos and I had to make sure I was merging the latest develop branches of these repositories I also wrote a small bash function to clone or pull the latest develop branches from the repositories I was about to merge.

function clone_or_pull_develop() {
    if [ ! -d "${1}" ] ; then
        git clone -b develop git@github.com:marcofranssen/${1}.git
    else
        pushd ${1}
        git reset HEAD~1 --hard
        git pull
        popd
    fi
}

This function clones get's the latest changes from the develop branch and removes the last move-into-subfolder commit so you can run the function over and over again for testing if all comes allong nicely.

Now we have a way to have all the repositories prepared to be merged as one repository without loosing the commit history. Also for this action I wrote a small bash function to be easily reused on the various repositories I wanted to merge.

function merge_old_repo_commits () {
    git remote add $1 ../$1
    git fetch $1
    git merge --no-edit --allow-unrelated-histories $1/develop
    git remote rm $1
}

To start we will have to prepare a new git repository first and then I can call my new bash function to merge the repositories in one.

rm -rf domain-a #So we can easily restart from scratch
mkdir -p domain-a
cd domain-a
 
git init
git remote add origin git@github.com:marcofranssen/domain-a.git
 
merge_old_repo_commits "commands"
merge_old_repo_commits "events"
merge_old_repo_commits "consumer"
merge_old_repo_commits "api"

The result is we got all the 4 separate repository commit histories merged together in this single repository. Last but not least we will do a few additional commits on top of the merged repositories to make it a fully working multi-module gradle repository. This is probably where you would to go your own route to do your project specific modifications. As a reference I will show you below what I did.

First off all I made a new commit adding a new README.md file in the root of the project which links to the README.md files in my subprojects.

touch README.md
echo '# Domain A' > README.md
echo '' >> README.md
echo '## Api' >> README.md
echo '' >> README.md
echo '[README.md](api/README.md)' >> README.md
echo '## Consumer' >> README.md
echo '' >> README.md
echo '[README.md](consumer/README.md)' >> README.md
echo '## Commands' >> README.md
echo '' >> README.md
echo '[README.md](commands/README.md)' >> README.md
echo '## Events' >> README.md
echo '' >> README.md
echo '[README.md](events/README.md)' >> README.md
git add README.md
git commit -m "Add readme to root of project, linking to the module README.md files"

Then I made a commit where I add a new .gitignore file which I manualy merged and remove the original .gitignore files.

cp ../.gitignore .
git add .gitignore
git rm api/.gitignore consumer/.gitignore events/.gitignore commands/.gitignore
git commit -m "Add .gitignore to root of project and remove .gitignore files from subfolders"

Same I did for my Gradle files so I have one file to manage the Gradle build from the root of the new project folder.

gradle init
cp ../gradle* ../settings.gradle .
git add .
git update-index --chmod=+x gradlew
git commit -m "Initialized gradle project"
 
cp ../build.gradle .
echo "include 'api', 'consumer', 'events', 'commands'" >> settings.gradle
git add build.gradle settings.gradle
git rm -r api/build.gradle api/settings.gradle api/gradle*
git rm -r consumer/build.gradle consumer/settings.gradle consumer/gradle*
git rm -r events/build.gradle events/settings.gradle events/gradle*
git rm -r commands/build.gradle commands/settings.gradle commands/gradle*
 
git commit -m "Add new gradle configuration and remove the old ones"

Last but not least I also put in place new docker-compose setup and remove the old files. And put in place my updated Jenkinsfile which I also merged manually.

cp ../docker-compose.yaml .
git add docker-compose.yaml
git rm api/docker-compose.yml consumer/docker-compose.yml
git commit -m "Add docker-compose file in the root of project replacing the specific module ones"
 
cp ../ApiDockerfile api/Dockerfile
cp ../ConsumerDockerfile consumer/Dockerfile
git add api/ consumer/
git commit -m "Updated Dockerfiles"
 
# Put in place new Jenkins config
cp ../Jenkinsfile .
git add Jenkinsfile
git rm api/Jenkinsfile
git rm consumer/Jenkinsfile
git rm events/Jenkinsfile
git rm commands/Jenkinsfile
git commit -m "Add new Jenkinsfile and remove the old ones"

As you can imagine I had to run my script of few times from scratch to fix the bugs and mistakes. So in the end I tested all by running the gradle build and pushing it to my repo to trigger the Jenkins build.

./gradlew build

Once all succeeded I planned with the impacted team a 10 minute code freeze (grabbed some coffee) and then we continued our work on the new repository. Once the first one finished the other ones where more easy as most our projects use a similar setup.

TL;DR

Here the whole summary in one script. Including my folder layout from where I executed this script.

terminal

$ tree -a my-merge-folder
my-merge-folder
├── .gitignore
├── ApiDockerfile
├── ConsumerDockerfile
├── Jenkinsfile
├── build.gradle
├── docker-compose.yml
├── gradle.proeprties
├── merge-repos.sh
└── settings.gradle
 
0 directories, 9 files

I omit all my project specific files that I merged manually, but will provide you with the full script below. It is up to you to put all together.

merge-repos.sh

#!/bin/bash
 
function move_into_subfolder () {
    local subfolder=$1
    if [ ! -z "$2" ] ; then
        subfolder=$2
    fi
 
    pushd domain-a-$1
    mkdir $subfolder
    # git mv !($1) $1 # this bashism doesn't seem to work on windows git and is replaced by following 2 lines
    git mv `ls -1 | grep -v $subfolder` $subfolder
    git mv .gitignore $subfolder
    git commit -a -m "Moving files into subfolder $subfolder"
 
    # navigate back to previous folder
    popd
}
 
function merge_old_repo_commits () {
    git remote add $1 ../$1
    git fetch $1
    git merge --no-edit --allow-unrelated-histories $1/develop
    git remote rm $1
}
 
function clone_or_pull_develop() {
    if [ ! -d "${1}" ] ; then
        git clone -b develop git@github.com:marcofranssen/${1}.git
    else
        pushd ${1}
        git reset HEAD~1 --hard
        git pull
        popd
    fi
}
 
# Pull the latest develop branches and move the files in a subfolder as preparation for the merge
clone_or_pull_develop "domain-a-api"
move_into_subfolder "domain-a-api" "api"
clone_or_pull_develop "domain-a-commands"
move_into_subfolder "domain-a-commands" "commands"
clone_or_pull_develop "domain-a-consumer"
move_into_subfolder "domain-a-consumer" "consumer"
clone_or_pull_develop "domain-a-events"
move_into_subfolder "domain-a-events" "events"
 
# Prepare a new git repo to execute the merge
rm -rf domain-a
mkdir -p domain-a
cd domain-a
 
git init
git remote add origin git@github.com:marcofranssen/domain-a.git
 
merge_old_repo_commits "commands"
merge_old_repo_commits "events"
merge_old_repo_commits "consumer"
merge_old_repo_commits "api"
 
#######################################################
# Below you would like to customize to your own needs #
#######################################################
 
# Add a new README.md to the root of the project.
 
touch README.md
echo '# Domain A' > README.md
echo '' >> README.md
echo '## Api' >> README.md
echo '' >> README.md
echo '[README.md](api/README.md)' >> README.md
echo '## Consumer' >> README.md
echo '' >> README.md
echo '[README.md](consumer/README.md)' >> README.md
echo '## Commands' >> README.md
echo '' >> README.md
echo '[README.md](commands/README.md)' >> README.md
echo '## Events' >> README.md
echo '' >> README.md
echo '[README.md](events/README.md)' >> README.md
git add README.md
git commit -m "Add readme to root of project, linking to the module README.md files"
 
# Put in place the manually merged .gitignore file and remove the old ones
 
cp ../.gitignore .
git add .gitignore
git rm api/.gitignore consumer/.gitignore event/.gitignore command/.gitignore
git commit -m "Add .gitignore to root of project and remove .gitignore files from subfolders"
 
# Put in place the manually merged gradle file and remove the old ones
gradle init
cp ../gradle* ../settings.gradle .
git add .
git update-index --chmod=+x gradlew
git commit -m "Initialize gradle project"
 
cp ../build.gradle .
echo "include 'api', 'consumer', 'events', 'commands'" >> settings.gradle
git add build.gradle settings.gradle
git rm -r api/build.gradle api/settings.gradle api/gradle*
git rm -r consumer/build.gradle consumer/settings.gradle consumer/gradle*
git rm -r events/build.gradle events/settings.gradle events/gradle*
git rm -r commands/build.gradle commands/settings.gradle commands/gradle*
 
git commit -m "Add new gradle configuration and remove the old ones"
 
# Add a new enhanced docker composer for better dev experience
cp ../docker-compose.yaml .
git add docker-compose.yaml
git rm api/docker-compose.yml consumer/docker-compose.yml
git commit -m "Add docker-compose file in the root of project replacing the specific module ones"
 
cp ../ApiDockerfile api/Dockerfile
cp ../ConsumerDockerfile consumer/Dockerfile
git add api/ consumer/
git commit -m "Update Dockerfiles"
 
# Put in place new Jenkins config
cp ../Jenkinsfile .
git add Jenkinsfile
git rm api/Jenkinsfile
git rm consumer/Jenkinsfile
git rm events/Jenkinsfile
git rm commands/Jenkinsfile
git commit -m "Add new Jenkinsfile and remove the old ones"
 
./gradlew build
 
git push -f -u origin master
git co -b develop
git push -f -u origin develop

I got inspired by an nice post on Medium.com from an ex colleague of mine. Thanks Fred! :) Please share this article if you liked it and as always I would love your feedback in the comments below.

Blog.

Merge multiple gradle projects including git history

TL;DR

More Stories

Start on your first Golang project

How to add network driver to Windows 10 PE

Responsive Adaptive Progressive impressive webpages

Upgrade Raspbian Jessie to RaspbianStretch