Easy way to pull latest of all git submodules

Brad Robinson Source

We're using git submodules to manage a couple of large projects that have dependencies on many other libraries we've developed. Each library is a separate repo brought into the dependant project as a submodule. During development, we often want to just go grab the latest version of every dependant submodule.

Does git have a built in command to do this? If not, how about a Windows batch file or similar that can do it?

gitgit-submodules

Answers

answered 9 years ago gahooa #1

Note: This is from 2009 and may have been good then but there are better options now.

We use this. It's called git-pup:

#!/bin/bash
# Exists to fully update the git repo that you are sitting in...

git pull && git submodule init && git submodule update && git submodule status

Just put it in a suitable bin directory (/usr/local/bin). If on Windows, you may need to modify the syntax to get it to work :)

Update:

In response to the comment by the original author about pulling in all of the HEADs of all of the submodules -- that is a good question.

I am pretty sure that git does not have a command for this internally. In order to do so, you would need to identify what HEAD really is for a submodule. That could be as simple as saying master is the most up to date branch, etc...

Following this, create a simple script that does the following:

  1. check git submodule status for "modified" repositories. The first character of the output lines indicates this. If a sub-repo is modified, you may NOT want to proceed.
  2. for each repo listed, cd into it's directory and run git checkout master && git pull. Check for errors.
  3. At the end, I suggest you print a display to the user to indicate the current status of the submodules -- perhaps prompt them to add all and commit?

I'd like to mention that this style is not really what git submodules were designed for. Typically, you want to say "LibraryX" is at version "2.32" and will stay that way until I tell it to "upgrade".

That is, in a sense, what you are doing with the described script, but just more automatically. Care is required!

Update 2:

If you are on a windows platform, you may want to look at using Python to implement the script as it is very capable in these areas. If you are on unix/linux, then I suggest just a bash script.

Need any clarifications? Just post a comment.

answered 9 years ago baudtack #2

I think you'll have to write a script to do this. To be honest, I might install python to do it so that you can use os.walk to cd to each directory and issue the appropriate commands. Using python or some other scripting language, other than batch, would allow you to easily add/remove subprojects with out having to modify the script.

answered 9 years ago Henrik Gustafsson #3

For git 1.8.2 or above the option --remote was added to support updating to latest tips of remote branches:

git submodule update --recursive --remote

This has the added benefit of respecting any "non default" branches specified in the .gitmodules or .git/config files (if you happen to have any, default is origin/master, in which case some of the other answers here would work as well).

For git 1.7.3 or above you can use (but the below gotchas around what update does still apply):

git submodule update --recursive

or:

git pull --recurse-submodules

if you want to pull your submodules to latest commits intead of what the repo points to.

Note: If that's the first time you checkout a repo you need to use --init first:

git submodule update --init --recursive

For older, git 1.6.1 or above you can use something similar to (modified to suit):

git submodule foreach git pull origin master

See git-submodule(1) for details

answered 9 years ago mturquette #4

Henrik is on the right track. The 'foreach' command can execute any arbitrary shell script. Two options to pull the very latest might be,

git submodule foreach git pull origin master

and,

git submodule foreach /path/to/some/cool/script.sh

That will iterate through all initialized submodules and run the given commands.

answered 9 years ago jerico.dev #5

Look at http://lists.zerezo.com/git/msg674976.html which introduces a --track parameter

answered 7 years ago antitoxic #6

Edit:

In the comments was pointed out (by philfreo ) that the latest version is required. If there is any nested submodules that need to be in their latest version :

git submodule foreach --recursive git pull

-----Outdated comment below-----

Isn't this the official way to do it ?

git submodule update --init

I use it every time. No problems so far.

Edit:

I just found that you can use:

git submodule foreach --recursive git submodule update --init 

Which will also recursively pull all of the submodules, i.e. dependancies.

answered 7 years ago zachleat #7

The following worked for me on Windows.

git submodule init
git submodule update

answered 7 years ago Jens Kohl #8

I don't know since which version of git this is working, but that's what you're searching for:

git submodule update --recursive

I use it with git pull to update the root repository, too:

git pull && git submodule update --recursive

answered 7 years ago Alexander Bartosh #9

If you need to pull stuff for submodules into your submodule repositories use

git pull --recurse-submodules

a feature git first learned in 1.7.3.

But this will not checkout proper commits(the ones your master repository points to) in submodules

To checkout proper commits in your submodules you should update them after pulling using

git submodule update --recursive --remote

answered 5 years ago Sebastien Varrette #10

As it may happens that the default branch of your submodules is not master, this is how I automate the full Git submodules upgrades:

git submodule init
git submodule update
git submodule foreach 'git fetch origin; git checkout $(git rev-parse --abbrev-ref HEAD); git reset --hard origin/$(git rev-parse --abbrev-ref HEAD); git submodule update --recursive; git clean -dfx'

answered 4 years ago abc123 #11

On init running the following command:

git submodule update --init --recursive

from within the git repo directory, works best for me.

This will pull all latest including submodules.

Explained

git - the base command to perform any git command
    submodule - Inspects, updates and manages submodules.
        update - Update the registered submodules to match what the superproject
        expects by cloning missing submodules and updating the working tree of the
        submodules. The "updating" can be done in several ways depending on command
        line options and the value of submodule.<name>.update configuration variable.
            --init without the explicit init step if you do not intend to customize
            any submodule locations.
            --recursive is specified, this command will recurse into the registered
            submodules, and update any nested submodules within.

After this you can just run:

git submodule update --recursive

from within the git repo directory, works best for me.

This will pull all latest including submodules.

Explained

git - the base command to perform any git command
    submodule - Inspects, updates and manages submodules.
        update - Update the registered submodules to match what the superproject
        expects by cloning missing submodules and updating the working tree of the
        submodules. The "updating" can be done in several ways depending on command
        line options and the value of submodule.<name>.update configuration variable.
            any submodule locations.
            --recursive is specified, this command will recurse into the registered
            submodules, and update any nested submodules within.

answered 3 years ago Yash #12

First time

Clone and Init Submodule

git clone [email protected]:speedovation/kiwi-resources.git resources
git submodule init

Rest

During development just pull and update submodule

git pull --recurse-submodules  && git submodule update --recursive

Update Git submodule to latest commit on origin

git submodule foreach git pull origin master

Preferred way should be below

git submodule update --remote --merge

note: last two commands have same behaviour

answered 3 years ago kenorb #13

Here is the command-line to pull from all of your git repositories whether they're or not submodules:

ROOT=$(git rev-parse --show-toplevel 2> /dev/null)
find "$ROOT" -name .git -type d -execdir git pull -v ';'

If you running it in your top git repository, you can replace "$ROOT" into ..

answered 3 years ago Orient #14

Remark: not too easy way, but workable and it has its own unique pros.

If one want to clone only HEAD revision of a repository and only HEADs of all the its submodules (i.e. to checkout "trunk"), then one can use following Lua script. Sometimes simple command git submodule update --init --recursive --remote --no-fetch --depth=1 can result in an unrecoverable git error. In this case one need to clean up subdirectory of .git/modules directory and clone submodule manually using git clone --separate-git-dir command. The only complexity is to find out URL, path of .git directory of submodule and path of submodule in superproject tree.

Remark: the script is only tested against https://github.com/boostorg/boost.git repository. Its peculiarities: all the submodules hosted on the same host and .gitmodules contains only relative URLs.

-- mkdir boost ; cd boost ; lua ../git-submodules-clone-HEAD.lua https://github.com/boostorg/boost.git .
local module_url = arg[1] or 'https://github.com/boostorg/boost.git'
local module = arg[2] or module_url:match('.+/([_%d%a]+)%.git')
local branch = arg[3] or 'master'
function execute(command)
    print('# ' .. command)
    return os.execute(command)
end
-- execute('rm -rf ' .. module)
if not execute('git clone --single-branch --branch master --depth=1 ' .. module_url .. ' ' .. module) then
    io.stderr:write('can\'t clone repository from ' .. module_url .. ' to ' .. module .. '\n')
    return 1
end
-- cd $module ; git submodule update --init --recursive --remote --no-fetch --depth=1
execute('mkdir -p ' .. module .. '/.git/modules')
assert(io.input(module .. '/.gitmodules'))
local lines = {}
for line in io.lines() do
    table.insert(lines, line)
end
local submodule
local path
local submodule_url
for _, line in ipairs(lines) do
    local submodule_ = line:match('^%[submodule %"([_%d%a]-)%"%]$')
    if submodule_ then
        submodule = submodule_
        path = nil
        submodule_url = nil
    else
        local path_ = line:match('^%s*path = (.+)$')
        if path_ then
            path = path_
        else
            submodule_url = line:match('^%s*url = (.+)$')
        end
        if submodule and path and submodule_url then
            -- execute('rm -rf ' .. path)
            local git_dir = module .. '/.git/modules/' .. path:match('^.-/(.+)$')
            -- execute('rm -rf ' .. git_dir)
            execute('mkdir -p $(dirname "' .. git_dir .. '")')
            if not execute('git clone --depth=1 --single-branch --branch=' .. branch .. ' --separate-git-dir ' .. git_dir .. ' ' .. module_url .. '/' .. submodule_url .. ' ' .. module .. '/' .. path) then
                io.stderr:write('can\'t clone submodule ' .. submodule .. '\n')
                return 1
            end
            path = nil
            submodule_url = nil
        end
    end
end

answered 3 years ago seoul #15

Git for windows 2.6.3:

git submodule update --rebase --remote

answered 3 years ago Tom #16

I did this by adapting gahooa's answer above:

Integrate it with a git [alias] ...

If your parent project has something like this in .gitmodules:

[submodule "opt/submodules/solarized"]
    path = opt/submodules/solarized
    url = [email protected]:altercation/solarized.git
[submodule "opt/submodules/intellij-colors-solarized"]
    path = opt/submodules/intellij-colors-solarized
    url = [email protected]:jkaving/intellij-colors-solarized.git

Add something like this inside your .gitconfig

[alias]
    updatesubs = "!sh -c \"git submodule init && git submodule update && git submodule status\" "

Then to update your submodules, run:

git updatesubs

I have an example of it in my environment setup repo.

answered 3 months ago JamesD #17

The above answers are good, however we were using git-hooks to make this easier but it turns out that in git 2.14, you can set git config submodule.recurse to true to enable submodules to to updated when you pull to your git repository.

This will have the side effect of pushing all submodules change you have if they are on branches however, but if you have need of that behaviour already this could do the job.

comments powered by Disqus