Page 1 of 1

[SOLVED] Synchronizing a git fork with the master

Posted: Fri Sep 15, 2017 7:45 pm
by mad_ady
I have forked hardkernel/linux a few months ago and made some changes related to netconsole which I pushed upstream. I'd like to make some other changes, but I haven't figured out how to synchronize my fork with hardkernel's release.
I tried the steps here: https://gist.github.com/CristinaSolana/1885435, but I got an error when I did git pull upstream odroidxu4-4.9.y:

Code: Select all

git clone git@github.com:mad-ady/linux.git
cd linux/
git remote add upstream git://github.com/hardkernel/linux
git fetch upstream
git pull upstream odroidxu4-4.9.y
...
Auto-merging Documentation/00-INDEX
warning: inexact rename detection was skipped due to too many files.
warning: you may want to set your merge.renamelimit variable to at least 32092 and retry the command.
Automatic merge failed; fix conflicts and then commit the result.
If I try to delete my forked git repo to start over, github warns that bad things happen, so I didn't do it.
Any ideas on what I should do?

Re: Synchronizing a git fork with the master

Posted: Fri Sep 15, 2017 11:57 pm
by LiquidAcid
I see multiple issues here.

1) First of all, when you clone your repository, the default branch is checked out. In this case the default branch of mad-ady/linux.git is odroid-3.0.y, which is obviously not the branch you want to merge changes in.
2) If you have changes in your branch which you plan to upstream at some point, I would advise against git-merging (which git-pull does internally). I would rather rebase your branch on upstream/odroidxu4-4.9.y, so that you have your changes "on top".

Re: Synchronizing a git fork with the master

Posted: Sat Sep 16, 2017 12:27 am
by mad_ady
I'm sure there are multiple issues which get amplified by my lack of git experience :)
My branch had changes which were pulled in the upstream branch and the rest is missing commits from upstream (which I'd like to pull)
I will look into rebasing, thanks

Re: Synchronizing a git fork with the master

Posted: Sat Sep 16, 2017 2:24 am
by elatllat
Advice to rebase instead of merge is misplaced, it's a lot more work for no advantage whatsoever; Before a pull request one should

Code: Select all

git merge --squash my_work
into a fresh branch anyway.


As stated the rename warning is likely that you are trying to merge the wrong branch;
you should run

Code: Select all

git status
and

Code: Select all

git diff --stat upstream/odroidxu4-4.9.y HEAD
and if that looks OK then

Code: Select all

git merge upstream/odroidxu4-4.9.y

Code: Select all

git branch -a
will show you all the branches made available by fetch as well as your own.
(They are local copies of remotes, you noramly only need to be online for fetch and push ).

Re: Synchronizing a git fork with the master

Posted: Sat Sep 16, 2017 2:44 am
by mad_ady
Hmm, this seems like a lot of work when my end goal is to send PR back upstream... What would happen to my previous PRs if I were to delete my fork and create a new one?

Re: Synchronizing a git fork with the master

Posted: Sat Sep 16, 2017 2:46 am
by elatllat
what work?
why would you delete anything?

if you know what your doing all you NEED is

Code: Select all

git fetch upstream && git merge upstream/odroidxu4-4.9.y && git push origin
the other examples are just to convince yourself you are doing the correct thing.
Also so long as there are no conflicts you don't even need to do that, it's just that merging and squashing is polite because it saves the person an extra step in some circumstances and produces a clean history.

Re: Synchronizing a git fork with the master

Posted: Sat Sep 16, 2017 2:52 am
by mad_ady
Ok, I'lll give it a try again on monday, thank you!

Re: Synchronizing a git fork with the master

Posted: Sat Sep 16, 2017 3:00 am
by elatllat
To be clear; in git it's all about branches you only ever need one fork.

...and everything is incremental so fetch/gc/merge will be faster and more manageable the more often you run them.
Also often you can "git clone --depth 1 branch" to speed up the initial download.

..and git is the best thing since Linux.. well maybe vim, rsync, and COW in there somewhere... just so much smart code but git and Linux make all the others better.

Re: Synchronizing a git fork with the master

Posted: Sat Sep 16, 2017 3:40 am
by LiquidAcid
elatllat wrote:Advice to rebase instead of merge is misplaced, it's a lot more work for no advantage whatsoever; Before a pull request one should

Code: Select all

git merge --squash my_work
into a fresh branch anyway..
That's only sensible if you just do _one_ single functional change, i.e. something that should be done in one commit. Once you're dealing with more changes, this approach doesn't work anymore.

Rebasing your work on a regular basis saves you the work to fix potential merge conflicts later (since upstream has moved on while you were working on your code).

Re: Synchronizing a git fork with the master

Posted: Sat Sep 16, 2017 5:33 am
by elatllat
What?
each functional change gets it's own branch, whether you merge or rebase has no impact on that (even if they are based off eachother)... and if you want > features on a branch you just _ merge them ;)
merge will update the code the same as rebase so no impact there either, only merge is better because of advanced merge strategies.

The only diffrence is mege has better strategies, both can produce the same result.

Re: Synchronizing a git fork with the master

Posted: Sat Sep 16, 2017 6:10 am
by LiquidAcid
I'm not sure what you want to tell me here. I was solely commenting on your "git merge --squash". This gives me exact one commit, since all commits are squashed. I was just saying that this might not be what I want.

Re: Synchronizing a git fork with the master

Posted: Sat Sep 16, 2017 7:40 pm
by elatllat
well one nice thing about git is there are so many options so if you and your upstream are happy it's all good.

I was just pointing out that a merge+squash per feature can be less work (less conflict resolution) than rebase with the same result, because conflicts are the only non scriptabe part of git... remember; branches are cheap so use a branch per feature instead of a commit per feature and you can avoide rebase and the extra conflicts it creates.

Re: Synchronizing a git fork with the master

Posted: Mon Sep 25, 2017 8:52 pm
by mad_ady
@elatilat: Thank you, I've just now had the time to try it.
This is what I did, and it worked (I'm leaving it here for a complete future reference):

Code: Select all

git clone -b odroidxu4-4.9.y git@github.com:mad-ady/linux.git
cd linux/
git remote add upstream git://github.com/hardkernel/linux
git fetch upstream
git merge upstream/odroidxu4-4.9.y
git push origin
... and now my branch is on par with HardKernel's

Thank you for your help!

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Thu Dec 14, 2017 6:10 pm
by mad_ady
Sorry to revive this, but I have a related question. I have forked the linux repo from Hardkernel a while ago, and now I'd like to clone their 4.14 branch in my repo, so that I can make some changes and push something back. Any suggestions how this could be done?

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Thu Dec 14, 2017 11:33 pm
by elatllat
start with

Code: Select all

cat .git/config
it will show you the current settings for your repo.
If you don't see something like this:

Code: Select all

[remote "hardkernel"]
	url = https://github.com/hardkernel/linux.git
	fetch = +refs/heads/*:refs/remotes/hardkernel/*
	pushurl = no_push
add it [1] . Once it's there and you have fetched[2] you can list remote branches with

Code: Select all

git branch -a | grep 4.14
and checkout a copy with

Code: Select all

git checkout -b my-4.14 remotes/hardkernel/odroidxu4-4.14.y

[1] To avoid breaking the config while editing (with vim etc) it's advised to use git commands like

Code: Select all

git remote add hardkernel https://github.com/hardkernel/linux.git
, and

Code: Select all

git remote -v
.
[2] When you have more than one remote and they are not all R/W you should spesify where you want git to send your changes

Code: Select all

git push origin
and where to get changes from

Code: Select all

git fetch --all
[3] be sure to skim the related help/manuals git-remote, git-branch, git-checkout, git-push, git-fetch, git-help.

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Fri Dec 15, 2017 12:14 am
by mad_ady
Thanks for the tips!
I've cloned the 4.9 branch from my forked repo:

Code: Select all

$ git clone -b odroidxu4-4.9.y git@github.com:mad-ady/linux.git                                                                     
Cloning into 'linux'...
remote: Counting objects: 5548803, done.
remote: Compressing objects: 100% (576/576), done.
remote: Total 5548803 (delta 709), reused 422 (delta 422), pack-reused 5547805
Receiving objects: 100% (5548803/5548803), 1.99 GiB | 7.25 MiB/s, done.
Resolving deltas: 100% (4543966/4543966), done.
Checking connectivity... done.
Checking out files: 100% (57634/57634), done.
And this is config:

Code: Select all

cat .git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = git@github.com:mad-ady/linux.git
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "odroidxu4-4.9.y"]
        remote = origin
        merge = refs/heads/odroidxu4-4.9.y

However, when I try to add origin, it complains:

Code: Select all

$ git remote add origin https://github.com/hardkernel/linux.git
fatal: remote origin already exists.

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Fri Dec 15, 2017 1:02 am
by elatllat
Name the remote "hardkernel" not "origin", and re-read the edits made to my last post.

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Fri Dec 15, 2017 4:37 pm
by mad_ady
Thanks again. Here's what I did:

Code: Select all

$ git remote add hardkernel https://github.com/hardkernel/linux.git
$ git remote -v
hardkernel      https://github.com/hardkernel/linux.git (fetch)
hardkernel      https://github.com/hardkernel/linux.git (push)
origin  git@github.com:mad-ady/linux.git (fetch)
origin  git@github.com:mad-ady/linux.git (push)
$ git push origin
warning: push.default is unset; its implicit value has changed in
Git 2.0 from 'matching' to 'simple'. To squelch this message
and maintain the traditional behavior, use:

  git config --global push.default matching

To squelch this message and adopt the new behavior now, use:

  git config --global push.default simple

When push.default is set to 'matching', git will push local branches
to the remote branches that already exist with the same name.

Since Git 2.0, Git defaults to the more conservative 'simple'
behavior, which only pushes the current branch to the corresponding
remote branch that 'git pull' uses to update the current branch.

See 'git help config' and search for 'push.default' for further information.
(the 'simple' mode was introduced in Git 1.7.11. Use the similar mode
'current' instead of 'simple' if you sometimes use older versions of Git)

Everything up-to-date
$ git fetch --all
Fetching origin
Fetching hardkernel
remote: Counting objects: 605278, done.
remote: Compressing objects: 100% (446/446), done.
remote: Total 605278 (delta 247245), reused 247076 (delta 247036), pack-reused 357796
Receiving objects: 100% (605278/605278), 246.99 MiB | 5.19 MiB/s, done.
Resolving deltas: 100% (510453/510453), completed with 49611 local objects.
From https://github.com/hardkernel/linux
 * [new branch]      odroid-3.0.y -> hardkernel/odroid-3.0.y
 * [new branch]      odroid-3.0.y-android -> hardkernel/odroid-3.0.y-android
 * [new branch]      odroid-3.13.y -> hardkernel/odroid-3.13.y
 * [new branch]      odroid-3.13.y-linaro -> hardkernel/odroid-3.13.y-linaro
 * [new branch]      odroid-3.14.y-linaro -> hardkernel/odroid-3.14.y-linaro
 * [new branch]      odroid-3.8.y -> hardkernel/odroid-3.8.y
 * [new branch]      odroid-3.8.y-rt -> hardkernel/odroid-3.8.y-rt
 * [new branch]      odroidc-3.10.y -> hardkernel/odroidc-3.10.y
 * [new branch]      odroidc-3.10.y-android -> hardkernel/odroidc-3.10.y-android
 * [new branch]      odroidc-3.10.y-android-lollipop -> hardkernel/odroidc-3.10.y-android-lollipop
 * [new branch]      odroidc2-3.14.y -> hardkernel/odroidc2-3.14.y
 * [new branch]      odroidc2-3.14.y-android -> hardkernel/odroidc2-3.14.y-android
 * [new branch]      odroidc2-3.14.y-android-m -> hardkernel/odroidc2-3.14.y-android-m
 * [new branch]      odroidw-3.12.y -> hardkernel/odroidw-3.12.y
 * [new branch]      odroidw-3.12.y-test -> hardkernel/odroidw-3.12.y-test
 * [new branch]      odroidxu-3.4.y -> hardkernel/odroidxu-3.4.y
 * [new branch]      odroidxu-3.4.y-android -> hardkernel/odroidxu-3.4.y-android
 * [new branch]      odroidxu-3.4.y-android-jb -> hardkernel/odroidxu-3.4.y-android-jb
...
 * [new tag]         jenkins-deb_kernel_c1-189 -> jenkins-deb_kernel_c1-189
 * [new tag]         jenkins-deb_kernel_c1-190 -> jenkins-deb_kernel_c1-190
 * [new tag]         s805_4.4.4_v3.6 -> s805_4.4.4_v3.6
 * [new tag]         s805_4.4.4_v3.7 -> s805_4.4.4_v3.7
 * [new tag]         s905_6.0.1_v2.8 -> s905_6.0.1_v2.8
 * [new tag]         s905_6.0.1_v3.0 -> s905_6.0.1_v3.0
 * [new tag]         s905_6.0.1_v3.1 -> s905_6.0.1_v3.1
 * [new tag]         s905_6.0.1_v3.2 -> s905_6.0.1_v3.2
 * [new tag]         s905_6.0.1_v3.3 -> s905_6.0.1_v3.3
 * [new tag]         s905_6.0.1_v3.4 -> s905_6.0.1_v3.4
$ git checkout odroidxu4-4.14.y      
Checking out files: 100% (41201/41201), done.
Branch odroidxu4-4.14.y set up to track remote branch odroidxu4-4.14.y from hardkernel.
Switched to a new branch 'odroidxu4-4.14.y'
$ git push
...
remote: Permission to hardkernel/linux.git denied to mad-ady.
fatal: unable to access 'https://github.com/hardkernel/linux.git/': The requested URL returned error: 403

So, it seems I have pulled the remote branch in my local repo, but I'm trying to synchronize it with my github copy, and it's trying to push to hardkernel instead... I'll be reading the documentation for a while... :)

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Fri Dec 15, 2017 5:44 pm
by mad_ady
I've also tried these steps (which are more or less what @elatilat suggested): https://help.github.com/articles/fork-a-repo/ https://help.github.com/articles/syncing-a-fork/, but I still end up trying to push changes to the Hardkernel repo, instead of to my own fork...

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Fri Dec 15, 2017 8:04 pm
by elatllat
Why did you 'git push' instead of 'git push origin' ?

You have to use the latter every time you want to push.

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Fri Dec 15, 2017 8:16 pm
by mad_ady
Thank you. That was the missing bit. It's now chugging along...

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Thu Jan 10, 2019 10:38 pm
by mad_ady
Ugh... I've been fighting with this again for half a day. Every few months I find myself revisiting this thread, but every time it's a pain...

Problem - I want to fork https://github.com/Owersun/linux-hardke ... le-patches and add that branch to my own github (https://github.com/mad_ady/linux), add a few patches and push them to Owersun.

This is what I've tried so far:

Code: Select all

git clone https://github.com/mad-ady/linux.git linux-owersun
cd linux-owersun/
git checkout -b odroidn1-4.4.y-with-le-patches
git remote add upstream https://github.com/Owersun/linux-hardkernel/
git fetch upstream
git config merge.renameLimit 999999
git merge remotes/upstream/odroidn1-4.4.y-with-le-patches
it crunches for a while, but eventually finishes with:

Code: Select all

Automatic merge failed; fix conflicts and then commit the result.
Now, here's one thing I don't get. I had created a new branch (which in my mind is comparable to a blank directory) and I wanted to pull all Owersun's changes from his branch into. Clearly this is not how things should be done, and I've been going in circles trying to figure out what's wrong. Last month I tried the same (I think) procedure to pull odroidn1-4.4.y from hardkernel's git into mine, and that worked. Is the problem that I already forked hardkernel's git and I can't import a branch from Oversun?

I'm deeply confused...

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Thu Jan 10, 2019 10:57 pm
by crashoverride
mad_ady wrote:
Thu Jan 10, 2019 10:38 pm
I had created a new branch (which in my mind is comparable to a blank directory) and I wanted to pull all Owersun's changes from his branch into.
A new branch is a fork at a specific commit. After creating the branch, you need to reset it to the commit you want as base. Otherwise, it will just be a copy of whatever it was before you created the branch. To pull/fetch from a different repo, you will need to reset the branch to the last point in history the two repos have in common if you intend to push to it.

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Thu Jan 10, 2019 11:14 pm
by mad_ady
I see. Is there a git command that can tell me where the two branches meet?

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Thu Jan 10, 2019 11:34 pm
by crashoverride
mad_ady wrote:
Thu Jan 10, 2019 11:14 pm
Is there a git command that can tell me where the two branches meet?
I have no idea. I commented once that I need to hire someone just to manage git! :lol:

I think what you want is to add the remote repository and then checkout the remote branch:

Code: Select all

git checkout -b local_branch_name remote_name/remote_branch_name

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Thu Jan 10, 2019 11:38 pm
by mad_ady
It seems there is a command: https://stackoverflow.com/questions/154 ... o-branches
Will play with it tomorrow, thanks

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Fri Jan 11, 2019 12:49 am
by elatllat
mad_ady wrote:
Thu Jan 10, 2019 10:38 pm
... I want to ... add a few patches and push them to Owersun...
Before you do this it helps if you identify how far apart the 2 branches you want to swap code from are from each other.
If they are close (there are only 2 change sets) then you can merge upstream into both branches then merge them together.
If they are far (they don't share a recent upstream) then you can cherry pick your commits to the other branch.

To find out haw far branches are from each other;
- same project (linux)
- same upstream (github/hk)
- same branch/version (4.19)
- same age/build (4.19.14)
- only one change on the fork (git diff --stat should be small)

If you get to the bottom of that list use merge then if there is a conflict resolve it;
git status
git mergetool -t vimdiff
git commit

If things differ closer to the top of the list cherry pick the commits you want to avoid fixing conflicts for changes you don't care about.

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Fri Jan 11, 2019 2:08 am
by mad_ady
Owersun's branch is ~ 1000 commits ahead Hardkernel and I only have 2-3 commits I'd like to add (and I don't particularly care to keep my own fork afterwards).

One way to do it would be to clone his repo, add the patches locally and create diff files for Owersun to merge himself - though this isn't cherrypicking as far as I understand and doesn't seem to being the correct way of doing things.
Ideally, I'd like to fork his repo (under a new name) so that I can add the patches and send him a pull request - completely ignoring all the existing branches I have under my existing linux fork. Most likely this is a fault in my reasoning and this goes against git's principles and that's why I'm having a hard time.

Am I missing something obvious, or is it more complicated because I'm learning on the linux git which is the most complex?

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Fri Jan 11, 2019 2:15 am
by elatllat
The cherry-pick option:

Code: Select all

cd /path/to/your/repo
git remote add Owersun https://github.com/Owersun/linux-hardkernel.git
git fetch --all
git checkout -b patch-for-owersun remotes/Owersun/odroidn1-4.4.y-with-le-patches
git cherry-pick commitID1..commitID3
#if conflicts #git mergetool -t vimdiff && git commit 
git push origin patch-for-owersun:patch-for-owersun.
then press the PR button on github

The merge option:

Code: Select all

cd /path/to/your/repo
git remote add Owersun https://github.com/Owersun/linux-hardkernel.git
git fetch --all
git checkout -b patch-for-owersun remotes/Owersun/odroidn1-4.4.y-with-le-patches
git checkout -b tmp master
git merge patch-for-owersun
#if conflicts #git mergetool -t vimdiff && git commit
git reset --soft patch-for-owersun
git commit -m patch-for-owersun
git push origin patch-for-owersun:patch-for-owersun.
then press the PR button on github

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Fri Jan 11, 2019 2:31 am
by mad_ady
I'll try it tomorrow, thanks. The cherry picks ids are from my own branch where I already have those commits, right (e.g. git can find them in a different branch)?

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Fri Jan 11, 2019 2:34 am
by elatllat
Yes;

A branch is just a named commit (so is a tag) what matters is the commit "tree" eg any commit can be merged into any other (non-parent) commit, but to keep things manageable there are best practices;
one minimal feature per branch/PR
merge before PR
squash before PR
merge branch heads of old versions into newer ones
branch and merge early and often

Re: [SOLVED] Synchronizing a git fork with the master

Posted: Fri Jan 11, 2019 10:18 pm
by mad_ady
@elatllat: I went with your suggestions for cherry picking. I didn't actually use the cherry pick command, but I added my changes on my local branch cloned from Owersun's repo. I pushed everything to my github now, but with your suggested commands, but when I try to send the push request, it tries to do it against hardkernel's repo, not Owersun's. I can select a different repo in github, but Owerun's doesn't appear there. I'm a bit lost again :)

Edit: I navigated to https://github.com/Owersun/linux-hardkernel and I could see a button to merge my branch with that repo. This worked.
Thanks everyone for the help! I can't claim I understand the whole process, but at least I have something written down for next time.