[ClusterLabs Developers] [RFC] Time to migrate authoritative source forge elsewhere?

[Nils Carlson]

On Wed, 13 Jun 2018, Adam Spiers wrote:

> Nils Carlson <nils.carlson at ludd.ltu.se> wrote: 
>> On 06/11/2018 08:31 PM, Adam Spiers wrote: 
>>> Nils Carlson <nils.carlson at ludd.ltu.se> wrote: 
>>>> On 06/11/2018 03:18 PM, Adam Spiers wrote: 
>>>>> Nils Carlson <pyssling at ludd.ltu.se> wrote: 
>>>>>> Gerrit is a much more powerful tool for code-review. The workflow is 
>>>>>> less intuitive however and has a far higher learning curve. 
>>>>> 
>>>>> I disagree, but please can you clarify which version of Gerrit you are 
>>>>> referring to?  The latest release (2.15) is an enormous leap forwards in 
>>>>> usability: 
>>>> 
>>>> Feel free to disagree, this isn't based on opinion on my part but 
>>>> experience, I have migrated two companies (with around 20-30 developers) 
>>>> to gerrit and gitlab respectively and developers have adjusted far faster 
>>>> to gitlab than to gerrit. The version was pre-2.15, 2.13 I believe, but 
>>>> nothing revolutionary seems to have happened (the same patch-set based 
>>>> workflow). 
>>> 
>>> 2.15 *is* revolutionary in terms of UX because the UI was rewritten from 
>>> scratch (PolyGerrit).  True, the workflow is more or less the same in 
>>> design, but how intuitive it will feel to newcomers (your original 
>>> criticism above) depends heavily on the UI. 
>> 
>> Maybe, but I doubt it would change much. People get over the bad UI pretty 
>> quickly (weeks,) the workflow is the hard part. 
>
> I'm still hoping to hear something which explains to me why the workflow is 
> hard.

I think you need to walk through the non-trivial use-cases. Things 
like what happens when you press the rebase button, how users can push 
their own task branches to the server as a backup/convencience, and why 
they don't merge these task branches and instead need to create reviews.

Gerrit in my view tries to re-create a kernel-like patch 
submission workflow with a web-gui instead of e-mail. Users coming from 
SCMs like Perforce, ClearCase that are used to a branch-centric world-view 
find this very strange. Users on Gitlab/GitHub are also used to a branch 
focused work-flow and find Gerrit strange.

Hard is probably the wrong word, maybe unusual and different are better 
terms, but users definitely experienced Gerrit as hard.

>>>> Personally I much prefer gerrit, but I have been using git since 2009 
>>>> including kernel coding which puts higher requirements on skill than most 
>>>> systems. 
>>> 
>>> Yes, as an occasional git hacker I'm in a similar boat. 
>>>> For your average developer Gerrit is a lot harder to adjust to it seems.
>>> 
>>> Yes, like I said I've heard this several times, but in the absence of 
>>> concrete details I'm really struggling to understand how this could be 
>>> more than natural inertia rather than genuinely significant practical 
>>> hurdles. 
>>> However I'm not only willing to be proven wrong but actively seeking good 
>>> explanations which *would* prove me wrong.  This is because I've already 
>>> volunteered to write a document which helps people familiar with GitHub 
>>> migrate to Gerrit (intended for the OpenStack community but of course it 
>>> could be shared to a wider audience).  And to do that well I really need 
>>> help understanding what people struggle with when learning Gerrit the 
>>> first time.  So if you can help with that I'd be really grateful, although 
>>> don't worry if you don't have time :-) [snipped]
>>> 
>>>>> but this is mostly an internal implementation detail; in practice 
>>>>> everyone uses a tool such as "git review" which handles all that 
>>>>> automatically behind scenes. 
>>>> 
>>>> Yes, extra tooling is more or less the norm when using gerrit. 
>>> 
>>> It's pretty common with GitHub too, I suspect.  But even if not, I don't 
>>> see a one-off installation of a simple tool as a huge hurdle. 
>> 
>> The difference is really on the IDE side. Eclipse has excellent integration 
>> with gerrit these days, but most other IDEs do not. Configuring your 
>> prefered IDE for Gerrit is not trivial in most cases. 
>> I've seen specific tools for GitHub, but never seen anyone use them. GitHub 
>> does not make use of specific review remotes as Gerrit does, so there is no 
>> need for specific tooling or configuration. This is the case with most of 
>> these tools, only Gerrit and Phabricator require/encourage specific 
>> tooling. 
>
> Yes, I accept that there is a need for most Gerrit users to install 
> git-review.  But as I already said, this is a one-off cost which is low or 
> even extremely low, and the pay-off is a workflow which is significantly 
> simpler than GitHub or GitLab. 
>>> Furthermore, without a helper tool or IDE which supports GitHub or GitLab, 
>>> creating a pull request or merge request requires several more steps than 
>>> with Gerrit:  0. Ensure that you have a remote fork to which you have push 
>>>     access (e.g. by clicking the merge button if you don't     have push 
>>> access to the target repository) 
>>>  1. Push your branch to a remote fork 
>>>  2. Locate that branch in the web UI 
>>>  3. Click the button(s) to bring up the window in which you     draft the 
>>> pull/merge request 
>>>  4. Give the request a title and description 
>>>  5. Submit it 
>>> Compared to the single step "type git review", I'm somewhat mystified how 
>>> anyone can claim that Gerrit is harder to get started with.
>> 
>> As any programmer will tell you, complexity isn't measured in number of 
>> steps. :-) 
>
> Hrm, I've been programming for 36 years and don't recall ever hearing that. 
> Most likely I did hear it once or twice and forgot, but I don't think it's a 
> common maxim.  A quick Google didn't reveal anything, at least.

Well, it is measured in number of iterations/operations for algorithms.

What I was thinking of was rather: "Give me 5 lines of normal C code 
rather than a perl regex one-liner".

>> Each step above is trivial and some are in fact not needed, and all are 
>> easy to explain. 
>
> Agreed, but surely you aren't arguing that they are simpler than the single 
> step "type git review"?

Simpler to understand - definitely. "git review" wraps up a lot of 
complexity in one command.

>> The above however isn't correct. For an open repo on Gitlab the steps would 
>> be as follows: 
>> 1. Push your commit to a branch of the origin repo (assuming you are part 
>> of the project, or have push access to a set of branches which can be 
>> granted to any signed-in user), get a URL on the command line for creating 
>> a merge-request. 
>
> That's two or three steps.  They don't become a single step just because you 
> put them under a single bullet item ;-)

One action - git push.


>> 2. Click the URL, review the contents of the merge-request (if you pushed 
>> only a single commit Gitlab will take the title and comment from that 
>> commit), click create at the bottom. 
> Ditto here - multiple steps. 
> The only difference I can see between that and what I wrote is that my step 2 
> can be circumvented by using the URL provided by the output of the CLI 
> command.  That's a nice trick - thanks.

and reusing the commit message title and text as the subject and description.

Two actions, following the url and clicking create.

>> Sure, gerrit can do this in one step without opening the we browser, but in 
>> general most people want to look at the review before forcing others to do 
>> so. 
>
> In Gerrit there is no difference between the review and the commit, therefore 
> this can be accomplished by running "git show" before "git review".  Gerrit's 
> 1:1 mapping between commits and reviews is a large part of why its workflow 
> is so much simpler than GitHub's and GitLab's.

For a single commit no. For a series of commits you need to know which branch
you're merging to to know the patch-sets that pushing will create.

>>>>>> The negative sides of Gerrit typically outweigh the positive for most 
>>>>>> organizations I'm afraid: - No central hosting like gitlab.com. 
>>>>> 
>>>>> That's not really true.  There are SaaS hosters of Gerrit, most notably 
>>>>> GerritForge who also offer completely free hosting for FLOSS projects on 
>>>>> http://gerrithub.io/ (although note that this particular free service 
>>>>> uses GitHub as the "back-end"). 
>>>> 
>>>> Github -> github.com Gitlab -> gitlab.com Gerrit -> many alternatives. 
>>>> I.e. not central. 
>>> 
>>> Why is that a problem though? 
>> 
>> Not a problem, but a barrier to entry. It typically means yet another 
>> sign-up and yet another account to manage. 
>
> So do GitHub and GitLab. 
>> I can expect most open-source interested developers to have a GitHub 
>> account, and many have a Gitlab account. 
>
> Yes, but sorry, that doesn't make the cost of signing up to GitHub / GitLab 
> magically vanish to zero; it just means that cost was already spent.  Given 
> that we're only talking about one-off costs here, I don't see how it's fair 
> to claim that they have any significant advantage.  If every Gerrit 
> repository ran on a different Gerrit instance then yes, sign-up would be an 
> O(n) cost with respect to the number of repositories a developer is involved 
> in, but that's just not representative of reality.  An unusually heavy user 
> of Gerrit would still be unlikely to have to sign up more than about 5 
> instances, e.g. an extreme case might be: 
> - gitlab.com
> - Their company's private Gerrit instance
> - Android Gerrit
> - Eclipse Gerrit
> - OpenStack Gerrit
>
> If each sign-up takes 5 minutes, that's 25 minutes cost over say, 10 years 
> (e.g. until the next git or Gerrit comes along), so it's basically a 
> negligible cost. 
> Perhaps we're having this debate because I'm thinking of learning curves / 
> start-up in terms of time, whereas you're seeing it more as the challenge 
> people tend to face in overcoming mental inertia.  I think the latter is a 
> perfectly valid point - everyone suffers from mental inertia, including me - 
> but as long as there are sufficient resources provided to help overcome that, 
> such as quickstart guides, cheatsheets etc., I tend to have much less 
> sympathy for that situation.  That's why I'm working on a quickstart guide 
> for GitHub users who are coming to OpenStack's Gerrit.

Mental intertia, yes. But also more accounts is a drag on quality of life.
Tragic fact. :-) A central open-source gerrit instance with an integrated
bug tracker would definitely make people happier. GerritHub is getting close,
but I would really like to see something decoupled from GitHub.

It's quite sad, but this is where GitHub wins a lot of credit for the network
effects of having users already signed-up.

>>>>>> - High threshold for new contributors (unusual workflow, hooks needed. 
>>>>>> ) 
>>>>> 
>>>>> I keep hearing this and I just don't understand it, sorry.  What's 
>>>>> unusual about the workflow?  You create a branch, commit to it, and then 
>>>>> type "git review".  Could it really be much simpler? And Gerrit requires 
>>>>> about the same amount of setup as GitHub.  I already addressed the hooks 
>>>>> comment above.  I think most people forget that GitHub requires some 
>>>>> setup because they already did it so many years ago.  If you have 
>>>>> specific thoughts on this I'd love to hear them so I can understand your 
>>>>> viewpoint better, because like I said you are not the only person I've 
>>>>> heard this perspective from. 
>>>> 
>>>> Again, not opinion. Painful experience. What you outline is the trivial 
>>>> use-case, and using an additional tool "git review". The more advanced 
>>>> use-cases are things like responding to a review with new commits 
>>> 
>>> With Gerrit:  1. Add / amend commits 
>>>  2. Run "git review" 
>>> With GitHub or GitLab:  1. Add / amend commits 
>>>  2. Run "git push", optionally with --force-with-lease, which requires     
>>> a fairly deep understanding of the implications of rewriting git     
>>> history.  This is a sufficiently complicated topic that I've     blogged 
>>> about it twice to try to help reduce confusion: 
>> 
>> No need to amend force push. Developers who are getting started with git 
>> and are really fresh can push new commits to the same branch. These can be 
>> squashed into a single commit at merge time to keep history clean, with a 
>> "squash and merge" button. It is of course better to do any 
>> squashing/amending locally, but users have different skill levels and 
>> pushing them doesn't always lead to good results. 
>
> Exactly.  So it seems we agree this is an area where GitHub's and GitLab's 
> workflows make things unnecessarily complicated for less experienced users.

No, not really. More steps yes, but the inexperienced users tend to not
mind more steps as long as they understand what's going on.

> In contrast Gerrit side-steps this complexity by preserving the 1:1 mapping 
> between commits and reviews.

Users didn't seem to experience it this way. Also note that if you push multiple
commits in gerrit these create multiple reviews, which tends to confuse beginners
who are used to fixing their bugs by pushing more commits onto the same branch,
as done in most SCMs that don't let you rewrite history.


>>> https://blog.adamspiers.org/2015/03/24/why-and-how-to-correctly-amend-github-pull-requests/
>>>        https://blog.adamspiers.org/2017/08/16/squash-merging-and-other-problems-with-github/ 
>>> 
>>>> or a complete re-write of the contribution. 
>>> 
>>> With Gerrit:  1. Create new branch and commit the rewrite to it 
>>>  2. Run "git review" 
>>>  3. Abandon the old review with a comment linking to the new one. 
>>> With GitHub or GitLab:  1. Create new branch and commit the rewrite to it 
>>>  2. Go through the 6 steps I listed above for submitting a new     
>>> pull/merge request 
>>>  3. Abandon the old review with a comment linking to the new one. 
>> 
>> Most users don't see it this way. They will run "git reset 
>> <starting-commit> --hard" to get back to the original state, then create 
>> new commits and force push. No new review necessary. 
>
> Yes, that may well be true.  Either way, Gerrit's workflow is simpler, since 
> no decision is required about whether to force push; you just run "git 
> review" and it always does the right thing.

Does it? I'll need to test this. I think if you reset and lose the
Change-Id's gerrit will create a new review.


>>>> I spent a lot more time supporting users on Gerrit when doing these 
>>>> things than I did Gitlab users. 
>>> 
>>> I don't dispute that.  But I'd love to hear more about the concrete 
>>> reasons why. 
>>>> Anecdotes on offer include things like "I re-cloned the the repo and my 
>>>> branch isn't there even though I pushed it". 
>>> 
>>> Yeah, that's a very good point that users who are used to pushing their 
>>> changes somewhere central would need to relearn the new model. And thanks 
>>> - that's a useful point I can capture in the document I'm working on. 
>>> But I still disagree that makes Gerrit inherently more complicated; it's 
>>> just *different*.  In fact the number of steps in the processes listed 
>>> above show quite clearly that its lack of requirement to first push 
>>> changes somewhere central significantly simplifies usage. 
>> 
>> Gerrit is more complex. If you want to think about it another way you can 
>> consider that github and gitlab both mantain a 1-1 relationship between 
>> local branches and branches on the server. 
>
> No, the *user* maintains those relationships, and this requires extra manual 
> work; not just when submitting reviews, but with the cleanup afterwards too.

Not sure I'm following. If I do "git fetch" from a remote I get all the
branches from the remote, including the ones that are in review under
<remote-name>/<branch> . Doing the same with a gerrit server doesn't fetch
all the branches (references) in review. Explaining git to people with
remotes and how remote branches are mapped to local branches with a prefix
takes some time. Explaining that the remote change branches (references)
are "magical" and aren't fetched like the other branches made a lot of
developers scratch their heads.

>> Gerrit breaks this with the review references. This makes it more complex 
>> for the user. Of course, one can blindly tell the user to use extra tooling 
>> and not think about how it works, but that isn't really constructive. 
>
> Why isn't that constructive?  I'm a seasoned Gerrit user and I've hardly 
> *ever* had to be more than dimly aware of refs/changes/ because git-review 
> takes care of every single use case I rely on.

But you are aware! :-) And you definitely sound like you aren't using
a graphical IDE in your day-to-day work (hope I haven't read you wrong)
which means that you can safely use "git review" for all your work.
People using graphical IDEs who need to set up a specific review remote
(one per target branch) find this a lot harder.

>>>> You then need to explain about refs/changes/... and that they need to 
>>>> find their review there and create a branch again from the correct 
>>>> change, which they first need to fetch. 
>>> 
>>> There's no need to explain about refs/changes/... at all; just point them 
>>> at git review -d:  -d CHANGE, --download CHANGE       Download the 
>>> contents of an existing gerrit review into a branch 
>>> Another simple one-liner. 
>> 
>> I generally follow the philosophy of explaining things to people. Ignorance 
>> is seldom bliss for programmers. 
>
> In general I agree, but being aware of the existence of refs/changes/ is 
> different and much more easily achievable than having the exact details fresh 
> at the top of your head for daily use.  The latter is simply unnecessary 
> overkill.

Maybe, but my experience tells me that a lot of people have a hard time
accepting the "Don't worry about the details" philosophy.

>>>> Gerrit is a fantastic tool, but it isn't for everyone is my conclusion. 
>>> 
>>> Is *any* tool for everyone? ;-) 
>> 
>> No, some people were quite happy in Perforce and ClearCase. Having only a 
>> single source of truth as in ClearCase appeals to a lot of people.
>
> Right ;-)
>
>> For a much less kind view of Gerrit you can see this article on lwn.net : 
>> https://lwn.net/Articles/702177/ . 
>
> Yeah, I was already aware of Greg K-H's dislike of Gerrit.  It's hard to miss 
> his rants when you're a SUSE employee ;-) 
> However that was from 2016 and the majority of his points in 
> https://www.youtube.com/watch?v=L8OOzaqS37s are hopelessly out of date.
>
> I know that you already fully appreciate the value of Gerrit, so forgive me 
> for preaching to the choir, but there are apparently readers on this list who 
> do not have experience with Gerrit, so hopefully the following is useful to 
> them. 
> Claim #1: Some Google employees hate Gerrit
>  Current reality: Google is heavily investing in Gerrit, are by far
>  the biggest contributors to it, and have been responsible for the
>  recent and ongoing quantum leaps forward in usability.  I know this
>  because I met the whole team at the Gerrit User Summit in London,
>  September 2017, saw several presentations on the improvements they
>  have already made and are still working on, and talked at length
>  with their UX expert who has been allocated to this project.
>
> Claim #2: Patch submission is hard
>  Reality: well, we've discussed this to death in this thread :-)  As
>  long as you accept the relatively tiny one-off up-front cost of
>  account sign-up and installing "git review", it's a trivial
>  one-liner.
>
> Claim #3: Managing patch series is hard
>  Reality: These days Gerrit automatically shows when reviews are
>  dependent on or conflict with other reviews.  Additionally there's
>  a new hashtags feature for more flexible grouping of reviews:
>
>    https://gerrit-review.googlesource.com/Documentation/intro-user.html#hashtags
>
> Claim #4: When you look at a patch which touches multiple files, you have to 
> scroll through multiple pages
>
>  Current reality: This is no longer true in the rewritten UI
>  (PolyGerrit).
>
> Claim #5: Local testing is slow
>  Reality: "git review -d $changeid" is easy and quick.
>
> Claim #6: If you update a patch series you get new ids
>  Reality: I have no idea what he was thinking when he said this.
>  The Change-Id in the commit message does not change unless you
>  decide to change it.  If he meant the git patch-id, then yes that
>  will change just like it would with every single other code review
>  system.
>
> Claim #7: Discussion is in a web UI which is separate from your development 
> team.
>  In the talk, Greg said "This forces your maintainers who are
>  assigned to a project to do all the work.  You do not get any
>  drive-by reviews.  Nobody on your team who isn't their job to do
>  reviewing will *ever* review a Gerrit patch.  [...]  You have to go
>  to an external site and search for what you want to review - it
>  doesn't come to you.  The discussion's not there.  And even on when
>  you do review it and you do make comments on it, it's local between
>  you and that developer only.  The rest of your team doesn't see
>  those comments."
>
>  Reality: Gerrit (and GitHub and GitLab) all provide bi-directional
>  gateways with email, allowing the user to choose exactly which
>  events they want to subscribe to via email notifications, and to
>  even reply by email.  Accomplishing the same via email filters is
>  tricky enough that if you subscribe to mailing lists like kernel or
>  git, you will typically end up drinking from a firehose of patches.
>  Additionally, empirical evidence shows that drive-by reviews *do*
>  happen.
>
> The fact that he can't think of any pros of Gerrit (other than
> "project managers like it because it gives them a sense that they know
> what is going on") doesn't mean they don't exist.  It just means he's
> either unaware of them or doesn't appreciate them.  The above
> statement is true for normal developers, not just project managers.
>
> But a kernel developer is very far from a typical modern developer
> anyway.  And when you've spent one or more decades reviewing patches
> on mailing lists, I can understand why you'd have mental inertia /
> natural resistance to switching to something else.
>
> I've submitted various series of patches to git, which has the same
> code review workflow as the kernel, and I found it really painful
> compared to modern review systems.  One big reason for that was that
> the status of any given patch series is not clearly visible anywhere,
> you're heavily reliant on regular updates from the maintainer in order
> to understand the latest status of both a specific patch and of all
> ongoing patches at an overview level.  That's an unpleasant and
> unnecessary bottleneck.  And since Greg is (or at least was) a
> lieutenant, he's effectively a maintainer who spends much more time
> receiving patches than sending them, therefore he's much less likely
> to be aware of this pain.  He even admits this problem:
>
>   "I had 85 outstanding patches that nobody was reviewing for a repo
>   and everybody thought everything was fine."
>
> OK, that's far too much time spent on this thread.  We're basically on
> the same page so I suggest we both return to more productive work ;-)


Lets. :-) Nice chatting, hope I gave some insight into what problems
I've faced, but this is hard stuff to argue because I myself don't
experience the problems, I just experience other people experiencing
them. :-)

Cheers,
Nils

> _______________________________________________
> Developers mailing list
> Developers at clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/developers
>