[ClusterLabs Developers] [RFC] Time to migrate authoritative source forge elsewhere?

[Adam Spiers]

Nils Carlson <nils.carlson at ludd.ltu.se> wrote: 
>On 06/11/2018 08:31 PM, Adam Spiers wrote: 
>>Nils Carlson <nils.carlson at ludd.ltu.se> wrote: 
>>>On 06/11/2018 03:18 PM, Adam Spiers wrote: 
>>>>Nils Carlson <pyssling at ludd.ltu.se> wrote: 
>>>>>Gerrit is a much more powerful tool for code-review. The 
>>>>>workflow is less intuitive however and has a far higher 
>>>>>learning curve. 
>>>>
>>>>I disagree, but please can you clarify which version of Gerrit 
>>>>you are referring to?  The latest release (2.15) is an enormous 
>>>>leap forwards in usability: 
>>>
>>>Feel free to disagree, this isn't based on opinion on my part but 
>>>experience, I have migrated two companies (with around 20-30 
>>>developers) to gerrit and gitlab respectively and developers have 
>>>adjusted far faster to gitlab than to gerrit. The version was 
>>>pre-2.15, 2.13 I believe, but nothing revolutionary seems to have 
>>>happened (the same patch-set based workflow). 
>>
>>2.15 *is* revolutionary in terms of UX because the UI was rewritten 
>>from scratch (PolyGerrit).  True, the workflow is more or less the 
>>same in design, but how intuitive it will feel to newcomers (your 
>>original criticism above) depends heavily on the UI. 
>
>Maybe, but I doubt it would change much. People get over the bad UI 
>pretty quickly (weeks,) the workflow is the hard part. 

I'm still hoping to hear something which explains to me why the 
workflow is hard. 

>>>Personally I much prefer gerrit, but I have been using git since 
>>>2009 including kernel coding which puts higher requirements on 
>>>skill than most systems. 
>>
>>Yes, as an occasional git hacker I'm in a similar boat. 
>>>For your average developer Gerrit is a lot harder to adjust to it 
>>>seems.
>>
>>Yes, like I said I've heard this several times, but in the absence 
>>of concrete details I'm really struggling to understand how this 
>>could be more than natural inertia rather than genuinely significant 
>>practical hurdles. 
>>
>>However I'm not only willing to be proven wrong but actively seeking 
>>good explanations which *would* prove me wrong.  This is because 
>>I've already volunteered to write a document which helps people 
>>familiar with GitHub migrate to Gerrit (intended for the OpenStack 
>>community but of course it could be shared to a wider audience).  
>>And to do that well I really need help understanding what people 
>>struggle with when learning Gerrit the first time.  So if you can 
>>help with that I'd be really grateful, although don't worry if you 
>>don't have time :-) 
>>[snipped]
>>
>>>>but this is mostly an internal implementation detail; in 
>>>>practice everyone uses a tool such as "git review" which handles 
>>>>all that automatically behind scenes. 
>>>
>>>Yes, extra tooling is more or less the norm when using gerrit. 
>>
>>It's pretty common with GitHub too, I suspect.  But even if not, I 
>>don't see a one-off installation of a simple tool as a huge hurdle. 
>
>The difference is really on the IDE side. Eclipse has excellent 
>integration with gerrit these days, but most other IDEs do not. 
>Configuring your prefered IDE for Gerrit is not trivial in most cases. 
>
>I've seen specific tools for GitHub, but never seen anyone use them. 
>GitHub does not make use of specific review remotes as Gerrit does, so 
>there is no need for specific tooling or configuration. This is the 
>case with most of these tools, only Gerrit and Phabricator 
>require/encourage specific tooling. 

Yes, I accept that there is a need for most Gerrit users to install 
git-review.  But as I already said, this is a one-off cost which is 
low or even extremely low, and the pay-off is a workflow which is 
significantly simpler than GitHub or GitLab. 

>>Furthermore, without a helper tool or IDE which supports GitHub or 
>>GitLab, creating a pull request or merge request requires several 
>>more steps than with Gerrit: 
>>  0. Ensure that you have a remote fork to which you have push 
>>     access (e.g. by clicking the merge button if you don't 
>>     have push access to the target repository) 
>>
>>  1. Push your branch to a remote fork 
>>
>>  2. Locate that branch in the web UI 
>>
>>  3. Click the button(s) to bring up the window in which you 
>>     draft the pull/merge request 
>>
>>  4. Give the request a title and description 
>>
>>  5. Submit it 
>>
>>Compared to the single step "type git review", I'm somewhat 
>>mystified how anyone can claim that Gerrit is harder to get started 
>>with.
>
>As any programmer will tell you, complexity isn't measured in number 
>of steps. :-) 

Hrm, I've been programming for 36 years and don't recall ever hearing 
that.  Most likely I did hear it once or twice and forgot, but I don't 
think it's a common maxim.  A quick Google didn't reveal anything, at 
least.

>Each step above is trivial and some are in fact not 
>needed, and all are easy to explain. 

Agreed, but surely you aren't arguing that they are simpler than the 
single step "type git review"? 

>The above however isn't correct. For an open repo on Gitlab the steps 
>would be as follows: 
>
>1. Push your commit to a branch of the origin repo (assuming you are 
>part of the project, or have push access to a set of branches which 
>can be granted to any signed-in user), get a URL on the command line 
>for creating a merge-request. 

That's two or three steps.  They don't become a single step just 
because you put them under a single bullet item ;-) 

>2. Click the URL, review the contents of the merge-request (if you 
>pushed only a single commit Gitlab will take the title and comment 
>from that commit), click create at the bottom. 

Ditto here - multiple steps. 

The only difference I can see between that and what I wrote is that my 
step 2 can be circumvented by using the URL provided by the output of 
the CLI command.  That's a nice trick - thanks. 

>Sure, gerrit can do this in one step without opening the we browser, 
>but in general most people want to look at the review before forcing 
>others to do so. 

In Gerrit there is no difference between the review and the commit, 
therefore this can be accomplished by running "git show" before "git 
review".  Gerrit's 1:1 mapping between commits and reviews is a large 
part of why its workflow is so much simpler than GitHub's and 
GitLab's. 

>>>>>The negative sides of Gerrit typically outweigh the positive 
>>>>>for most organizations I'm afraid: - No central hosting like 
>>>>>gitlab.com. 
>>>>
>>>>That's not really true.  There are SaaS hosters of Gerrit, most 
>>>>notably GerritForge who also offer completely free hosting for 
>>>>FLOSS projects on http://gerrithub.io/ (although note that this 
>>>>particular free service uses GitHub as the "back-end"). 
>>>
>>>Github -> github.com Gitlab -> gitlab.com Gerrit -> many 
>>>alternatives. I.e. not central. 
>>
>>Why is that a problem though? 
>
>Not a problem, but a barrier to entry. It typically means yet another 
>sign-up and yet another account to manage. 

So do GitHub and GitLab. 

>I can expect most open-source interested developers to have a GitHub 
>account, and many have a Gitlab account. 

Yes, but sorry, that doesn't make the cost of signing up to GitHub / 
GitLab magically vanish to zero; it just means that cost was already 
spent.  Given that we're only talking about one-off costs here, I 
don't see how it's fair to claim that they have any significant 
advantage.  If every Gerrit repository ran on a different Gerrit 
instance then yes, sign-up would be an O(n) cost with respect to the 
number of repositories a developer is involved in, but that's just not 
representative of reality.  An unusually heavy user of Gerrit would 
still be unlikely to have to sign up more than about 5 instances, 
e.g. an extreme case might be: 

  - gitlab.com
  - Their company's private Gerrit instance
  - Android Gerrit
  - Eclipse Gerrit
  - OpenStack Gerrit

If each sign-up takes 5 minutes, that's 25 minutes cost over say, 10 
years (e.g. until the next git or Gerrit comes along), so it's 
basically a negligible cost. 

Perhaps we're having this debate because I'm thinking of learning 
curves / start-up in terms of time, whereas you're seeing it more as 
the challenge people tend to face in overcoming mental inertia.  I 
think the latter is a perfectly valid point - everyone suffers from 
mental inertia, including me - but as long as there are sufficient 
resources provided to help overcome that, such as quickstart guides, 
cheatsheets etc., I tend to have much less sympathy for that 
situation.  That's why I'm working on a quickstart guide for GitHub 
users who are coming to OpenStack's Gerrit. 

>>>>>- High threshold for new contributors (unusual workflow, hooks 
>>>>>needed. ) 
>>>>
>>>>I keep hearing this and I just don't understand it, sorry.  
>>>>What's unusual about the workflow?  You create a branch, commit 
>>>>to it, and then type "git review".  Could it really be much 
>>>>simpler? And Gerrit requires about the same amount of setup as 
>>>>GitHub.  I already addressed the hooks comment above.  I think 
>>>>most people forget that GitHub requires some setup because they 
>>>>already did it so many years ago.  If you have specific thoughts 
>>>>on this I'd love to hear them so I can understand your viewpoint 
>>>>better, because like I said you are not the only person I've 
>>>>heard this perspective from. 
>>>
>>>Again, not opinion. Painful experience. 
>>>What you outline is the trivial use-case, and using an additional 
>>>tool "git review". The more advanced use-cases are things like 
>>>responding to a review with new commits 
>>
>>With Gerrit: 
>>  1. Add / amend commits 
>>
>>  2. Run "git review" 
>>
>>With GitHub or GitLab: 
>>  1. Add / amend commits 
>>
>>  2. Run "git push", optionally with --force-with-lease, which requires 
>>     a fairly deep understanding of the implications of rewriting git 
>>     history.  This is a sufficiently complicated topic that I've 
>>     blogged about it twice to try to help reduce confusion: 
>
>No need to amend force push. Developers who are getting started with 
>git and are really fresh can push new commits to the same branch. 
>These can be squashed into a single commit at merge time to keep 
>history clean, with a "squash and merge" button. It is of course 
>better to do any squashing/amending locally, but users have different 
>skill levels and pushing them doesn't always lead to good results. 

Exactly.  So it seems we agree this is an area where GitHub's and 
GitLab's workflows make things unnecessarily complicated for less 
experienced users.  In contrast Gerrit side-steps this complexity 
by preserving the 1:1 mapping between commits and reviews. 

>>https://blog.adamspiers.org/2015/03/24/why-and-how-to-correctly-amend-github-pull-requests/ 
>>
>>        https://blog.adamspiers.org/2017/08/16/squash-merging-and-other-problems-with-github/ 
>>
>>
>>>or a complete re-write of the contribution. 
>>
>>With Gerrit: 
>>  1. Create new branch and commit the rewrite to it 
>>
>>  2. Run "git review" 
>>
>>  3. Abandon the old review with a comment linking to the new one. 
>>
>>With GitHub or GitLab: 
>>  1. Create new branch and commit the rewrite to it 
>>
>>  2. Go through the 6 steps I listed above for submitting a new 
>>     pull/merge request 
>>
>>  3. Abandon the old review with a comment linking to the new one. 
>
>Most users don't see it this way. They will run "git reset 
><starting-commit> --hard" to get back to the original state, then 
>create new commits and force push. No new review necessary. 

Yes, that may well be true.  Either way, Gerrit's workflow is simpler, 
since no decision is required about whether to force push; you just 
run "git review" and it always does the right thing. 

>>>I spent a lot more time supporting users on Gerrit when doing 
>>>these things than I did Gitlab users. 
>>
>>I don't dispute that.  But I'd love to hear more about the concrete 
>>reasons why. 
>>>Anecdotes on offer include things like "I re-cloned the the repo 
>>>and my branch isn't there even though I pushed it". 
>>
>>Yeah, that's a very good point that users who are used to pushing 
>>their changes somewhere central would need to relearn the new model. 
>>And thanks - that's a useful point I can capture in the document I'm 
>>working on. 
>>
>>But I still disagree that makes Gerrit inherently more complicated; 
>>it's just *different*.  In fact the number of steps in the processes 
>>listed above show quite clearly that its lack of requirement to 
>>first push changes somewhere central significantly simplifies usage. 
>
>Gerrit is more complex. If you want to think about it another way you 
>can consider that github and gitlab both mantain a 1-1 relationship 
>between local branches and branches on the server. 

No, the *user* maintains those relationships, and this requires extra 
manual work; not just when submitting reviews, but with the cleanup 
afterwards too. 

>Gerrit breaks this 
>with the review references. This makes it more complex for the user. 
>Of course, one can blindly tell the user to use extra tooling and not 
>think about how it works, but that isn't really constructive. 

Why isn't that constructive?  I'm a seasoned Gerrit user and I've 
hardly *ever* had to be more than dimly aware of refs/changes/ because 
git-review takes care of every single use case I rely on. 

>>>You then need to explain about refs/changes/... and that they need 
>>>to find their review there and create a branch again from the 
>>>correct change, which they first need to fetch. 
>>
>>There's no need to explain about refs/changes/... at all; just point 
>>them at git review -d: 
>>  -d CHANGE, --download CHANGE 
>>       Download the contents of an existing gerrit review into a branch 
>>
>>Another simple one-liner. 
>
>I generally follow the philosophy of explaining things to people. 
>Ignorance is seldom bliss for programmers. 

In general I agree, but being aware of the existence of refs/changes/ 
is different and much more easily achievable than having the exact 
details fresh at the top of your head for daily use.  The latter is 
simply unnecessary overkill. 

>>>Gerrit is a fantastic tool, but it isn't for everyone is my 
>>>conclusion. 
>>
>>Is *any* tool for everyone? ;-) 
>
>No, some people were quite happy in Perforce and ClearCase. Having 
>only a single source of truth as in ClearCase appeals to a lot of 
>people.

Right ;-)

>For a much less kind view of Gerrit you can see this article on 
>lwn.net : https://lwn.net/Articles/702177/ . 

Yeah, I was already aware of Greg K-H's dislike of Gerrit.  It's hard 
to miss his rants when you're a SUSE employee ;-) 

However that was from 2016 and the majority of his points in 
https://www.youtube.com/watch?v=L8OOzaqS37s are hopelessly out of 
date.

I know that you already fully appreciate the value of Gerrit, so 
forgive me for preaching to the choir, but there are apparently 
readers on this list who do not have experience with Gerrit, so 
hopefully the following is useful to them. 

Claim #1: Some Google employees hate Gerrit 

   Current reality: Google is heavily investing in Gerrit, are by far
   the biggest contributors to it, and have been responsible for the
   recent and ongoing quantum leaps forward in usability.  I know this
   because I met the whole team at the Gerrit User Summit in London,
   September 2017, saw several presentations on the improvements they
   have already made and are still working on, and talked at length
   with their UX expert who has been allocated to this project.

Claim #2: Patch submission is hard 

   Reality: well, we've discussed this to death in this thread :-)  As
   long as you accept the relatively tiny one-off up-front cost of
   account sign-up and installing "git review", it's a trivial
   one-liner.

Claim #3: Managing patch series is hard 

   Reality: These days Gerrit automatically shows when reviews are
   dependent on or conflict with other reviews.  Additionally there's
   a new hashtags feature for more flexible grouping of reviews:

     https://gerrit-review.googlesource.com/Documentation/intro-user.html#hashtags

Claim #4: When you look at a patch which touches multiple files, you have 
to scroll through multiple pages

   Current reality: This is no longer true in the rewritten UI
   (PolyGerrit).

Claim #5: Local testing is slow 

   Reality: "git review -d $changeid" is easy and quick.

Claim #6: If you update a patch series you get new ids 

   Reality: I have no idea what he was thinking when he said this.
   The Change-Id in the commit message does not change unless you
   decide to change it.  If he meant the git patch-id, then yes that
   will change just like it would with every single other code review
   system.

Claim #7: Discussion is in a web UI which is separate from your 
development team. 

   In the talk, Greg said "This forces your maintainers who are
   assigned to a project to do all the work.  You do not get any
   drive-by reviews.  Nobody on your team who isn't their job to do
   reviewing will *ever* review a Gerrit patch.  [...]  You have to go
   to an external site and search for what you want to review - it
   doesn't come to you.  The discussion's not there.  And even on when
   you do review it and you do make comments on it, it's local between
   you and that developer only.  The rest of your team doesn't see
   those comments."

   Reality: Gerrit (and GitHub and GitLab) all provide bi-directional
   gateways with email, allowing the user to choose exactly which
   events they want to subscribe to via email notifications, and to
   even reply by email.  Accomplishing the same via email filters is
   tricky enough that if you subscribe to mailing lists like kernel or
   git, you will typically end up drinking from a firehose of patches.
   Additionally, empirical evidence shows that drive-by reviews *do*
   happen.

The fact that he can't think of any pros of Gerrit (other than
"project managers like it because it gives them a sense that they know
what is going on") doesn't mean they don't exist.  It just means he's
either unaware of them or doesn't appreciate them.  The above
statement is true for normal developers, not just project managers.

But a kernel developer is very far from a typical modern developer
anyway.  And when you've spent one or more decades reviewing patches
on mailing lists, I can understand why you'd have mental inertia /
natural resistance to switching to something else.

I've submitted various series of patches to git, which has the same
code review workflow as the kernel, and I found it really painful
compared to modern review systems.  One big reason for that was that
the status of any given patch series is not clearly visible anywhere,
you're heavily reliant on regular updates from the maintainer in order
to understand the latest status of both a specific patch and of all
ongoing patches at an overview level.  That's an unpleasant and
unnecessary bottleneck.  And since Greg is (or at least was) a
lieutenant, he's effectively a maintainer who spends much more time
receiving patches than sending them, therefore he's much less likely
to be aware of this pain.  He even admits this problem:

    "I had 85 outstanding patches that nobody was reviewing for a repo
    and everybody thought everything was fine."

OK, that's far too much time spent on this thread.  We're basically on
the same page so I suggest we both return to more productive work ;-)