Best practices for generating patches and interdiffs

By:

on

May 27, 2015

I've often been asked how I generate both patches and interdiffs at the same time, because the instructions on drupal.org currently detail the two processes separately, and different documentation pages give different instructions.

So, I thought I'd share the process that works for me, providing real-world examples from an issue that I've worked on.

If you find a better process, please blog about it and post a link in the comments!

This tutorial assumes that:

  • You know what patchfiles are,
  • You know how to use the command-line (the instructions should work in both *NIX and Windows), and,
  • You have a beginner-level knowledge about Git.

High-level workflow overview

My patching workflow works roughly like this:

  1. Before you make a patch, make an issue to post it to.
  2. Make your patch in an environment that makes as few assumptions as possible. I use a fresh Drupal site installed with the Minimal install profile, and I only install the bare minimum set of modules that I need to make the module I want to patch work.
  3. To make the first patch in an issue:
    1. Before making any changes, make a new branch named after the NID of the issue and the comment number you intend to post the patch to.
    2. Make your changes and commit them to the new branch. If you make more than one commit, squash them down to a single commit when you feel that you're ready to post the patch.
    3. Generate a patchfile with git format-patch — this will add extra metadata to help Git and other humans in the future.
  4. If there are already patches in the issue, and the most-recent patch no longer applies, re-roll the old patch first and upload it, then make your changes in a separate patch.
  5. If there are already patches in the issue:
    1. Download the most-recent patch and commit its changes to a branch named after the NID of the issue and the comment number that you got the patchfile from.
      • If it no longer applies, re-roll and upload it first, then make your changes in a separate patch.
    2. Checkout the starting branch again.
    3. Make a new branch named after the NID of the issue and the comment number you intend to post the patch to.
    4. Apply the most-recent patch to the new branch but don't commit it yet.
    5. Make your changes in the new branch and commit them to the new branch. If you make more than one commit, squash them down to a single commit when you feel that you're ready to post the patch.
    6. Generate a patchfile with git format-patch — this will add extra metadata to help Git and other humans in the future.
    7. Generate an interdiff by git diff-ing against the branch from step 5a.

... if none of this makes sense yet, don't worry: I'll explain it below. Hopefully once you've run through it a couple of times, you can use this high-level overview as a reference.

Detailed workflow

1. Making a new issue

Before you make a patch, you should have an issue to post it to.

  1. Create a ticket on Drupal.org, and propose your changes.
    • Use the Issue summary template.
      • If you're reporting a bug, add a section with Steps to Reproduce the bug. This will help module maintainers see the need for your patch!
      • Especially for bug reports, it's helpful to compare what the module does now with what you want it to do. This also helps module maintainers see the problem.
    • Good metadata is important! I usually need to spend a bit of time finding a succinct Title, choosing the most-appropriate Category and Component, re-reading the guidelines for setting a Priority, and choosing good issue tags.
      • Since this is a new issue, and you haven't attached a patch yet, leave the issue status set to Active.
      • If you intend to provide a patch right away, you may assign the issue to yourself.
    • Take note of the node ID in the URL of the issue (e.g.: for https://www.drupal.org/node/2290031 the Node ID is 2290031) — you'll need it when patching.

2. Set up your environment for patching

Accidentally making assumptions about the Drupal site you're working on increases the risk that your patch will break other people's work, won't get accepted, will take a long time to be accepted, or will be rolled back later on. So, it's important to write your patch on an environment that makes as few assumptions as possible.

In some cases, trying to reproduce a problem (or adding a feature) in a clean environment will also help you to identify future problems, or see a solution that doesn't require patching.

  1. Set up a fresh Drupal site to write your patch on.
    • Use the Minimal install profile.
  2. Enable only the bare minimum set of modules that you need to make the module you want to patch work (e.g.: the Facet API module requires ctools and a search module, like core's search).
  3. Clone the module you want to patch (e.g.: git clone --branch 7.x-1.x http://git.drupal.org/project/facetapi.git).
    • You can copy and paste the Git command from the project page's Version control tab (e.g.: https://www.drupal.org/project/facetapi/git-instructions).
    • Take note of the branch you start from.

3. Making the first patch in an issue

The first patch to an issue does not need an interdiff.txt, but generating your first patch as follows makes generating interdiffs easier in the future.

  1. From the starting branch, create a new branch named after the Node ID of the issue and the comment number you intend to post it to (e.g.: 2290031-3git checkout -b 2290031-3).
    • The Node ID is in the URL of the issue (e.g.: for https://www.drupal.org/node/2290031 the Node ID is 2290031).
    • Note that if you're logged into Drupal.org, the "Add comment" section title shows the next comment number (e.g. Add comment #10).
  2. Make your changes in this branch, add them, and commit them.
    • If you make more than one commit, you'll need to squash it down to a single commit using git-rebase.
    • The commit message you enter here will be seen by others, but it will not be used as the final commit message. I simply enter the name of the branch (e.g.: git commit -m "2290031-3").
  3. Run git format-patch $starting_branch (e.g.: git format-patch 7.x-1.x).
    • This generates a patch file and places it in the repository.
    • Alternately, you could generate a patch with git diff $starting_branch, but I like git-format-patch better because it adds a bit of metadata that will help Git to apply your patch without someone having to re-roll it as often.
  4. Move the patch file out of the git repository to prevent you accidentally committing it later.
    • This is a good time to rename your patch.
    • Note that if you use Dreditor, you can use it to generate a patch name for you by clicking the Patchname suggestion button in the Files fieldset.
  5. Check out the starting branch, to prevent you from accidentally branching off it later.
  6. Upload the patch to the issue.
    • Don't forget to un-assign the issue! Many maintainers ignore assigned issues, regardless of what the issue's Status is, because they assume assigned issues have someone actively working on it right now.
    • Don't forget to set the Status to Needs review. This helps others to know there's a patch attached and that patch is ready for someone else to look at.

4. Re-rolling a patch

If the maintainer(s) have been making a lot of commits to the starting branch, git(1) or patch(1) may not be able to figure out how to apply the patch file any longer. If this happens, the patch will need to be rerolled.

If you're making a subsequent patch to an issue, and the previous patch doesn't apply, re-roll the patch first, then make your changes in a separate patch. Doing so makes it easier for both you to generate interdiffs; and it makes it easier for the maintainer(s) (and anyone else following the issue) to understand what's going on.

  1. Make sure your repository is up-to-date (git pull).
  2. Download the most-recent patch.
  3. Double-check whether it is necessary to re-roll the patch (git apply --check /path/to/most-recent.patch).
    • If you see nothing, then the patch does not need to be re-rolled and you can skip this section entirely.
    • If you see error: patch failed, then you will need to re-roll the patch.
  4. Find when the most-recent patch was made.
    • If the most-recent patch was made with git-format-patch, the first line of the patchfile (From $commit_id $date) will tell you everything you need to know. Take note of the commit ID and jump down to step 6.
    • If the commit ID or date is not available in the patchfile, you can get a rough guess by looking at the date-stamp on the comment that the most-recent patch was uploaded in will work instead.
      • Sometimes Drupal.org displays the time since the comment was posted (e.g.: "yesterday"). Double-click to change to an exact date.
    • Take note of how to write the date in ISO 8601 format (yyyy-mm-dd, e.g.: 2014-06-20).
  5. Search git log to find a commit from that day or earlier (git log --before=$iso_8601_date e.g.: git log --before=2014-06-20).
    • Take note of the commit ID (e.g.: 9e26034949aac609bb8537ef5a189033a3665baa).
  6. Check out the commit ID (e.g.: git checkout 9e26034949aac609bb8537ef5a189033a3665baa).
    • Git will tell you that You are in 'detached HEAD' state. This is okay.
  7. Create a new branch named after the Node ID of the issue and the comment number you intend to post it to (e.g.: 2290031-6git checkout -b 2290031-6).
  8. Apply the patch (git apply --index /path/to/most-recent.patch e.g.: git apply --index make_inactive_active-2290031-4.patch).
  9. Commit the patch.
    • The commit message you enter here will be seen by others, but it will not be used as the final commit message. I simply enter the name of the branch (e.g.: git commit -m "2290031-6").
  10. Rebase the branch onto the most-recent commit of the starting branch: git rebase $starting_branch (e.g.: git rebase 7.x-1.x).
    • You might get merge conflicts. If you do, fix them, taking notes on what you did, and follow Git's instructions for finishing the rebase.
    • In rare cases, if you get merge conflicts, it's because the maintainer(s) significantly changed how the code works. If this is the case, you might have to abort the rebase and start the patch fresh (see 3. Making the first patch in an issue). Make sure to very clearly state in the comment where you post the patch that you tried re-rolling but the code had changed too much and you were forced to start from scratch.
  11. If and only if the rebase completed successfully, run git format-patch $starting_branch (e.g.: git format-patch 7.x-1.x).
    • This generates a patch file and places it in the repository.
    • Alternately, you could generate a patch with git diff $starting_branch, but I like git-format-patch better because it adds a bit of metadata that will help Git to apply your patch without someone having to re-roll it as often.
  12. Move the patch file out of the git repository to prevent you accidentally committing it later.
    • This is a good time to rename your patch.
    • Note that if you use Dreditor, you can use it to generate a patch name for you by clicking the Patchname suggestion button in the Files fieldset.
  13. Check out the starting branch, to prevent you from accidentally branching off it later (git checkout $starting_branch, e.g.: git checkout 7.x-1.x).
  14. Upload the patch to the issue.
    • Don't forget to un-assign the issue! Many maintainers ignore assigned issues, regardless of what the issue's Status is, because they assume assigned issues have someone actively working on it right now.
    • Don't forget to set the Status to Needs review. This helps others to know there's a patch attached and that patch is ready for someone else to look at.
    • Make sure you very clearly state that you are just re-rolling and there are no changes (or, if you got merge conflicts, state how you resolved them and why).

5. Making a subsequent patch in an issue

If there's already a patch in an issue, and you want to change it, you will need to generate an interdiff to help the maintainer(s) understand what you're changing.

  1. Make sure your repository is up-to-date (git pull).
  2. Download the most-recent patch.
  3. Create a new branch named after the Node ID of the issue and the comment number of the most-recent patch (e.g.: 2290031-6git checkout -b 2290031-6).
    • If you generated the last patch and the branch is still in your repository, you can skip down to step 7.
  4. Apply the patch (git apply --index /path/to/most-recent.patch e.g.: git apply --index make_inactive_active-2290031-6.patch).
  5. Commit the patch.
    • I simply enter the name of the branch (e.g.: git commit -m "2290031-6").
  6. Checkout the starting branch (git checkout $starting_branch, e.g.: git checkout 7.x-1.x).
  7. Create a new branch named after the Node ID of the issue and the comment number you intend to post it to (e.g.: 2290031-7git checkout -b 2290031-7).
  8. Apply the patch again (git apply --index /path/to/most-recent.patch e.g.: git apply --index make_inactive_active-2290031-6.patch).
    • If you want to take the patch in an entirely new direction, you can skip this step, but you should have a really good reason for throwing away other people's work (which they might have volunteered), and you should very clearly state those reasons in the comment you post in step 15. Make sure you follow the Drupal Code of Conduct.
  9. Make your changes in this branch, add them, and commit them.
    • If you make more than one commit, you'll need to squash it down to a single commit using git-rebase.
    • The commit message you enter here will be seen by others, but it will not be used as the final commit message. I simply enter the name of the branch (e.g.: git commit -m "2290031-7").
  10. Run git format-patch $starting_branch (e.g.: git format-patch 7.x-1.x).
    • This generates a patch file and places it in the repository.
    • Alternately, you could generate a patch with git diff $starting_branch, but I like git-format-patch better because it adds a bit of metadata that will help Git to apply your patch without someone having to re-roll it as often.
  11. Move the patch file out of the git repository to prevent you accidentally committing it later.
    • This is a good time to rename your patch.
    • Note that if you use Dreditor, you can use it to generate a patch name for you by clicking the Patchname suggestion button in the Files fieldset.
  12. Generate an interdiff (git diff $previous_patch_branch > interdiff.txt e.g.: git diff 2290031-6).
  13. Move the interdiff out of the git repository to prevent you accidentally committing it later.
    • Interdiffs are usually just named interdiff.txt, regardless of what issue number or comments they relate to, so take careful note of where you move it to so that you don't confuse it with another interdiff later.
  14. Check out the starting branch, to prevent you from accidentally branching off it later (git checkout $starting_branch, e.g.: git checkout 7.x-1.x).
  15. Upload both the patch and the interdiff to the issue.
    • Don't forget to un-assign the issue! Many maintainers ignore assigned issues, regardless of what the issue's Status is, because they assume assigned issues have someone actively working on it right now.
    • Don't forget to set the Status to Needs review. This helps others to know there's a patch attached and that patch is ready for someone else to look at.
    • Make sure you upload the correct interdiff! Interdiffs are usually just named interdiff.txt, regardless of what issue number or comments they relate to, so make sure you upload the right one!
  16. Delete the interdiff.txt so you don't get confused and upload an old interdiff later.

Conclusion

I'm posting this in the hopes that it will help others in the future. If you have questions, ask them in the comments and I'll try to answer them and update the blog post with corrections, so you can refer to this post in the future without digging through the comments as well.