How to De-Risk Patching Third Party Software Packages

Originally published by Vanta.

There are several steps your organization must take to protect itself from potentially exploitable packages. First, you’ll need to carefully review and triage the package vulnerabilities that present risk to your organization, then you’ll need to patch each one. Patching a package may sound easy, but doing so without breaking your product can be tricky.

‍Before patching, you may review the changelog between versions. Opening the changelog, however, could further the patch dread. What if you miss a detail in the changelog out of the dozens of versions in between the version you use and the version that you need for the patch? What if the package maintainer did not mention some change to the package that you happen to depend on?

https://xkcd.com/1172/

Below are some tricks we use at Vanta to de-risk some of our npm package patches for our own software. We will be using Node and Yarn, but these principles should apply to other frameworks as well.

‍For example, let us say you want to patch the jsonwebtoken npm package from version 8.5.1 to 9.0.0 to mitigate a vulnerability at that version. Not patching carries security risks, but patching could carry some reliability risks as well. Could we inadvertently break all current user sessions with the changes in .verify()? Will the sign() function no longer be compatible with our secret key format?

‍Tip: Using JWTs for authentication has its own footguns. When possible, consider using a randomly generated Session ID instead.

‍

#1 Install both versions of the package at once

Some package managers allow you to create an alias for a particular version of the package. For example, in Yarn, you can alias a package name so that you can have individual package versions on their own node module resolution paths. Let us add jsonwebtoken version 9.0.0 without affecting our old installation:
‍

$ yarn add jsonwebtoken-9-0-0@npm:[email protected]

We can now reference both packages as follows:

const oldJwt = require("jsonwebtoken");
const newJwt = require("jsonwebtoken-9-0-0");

With both package versions installed, you can use application code to control which version you want to use.

‍

#2 Require all new code to use the new version

When working on a large codebase, there may be many developers using the same package and introducing new usages of that package. To save time on the migration, you may want to add guardrails for developers to use the new version going forward. This can be accomplished with static analysis, lint rules, or CODEOWNER review. There may be cases where this approach is not worthwhile nor practical: for example, if usages of the two versions interact with each other, there may be compatibility challenges.

‍

#3 Blue green deploy if applicable

There is some art to deciding how to apply a blue green deployment to your package. For example, one could apply a dual-read approach, but that may be infeasible if each package invocation consumes a lot of resources. In our particular example, using a dual-read approach works well with jwt.verify(), but jwt.sign() might be trickier to encapsulate in a dual-read and/or dual-write approach since .sign() relies on randomness and the output might be consumed externally.

Zeroing in on the dual-read approach for jwt.verify(), we could wrap all of our existing usages with this wrapper:

‍const oldJwt = require("jsonwebtoken");
const newJwt = require("jsonwebtoken-9-0-0");
const fs = require("fs");
 const compareJwtVerify = (token) => {
  const cert = fs.readFileSync('public.pem');
  const oldResult = oldJwt.verify(token, cert);
  try {
    const newResult = newJwt.verify(token, cert);
    if (oldResult !== newResult) {
      // Reminder: When logging information, be mindful not to log sensitive
      //  information such as the token!
       console.log("JWT migration: oldResult and newResult are different",
         oldResult, newResult);
    }
  } catch (e) {
    console.log("JWT migration: newJwt.verify() gave an error", e);
  }
  return oldResult;
};

// old callsite: jwt.verify(token, cert); // new callsite: compareJwtVerify(token);

Assuming that running newJwt.verify() does not have negative side effects (we would be in trouble if .verify() decided to run process.exit(1), for example), we can have the benefit of trying out the new package version without affecting our existing production traffic.

After some time and when no more discrepancies are logged, we can then replace all calls to compareJwtVerify() with newJwt.verify().

If patching the package also requires code changes to the callsites, you could extend the wrapper to take functions as input:

const oldJwt = require("jsonwebtoken");
const newJwt = require("jsonwebtoken-9-0-0");
const fs = require("fs");
 const compareJwtVerify = (oldJwtFn, newJwtFn) => {
  const oldResult = oldJwtFn();
  try {
    const newResult = newJwtFn();
    if (oldResult !== newResult) {
      // Reminder: When logging information, be mindful not to log sensitive
      //   information such as the token!
       console.log("JWT migration: oldResult and newResult are different",
         oldResult, newResult);
    }
  } catch (e) {
    console.log("JWT migration: newJwt.verify() gave an error", e);
  }
  return oldResult;
};
// old callsite: jwt.verify(token, cert); 
// new callsite: compareJwtVerify(() => oldJwt.verify(token,cert),
// () => newJwt.verify(token, newCert))

Want to use a library for this? The Trello and Github teams have made packages that help with these code migrations. See Github’s blog post for more information about techniques to de-risk refactoring critical code.

‍

#4 Automate the code refactor via codemod scripts

Although wrapping usages of jsonwebtoken in a blue-green fashion may de-risk the rollout, this refactor may be time-intensive, especially if there are many call sites for the package at hand. Thankfully, we can write a codemod script to replace all existing usages with this wrapper.

We have used ts-morph to both migrate all existing call sites of a package to a dual-read wrapper and then clean-up existing call sites to use the new version once we’re confident that the dual read will have minimal impact on production traffic.

‍

#5 Control the rollout via a feature flag

You can also control the rollout via a feature flag. This way, you can introduce the new version gradually across your user base and revert the rollout without having to revert your code and wait for a redeployment. You may also identify a lower-risk user base to roll out to first.

‍

#6 Uninstall the old version

Once you’ve safely migrated all existing code paths to the new version, don’t forget to uninstall the old version and rename the alias!