How I Escaped Refactoring Hell with Dynamic Imports (NodeJS x NPM)

Big Ideas:

• Legacy code may need refactoring as dependencies start leveraging ES Modules

• Avoid changing an entire application over to ES Modules w/ Dynamic Imports

• Dynamic Imports can reduce code refactor size dramatically

The Story:

Let's set the scene. There's a dependency vulnerability with a request library we're using. My co-worker sees that we need to bump up our request library to the next major version to resolve it; he's concerned about the breaking changes, so I hop in to help.

First, I check out the dependency's Github page to see the breaking changes. The newer version is now an ES module and must be imported; problem is, our application was written using CommonJS modules. If you're not familiar, here are the standard ways of using dependencies in CommonJS vs ES Modules.

//CommonJS modules
const commonJSModule = require('./path');

//ES Modules
import esModule from './path';

The problem comes when trying to import an ES Module into a CommonJS module. The "import" keyword isn't supported in CommonJS modules. This is because ES Modules are loaded asynchronously, and CommonJS Modules load synchronously. It would seem that our only option is to either..

A: Refactor the entire codebase to use ES Modules instead of CommonJS Modules to accommodate this one critical dependency

B: Find a comparable request library, and replace our current library ('got') with this new one

Both options would bring about risks of introducing new bugs. Option A requires a lot of code changes across many files in particular. With this in mind, things weren't looking so good.

The Solution: Dynamic Imports

After a bit of googling, I ran across dynamic imports. In later versions (+13 I believe), Node is equipped with import(), a function used in CommonJS modules. Simply put, it returns a promise that resolves with the module you specify.

//Sample Legacy Code
const library = require('library');
function doSomething(param, callback){
    return library.action(param, callback);
}

//Refactored w/ ES Dynamic Import
const libPromise = import('library');
async function doSomething(param, callback){
    const library = await libPromise;
    return library.action(param, callback);
}

Simple enough, right? Well, our legacy code was a bit trickier. We were working with two files. Like most request libraries, 'got' lets you add special options to configure its behavior. The first file we're working with - let's call it http.js- configures our 'got' object with some special functions. Something like this:

const got = require('got');
const specialFunction = (params) => {
    //do stuff
}
const otherFunction = (params) => {
    //do other stuff
}
const options = { special: specialFunction, other: otherFunction }
module.exports = got.extend(options);

We couldn't refactor by simply using an "async await" pattern as before because of the export. You can't call await as a top level call in a CommonJS file, and there's no way to await the library and then export it afterward.

//This won't work
const gotLoader = import('got');
const got = await gotLoader; 
//Error -> can't use await outside of an async function

//Neither will this
const gotLoader = import('got');
const loader = async()=> {
    const options = { special: specialFunction, other: otherFunction }
    const got = await gotLoader;
    module.export = got.extend(options);
    //When executing, the file that uses this export will
    //access an empty export object {}
}

My co-worker, Roberto Rubet, had a great idea; rather than exporting 'got' itself, we should export a function for getting 'got'.

//Use import vs require
const gotLoader = import('got');
const specialFunction = (params) => {
    //do stuff
}
const otherFunction = (params) => {
    //do stuff
}

//Exporting the async function means the file that uses this export can get 'got' via an await
module.exports = (async function(){
    const options = { special: specialFunction, other: otherFunction }
    const {default: got} = await gotLoader; 
    return got.extend(options);
})().catch(err => { console.log(err); throw err })

With this improvement, our second file that uses this http.js export can simply use an await to get 'got'. This way we could keep all of our code grouped into the original separated files.

const gotESM = require('./https.js');
const doSomething = async() => {
    const got = await gotESM;
    //do stuff w/ got
}

But even with all this in place, there was still a fair amount of refactoring to do. The files that used 'http.js' defines a decent number of functions that all use 'got'. The legacy code needed some tweaking to utilize the new import scheme. Here's a sample of what we were working with:

const got = require('./path-to-module'); 

const funcA = (params) => {
    return got.get(params);
}

const funcB = (param1, param2) => {
    return got.post(param1, param2);
}

const funcC = (params) => {
    //do stuff
    return got.get(params);
}

const funcD = (param1, param2) => {
    //do stuff
    return got.post(param1, param2);
}      


//..and so on

module.exports = { funcA, funcB, funcC, funcD /*, ..and so on */ }

To prevent having to redefine all of our functions, I turned the legacy 'got' function calls into a wrapper function that would await got first:

//Remeber, gotESM = the async function exported from http.js
const gotESM = require('./http.js'); 

/*
This redefines 'got.get' and 'got.post' as a Wrapper function.
*/
const got = {
    get: async function(path){ 
        //Becuase this is async, we can await our export from http.js
        //Once we have 'got' (defined below as g), we can use it's         
        //built in 'get' function
        let g = await gotESM; return g.get(path);
    },
    post: async function(path, options){ 
        let g = await gotESM; return g.post(path, options);
    }
}

//No need to modify funcA,B,C,D because of our new wrapper function
//End of Refactor
const funcA = (params) => {
    return got.get(params);
}

const funcB = (param1, param2) => {
    return got.post(param1, param2);
}

const funcC = (params) => {
    //do stuff
    return got.get(params);
}

const funcD = (param1, param2) => {
    //do stuff
    return got.post(param1, param2);
}      

module.exports = { funcA, funcB, funcC, funcD }

Note: funcA through funcD returns promises, and the code that calls these are expecting promises. If they didn't, then we'd need to do a little more refactoring.

Conclusion:

Through dynamic imports, we were able to avoid changing over all of our code to ES Modules, or finding a replacement library. All this, at the price of merely a few lines of code!

S/O to Roberto Rubet - it's always fun pair programming with you!