Little tutorial for JS-Team beginners.
The tools
To package node/javascript modules, you need at least:
devscripts: scripts to make the life of a Debian Package maintainer easier
pkg-js-tools: collection of tools to aid packaging Node modules in Debian
git-buildpackage: suite to help with Debian packages in Git repositories (or git-dpm)
npm2deb: tool to help debianize Node.js modules (version ≥ 0.3.0 is recommended)
schroot: securely enter a chroot environment
lintian: static analysis tool for Debian packages
duck: the Debian URL checker
Little howtos
Little schroot/sbuild howto
1. To build a minimal Debian unstable environment, you can use this:
$ sudo sbuild-createchroot unstable /srv/chroot/unstable-amd64-sbuild # or to use ccache and eatmydata: $ sudo sbuild-createchroot --include=eatmydata,ccache unstable /srv/chroot/unstable-amd64-sbuild
2. To update it:
$ sudo sbuild-update unstable-amd64-sbuild -u -g -a -r
3. List built schroot
$ sudo schroot --list
Schroot improvements
1. Save bandwith using apt-cacher-ng
2. Use tmpfs to increase build speed
Little salsa howto
You need to generate a Personal Access Token in Salsa settings. Select all checkboxes in section Select scopes. Copy the tokens value into ~/.devscripts:
SALSA_TOKEN=xxxxxxxxxxx
Verify that it works:
$ salsa whoami Id : 1234 Username: myaccount-guest Name : My Name Email : my@mail.org State : active
You must also be member of js-team group. Verify this:
$ salsa list_groups Id : 2099 Name : Debian JavaScript Maintainers Full path: js-team
You must also set your public OpenPGP key in your Salsa account.
Packaging walkthrough
Working with existing packages
1. Clone locally the package:
$ salsa --group js-team co node-temporary $ cd node-temporary
Of course you need to configure salsa(1) with a valid GitLab token.
2. Verify debian/gbp.conf to not import upstream .gitignore. It may contain:
3. Verify that debian/watch is compliant with npmjs registry:
$ debcheck-node-repo && echo OK
If result is bad, investigate manually. If upstream forgot to insert tags in its repository, please open an issue: it is better to use source repository than npm registry (which may contain some generated files)
4. Update the package:
$ uscan ... $ gbp import-orig --pristine-tar ../node-temporary_9.9.9.orig.tar.xz
5. Fix debian/* files, then push the result:
$ git push origin master upstream pristine-tar $ git push --tags
Always visit Debian Tracker and check work to do for this package.
Switch test and install to pkg-js-tools
pkg-js-tools provides hooks for auto-configuration, auto-test and auto-install and also autopkgtest.
To enable it:
- debian/control:
add Testsuite: autopkgtest-pkg-nodejs header
add dh-sequence-nodejs in build dependencies
remove debian/install or adapt it if your source package provides more than one binary package (see pkg-js-tools doc)
move build commands from debian/rules (override_dh_auto_build) to debian/nodejs/build (especially if you want to build components using debian/nodejs/<component>/build)
move test from debian/rules (override_dh_auto_test) to debian/tests/pkg-js/test
remove useless files in debian/tests (control, require,...)
Write good tests
Using pkg-js-tools, you just have to examine package.json (scripts -> test) to find upstream test and write it in debian/tests/pkg-js/test. Example:
$ cat debian/tests/pkg-js/test mocha -R spec --timeout 10000 test/
See pkg-js-tools full example for more.
If upstream test is launched using mocha, don't hesitate to increase timeout (default is 2000 which is often too short for Debian-CI machines).
If debian/tests/pkg-js/test does not exist, pkg-js-tools will automatically add a minimal test (node require(".")).
Don't forget to remove debian/tests/control and debian/tests/require if exists.
Notes on `jest`
jest is a powerful test framework. When using it in Debian packaging test, be careful to
use dh-sequence-nodejs/pkg-js-tools ≥ 50 (this installs build-files in autopkgtest test tree)
use jest --ci to avoid writing snapshots
use jest --ci -u when upstream includes test snapshots in its source tree. This will avoid false-positive autopkgtest regressions if build dependencies are upgraded
If you have embedded components and both uses jest, avoid conflicts by limiting the main jest using a regexp argument like jest --ci test/*.js.
you'll often need a debian/tests/pkg-js/files to fix missing files in autopkgtest (missing src/). Don't forget then to include all needed test files. Common example:
babel.config.js src test
Notes on `ava`
ava is not available in Debian, but there are some workarounds:
- for simple test files, use node-tape
for more complex files, use jest (via jest-codemods)
Ava to tape
## ORIGINAL TEST ### TAPE TEST ##
#
import test from 'ava' # const test = require('tape')
import foo from '.' # const foo = require('.')
#
test('bar', t => { # test('bar', t=> {
t.is( foo(0), 0) # t.is( foo(0), 0)
} # t.end()
# }Just to transform in commonjs and add a t.end() at each test; and adapt some function names. For example, t.notThrows becomes t.doesNotThrow. See /usr/share/doc/node-tape/readme.markdown.gz to find them.
Ava to jest
Launch jest-codemods:
$ npx jest-codemods ? Which parser do you want to use? Babel ? Which test library would you like to migrate from? AVA ? Are you using the global object for assertions? No ? Will you be using Jest on Node.js as your test runner? Yes ? On which files or directory should the codemods be applied? test.js Executing command: jscodeshift -t [...] test.js --ignore-pattern node_modules --parser babel Processing 1 files... Spawning 1 workers... Sending 1 files to free worker... All done. Results: 0 errors 0 unmodified 0 skipped 1 ok
- Test the result:
$ jest --ci --testRegex test.js
It may need a babel file. If so, add a babel.config.json:
{
"presets": [ "@babel/preset-env" ],
"plugins": [ "@babel/plugin-transform-runtime" ]
}Anyway, jest has not exactly the same features than ava. Especially some throwsAsync / notThrowsAsync test should be removed or modified.
Fully test your package
Dependencies test
Launch test in a minimal and up-to-date unstable environment to verify that all needed packages are declared in debian/control:
sbuild -j5 --no-apt-update -d unstable --no-clean-source --run-lintian --lintian-opts='--color always --display-info --display-experimental --pedantic'
or using gbp:
$ gbp buildpackage --git-ignore-branch --git-builder="sbuild -j5 --no-apt-update -d unstable --no-clean-source --run-lintian --lintian-opts='--color always --display-info --display-experimental --pedantic'" --git-export=WC
This also launches lintian. Check the result and fix all errors/warnings (or override them only if this is a false positive) and take care of other lintian checks.
Result test
pkg-js-tools helps to write good autopkgtest. If you write them yourself, remember that you must test installed files, not source ones. Launch autopkgtest (example with schroot - where ../foo.changes is the package being tested):
$ autopkgtest -B ../foo.changes -- schroot unstable-amd64-sbuild
Notes:
In the event when an error like this is (autopkgtest-schroot requires ephemeral schroot sessions. Set a "union-type" or use a tarball schroot) is thrown after running this command. Edit with your favorite text-editor /etc/schroot/chroot.d/unstable-amd64-sbuild-<randomly-generated> file by adding this options to fix it:
union-type=overlay union-overlay-directory=/dev/shm
and check result. By default, pkg-js-autopkgtest copies only test* files. If some other are needed, creates debian/tests/pkg-js/files and list all needed test files. Example:
$ cat debian/tests/pkg-js/files Makefile test.js
If the test depends on a new package, where ../packages/bar.deb is the test dependency, pass the local .deb dependency as follows:
$ autopkgtest -B '../packages/bar.deb' '../foo.deb' -- schroot unstable-amd64-sbuild
You can also run autopkgtest with sbuild by adding the following lines to ~/.sbuildrc :
URLs test
Simply launch duck in the Debian source directory and check the result.
Clean test
Launch 2 times dpkg-buildpackage -us -uc. If build files are not cleaned, the second will fail. If so, set them in a debian/clean. Example:
$ cat debian/clean build/ generated.min.js
Starting a new package
First you may check if needed dependencies exists. Example to build ldapjs:
$ pkgjs-depends ldapjs
# ldapjs@2.3.1
DEPENDENCIES:
node-asn1 (asn1)
node-assert-plus (assert-plus)
node-once (once)
node-vasync (vasync)
node-verror (verror)
MISSING:
ldapjs
└── abstract-logging (2.0.1)
└── backoff (2.5.0)
└── precond (0.2.3)
└── ldap-filter (0.3.3)Here, we can see that 4 modules are missing in Debian to package it: abstract-logging, backoff, ldap-filter and precond. A quick analysis shows that at least ldap-filter could be embedded.
This package can be built in Debian. Let's do it:
$ npm2deb create ldapjs ...
- Observe the list of FIX_ME lines printed at the end of the output.
- Save these lines to a temporary file since npm2deb will not generate them again.
- Fix all the FIX_MEs in the debian directory.
Once you are done with your changes, run debuild to refresh the generated files
- Run lintian against the newly generated changes file and make sure that there are no errors.
When all is OK and package builds well and lintian is clean, create a git repo:
$ cd node-vasync-2.2.0 $ git init $ gbp import-dsc ../path/to/node-vasync_2.2.0-1.dsc
Then, you can push it using salsa(1) in the root of Debian source tree.
If your level access in js-team is only "developer", ask to a maintainer to create repo first, then no need to add --tagpending or --desc (added by maintainer using salsa create_repo --tagpending --desc node-<npm-name>).
$ salsa --group js-team --tagpending --desc push_repo .
If you want to push in your personal area, simply launch:
$ salsa --desc push_repo .
Debugging tips
Here're some useful things that will help you debug various things.
Running node -e "console.log('Hello')" will execute console.log('Hello') in node.
Running node -e "require('package')" will try to require package and will fail if package is not available.
Running node -e "console.log(require.resolve('package'))" (or simply nodepath package) will log the path to the resolved package.
Adding NODE_DEBUG=module to the above commands will show debug information about module lookup paths. For example NODE_DEBUG=module node -e "require('package')"
Helpful things to know
Latest versions of node have an "exports" field in package.json. If you're inside the source directory (for example, when running autopkgtest), self-referencing kicks in and exports field is used to resolve package names. Read about self-referencing https://nodejs.org/api/packages.html#packages_self_referencing_a_package_using_its_name
/var/lib/dpkg/status has the control file information of all installed packages. apt show package can be used to view this information only for one package.
dpkg -S /path/to/file tells you which package the file at /path/to/file belongs to
pkg-js-tools is involved a lot. The documentation is at https://salsa.debian.org/js-team/pkg-js-tools/-/tree/master/doc/tools and https://salsa.debian.org/js-team/pkg-js-tools/-/tree/master/doc/autopkgtest
- The "blame" feature on github or git can be used to find which commit introduced a particular line (in package.json, yarn.lock, etc). This might sometimes give clues on why a package suddenly stopped working.
When a package builds its own typescript definitions (*.d.ts files), dh-nodejs will install them, as long as they are declared in package.json. They won't go to /usr/share/nodejs/@types/thatmodule/, and instead are kept inside /usr/share/nodejs/thatmodule/, and that's perfectly fine.