Adding a Dependency Based on Autotools to a Bazel Project

Motivation

In my spare time I have been experimenting with the Bazel build runner. I wanted to add Libsodium as a dependency to my Bazel C/C++ project. The Libsodium project is not a Bazel package and therefore Bazel will not know how to treat it as a dependency. If I wish to add Libsodium as a repository based dependency to my project, then It makes sense to write a BUILD file defining how Bazel should build the Libsodium library.

What you need to know

Rules_foreign_cc is an experimental Bazel extension with build rules for interfacing with non-Bazel build systems. By the time you read this post, the API may be many versions ahead and might not maintain backwards compatibility. With that in mind, the example contained within this post will target a fixed version of the API right before they removed backwards compatibility with Bazel v0.22.

This is not a tutorial on using Bazel to build C/C++ projects. If you are not familiar with the basics of building C/C++ projects with Bazel, then I recommend that you start with an Introduction to Bazel: Building a C++ Project before reading on. In fact, I recommend that you read through everything in the C++ and Bazel documentation, but that is not strictly necessary.

I would also like to point out that I have only tested this with Bazel v0.20.0 on a Gentoo system. If you are planning to try this from a system running Microsoft Windows, then you should install Bazel v0.23+. See Bazel versions compatibility for up to date compatibility information.

Introducing rules_foreign_cc

Initially I thought that I would have to write a bazel extension with custom build rules, but then I discovered the experimental rules_foreign_cc project after coming across this old Google Groups bazel-discuss thread. The rules_foreign_cc project contains a Bazel extension named rules_foreign_cc//:workspace_definitions.bzl that provides support for adding dependencies that require foreign toolchains. In fact, the workspace definitions extension has the CMake toolchain as a registered native default toolchain and is sufficient for building GNU configure/make projects like Libsodium. With that in mind, I will restrict my focus to what I needed to do to add Libsodium to my project so that the project builds successfully and Libsodium is able to be initialized at run-time.

Project layout

The full source code is available at my BloggerBust/bazelLibsodiumTest GitHub repository. Before we get started, let’s take a brief look at the project’s layout:

tree ~/dev/bazelLibsodiumTest/ -I "bazel*"

/home/dustfinger/dev/bazelLibsodiumTest/
├── BUILD
├── README.org
├── src
│   ├── BUILD
│   ├── lib
│   │   ├── BUILD
│   │   ├── TestLibsodium.cpp
│   │   └── TestLibsodium.h
│   └── main.cpp
└── WORKSPACE

2 directories, 8 files

README.org: a README in org-mode format explaining what the project is about
WORKSPACE: a Bazel workspace where we will load the rules_foreign_cc workspace definitions extension
BUILD: a build file where we will load the rules_foreign_cc configure extension and provide parameters needed to build the Libsodium library.
src/BUILD: defines the main package and calls the built in cc_binary rule for building C/C++ binaries.
src/main.cpp: This is the bazelLibsodiumTest entry point.
src/lib/BUILD: defines the test_libsodium package. This depends on //:libsodium_configure
src/lib/TestLibsodium.cpp: Initializes Libsodium and prints to the standard output a success or failure message

Load the rules_foreign_cc workspace definitions extension

On March 11, 2019 the rules_foreign_cc project removed support for Bazel version less than 0.22. The workspace file in my GitHub repository has a comment explaining how to set the rules_foreign_cc master branch as the dependency, but in this post I will only discuss setting the final commit that provided support for Bazel pre version 0.22, since many distributions do not include the latest Bazel version in their stable package tree.

Determine the sha256 hash

The purpose of the sha256 hash validation is to ensure that subsequent downloads from either the same or different environments are building exactly the same dependencies. It is more about build integrity than security. Please download the zip archive for commit a3593905f73ce19c09d21f9968f1d3f5bc115157 and then calculate its sha256 hash. You can determine the URI of a commit using the pattern: https://github.com/<user-name>/<commit-hash>.zip

wget https://github.com/bazelbuild/rules_foreign_cc/archive/a3593905f73ce19c09d21f9968f1d3f5bc115157.zip
sha256sum a3593905f73ce19c09d21f9968f1d3f5bc115157.zip
rm a3593905f73ce19c09d21f9968f1d3f5bc115157.zip

6f3484eacc172c90d605e79130f9f01ec827a98b99c499c396eddc597a9c219d  a3593905f73ce19c09d21f9968f1d3f5bc115157.zip

Now we know that the sha256 hash is 6f3484eacc172c90d605e79130f9f01ec827a98b99c499c396eddc597a9c219d and with that we can configure the build to validate the hash before building the dependency.

It is important to understand that what I have just explained is a flawed protocol if my intention was to protect myself against a sophisticated man in the middle attack. Consider that if I download the archive and calculate the hash, then ask the build to download the same archive and compare its calculated hash to what I had previously calculated, of course I should expect the hashes to be the same. Following this protocol I should expect the hash to validate even if a hostile proxy was delivering a modified archive. To mitigate the threat of a man in the middle attack, the Libsodium project supports cryptographic integrity checking which I will not be covering in this post.

Add rules_foreign_cc as a dependency

When the archive is extracted the root directory will be the name of the archive, followed by a hyphen, followed by the name of the branch or commit. It is best to strip away this top level archive directory since it causes issues with target paths. We do this by setting strip_prefix to rules_foreign_cc-a3593905f73ce19c09d21f9968f1d3f5bc115157. Additionally, we want to set the sha256 hash of the archive. If the sha256 hash of the downloaded archive does not match the value of the sha256 attribute, then the build will be halted.

workspace(name = "com_github_bloggerbust_bazelLibsodiumTest")
load('@bazel_tools//tools/build_defs/repo:http.bzl', 'http_archive')

# for Bazel 0.22 and below use last supported commit
http_archive(
   name = "rules_foreign_cc",
   strip_prefix = "rules_foreign_cc-a3593905f73ce19c09d21f9968f1d3f5bc115157",
   url = "https://github.com/bazelbuild/rules_foreign_cc/archive/a3593905f73ce19c09d21f9968f1d3f5bc115157.zip",
   sha256 = "6f3484eacc172c90d605e79130f9f01ec827a98b99c499c396eddc597a9c219d"
)

Initialize rules_foreign_cc

To initialize rules_foreign_cc we must load the rules_foreign_cc_dependencies function from the workspace_definitions extension. Once loaded, we can immediately call the function without passing any arguments so that the registered default toolchain will be selected. Calling this function will also initiate necessary code generation needed for C++ and Starlark API support.

load("@rules_foreign_cc//:workspace_definitions.bzl", "rules_foreign_cc_dependencies")
rules_foreign_cc_dependencies()

Configure the Libsodium dependency

The simplest way to ensure that we capture all of the source files that must be included in the Libsodium library is to create a file group using wildcard globs and assign it to the build_file_content attribute. I named the file group libsodium_all, but you can name it whatever you like. Use the same technique that we used under the subheading Determine the sha256 hash to determine the sha256 hash for the Libsodium archive.

all_content = """filegroup(name = "libsodium_all", srcs = glob(["**"]), visibility = ["//visibility:public"])"""

http_archive(
  name = "org_libsodium",
  url = "https://github.com/jedisct1/libsodium/releases/download/1.0.17/libsodium-1.0.17.tar.gz",
  sha256 = "0cc3dae33e642cc187b5ceb467e0ad0e1b51dcba577de1190e9ffa17766ac2b1",
  strip_prefix = "libsodium-1.0.17",
  build_file_content = all_content
)

Create the a BUILD file for Libsodium

Now it is time to configure make. We need to let make know the following:

Where the library source code is for Libsodium: This is satisfied by setting the lib_source attribute to a label targeting the file group that we created under Configure the Libsodium dependency.
The path to where the compiled library binaries will be written to following a successful build: This is satisfied by setting the out_lib_dir attribute to a directory relative to the libsodium_configure package. I simply named this directory lib/.
The name of the resulting shared library that we would like bazelLibsodiumTest binary to link to: This is satisfied by setting the shared_libraries attribute. Alternatively you can specify a static library with the static_libraries attribute.

load("@rules_foreign_cc//tools/build_defs:configure.bzl", "configure_make")

configure_make(
    name = "libsodium_configure",
    lib_source = "@org_libsodium//:libsodium_all",
    out_lib_dir = "lib",
    shared_libraries = ["libsodium.so.23"],
    visibility = ["//visibility:public"]
)

Since the BUILD file is in the root of our project the label for the configure_make target will be //:libsodium_configure.

Create a BUILD file for our TestLibsodium library

To add the Libsodium library as a dependency of the TestLibsodium Library package, simply assign the //:libsodium_configure label to the deps attribute.

cc_library(
    name = "test_libsodium",
    srcs = glob(["*.cpp"]),
    hdrs = glob(["*.h"]),
    deps = ["//:libsodium_configure"],
    visibility = ["//visibility:public"]
)

Time to build and run

There are a few strange things in this build command that I think need some explanation:

env TMPDIR=~/dev/tmp: For security reasons I mount /tmp as non executable, therefore I must override the default path since Bazel executes the configure script from this location. You probably won’t need this part of the command.
--experimental_cc_skylark_api_enabled_packages=@rules_foreign_cc//tools/build_defs,tools/build_defs,@foreign_cc_impl: Required for Bazel 0.20 to 0.21
2>&1: I discovered that all the compiler output was sent to stderr. This redirects stderr to stdout. Perhaps this has been fixed in a more recent version of Bazel.

env TMPDIR=~/dev/tmp bazel build --experimental_cc_skylark_api_enabled_packages=@rules_foreign_cc//tools/build_defs,tools/build_defs,@foreign_cc_impl //src:main 2>&1

INFO: Invocation ID: b03a9e49-ba0f-46bb-a232-43cf82087212
Loading:
Loading: 0 packages loaded
Analyzing: target //src:main (0 packages loaded, 0 targets configured)
INFO: Analysed target //src:main (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
[0 / 1] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //src:main up-to-date:
  bazel-bin/src/main
INFO: Elapsed time: 0.062s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

Since the build was successful we should be able to run the main binary and see a message stating that Libsodium has been initialized.

bazel run --experimental_cc_skylark_api_enabled_packages=@rules_foreign_cc//tools/build_defs,tools/build_defs,@foreign_cc_impl //src:main

going to initialize Libsodium, wish me luck
//////////////////////////////////////////////////////
// W00t! the Libsodium library has been initialized //
//////////////////////////////////////////////////////

If anything goes wrong, then add --verbose_failures --sandbox_debug optional flags to the build command.

Conclusion

Thanks to the rules_foreign_cc project adding Libsodium as a dependency was easy. For projects based on GNU configure/make Autotools that have foreign dependencies of their own, additional work would be needed. Also, keep in mind that the project is experimental and is unlikely to maintain backwards compatibility until it is deemed stable. I hope that you found this post useful.

Comments

Your comment has been submitted and is now pending moderation

Be the first to comment on this article.