Package Management with Conan

During my internship at Arm, I’ve worked quite a bit with a neat piece of software called Conan. Conan isn’t a household name in the world of software yet, so it’s no big deal if you haven’t heard of it, but I think it’s awesome. It is a C/C++ package manager that’s built for developers. It’s open source, decentralised, multi-platform, and it uses a Python class as its package descriptor format. If this sounds interesting to you, keep reading!

In this post, I’ll try my best to introduce you to the basics of Conan and how to use it. In my experience, Conan has a bit of a learning curve. I wished a straightforward tutorial existed when I first used it, so hopefully this post will made someone’s life easier.

This post should be used as a supplement to the Conan docs (which are very well written, but I believe they assume some prior knowledge of package management systems to fully appreciate). The developers are also very responsive on GitHub and I’m sure they’re happy to answer queries if you have them. Also, if you have any questions for me, drop them in the comments below.

Why Conan?

There are a few reasons why Conan is different to other package managers that are available. First of all, it’s built for C/C++, with developers in mind. This is different to package managers like Apt, pacman, and Homebrew, which are all awesome but are decidedly more consumer focused, at least in my experience. Conan is built to solve a different problem, seeking to provide a robust system for sharing the build instructions and native binaries with as much flexibility as possible. For example, a company could use Conan to build their product and share it amongst employees who can then proceed to test it without the need for the build tools.

Conan is also rapidly gaining popularity because of how easy it is to automate, and how extremely well it integrates with IDEs, build systems, and in particular CI.

The official website touts Conan’s advantages with regards to:

  • Flexibility. Conan is available on all platforms, is build-system agnostic, and uses Python as its package descriptor. Using Python for package recipes is really clever – it means that recipes are easy to create and read, because these days, everyone knows Python.

  • Speed. Conan is great at managing binaries as well as recipes. Recipes can create different ‘configurations’ of binaries – for each different architecture and operating system, say – and Conan handles the distinction between them seamlessly. Conan also manages dependencies really nicely with transitive dependencies, conflict detection, and conditional dependencies.

  • Control. Conan is decentralised, meaning you can run your own private Conan server. This is ideal for sharing tools within an organisation, or for distributing builds of proprietary software internally. The Python descriptor format is dead powerful and the Conan Python libraries provide you with an assortment of different tools so that you can achieve robust packaging in very few lines of code.

In my experience, Conan is also awesome because:

  • It supports pre-compiled binaries. These are sometimes referred to as ‘pure binary packages’. They are just packages for which the source code isn’t available for whatever reason, and you only have the binaries. For example, binaries from a third party, or previously built by another process or team not using Conan. This is useful when you have a set of artifacts you’d like to oversee with Conan, but have no easy or realistic way of building them from scratch.

How Conan Works

Conan uses a Python class as its package descriptor. This means that you describe your package using Python code. Specifically, you use Python to define how your code is sourced, built, and packaged. The file responsible for doing this is called the conanfile, and is given the filename conanfile.py.

This is what Conan refers to as the package recipe. There is a distinction to be made between a recipe and the binary package. Conan handles both. Intuitively, the recipe is responsible for creating the binary packages. A single recipe will almost always produce different binary packages depending on the settings supplied to it. For instance, the exact same program will have two separate binaries for different computer architectures or operating systems. Conan distinguishes between the recipe and the packages in order to support different settings configurations.

If this doesn’t make sense, check out this diagram:

The difference between Conan recipes and binary
packages

This recipe, along with any pre-built binaries, can be packaged up and stored in a Conan repository. You can define exactly which ‘settings’ distinguish one package from another using the settings tuple in the conanfile.py, but we’ll come to that later.

A conan package is somewhat symbolic – it just refers to the thing that is defined by the package recipe. A package is referenced by its name, version, and two additional fields called the user and channel. A package reference looks like this: PackageName/1.2.3@user/channel.

When I first started using Conan, I was a little confused about the user and channel part. As it turns out, these are purely symbolic and you can use whatever you like here. For example, you might use the user ‘conan’ and the channel ‘stable’ for your stable releases, meaning your package recipe would look something like this: MyPackage/1.2.3@conan/stable.

So really, our diagram should look like this:

Conan packages

For now, I think that’s enough about how Conan works. There’s definitely more to be learned, though. This is just the basics to give you a general idea about how packages are structured within Conan and the constituent parts that make them up.

Getting Conan

Now that we know a little bit about what Conan is and how it works, let’s install it. This is super easy. The recommended way of doing it is to use pip. Type the following into your terminal:

pip install conan

And there you go. Done.

Using Conan

First, let’s consider using Conan as a consumer. That is, you want to download and install packages from a repository. But before anything, let’s talk about remotes. Remotes are just servers used as binary repositories that store packages by reference – they are Conan servers. Conan is shipped with only one remote by default, called conan-center, which is ‘the place to find and share popular C/C++ Conan packages’. There’s packages in there for OpenSSL, Poco, etc.

The great thing about Conan is that it’s decentralised and you can host your own Conan server to keep your packages private. For now, we’ll look at how we can install packages from conan-center to keep things nice and easy.

When you go to install a package, Conan checks if it is already in your local cache. If it is, Conan will install the package from there. Otherwise, Conan will systematically search the remotes until it finds one where the package is available. To install packages, we use conan install. If we wanted to install OpenSSL from conan-center, for instance, we type:

conan install OpenSSL/1.1.0g@conan/stable

When Conan sees this command, here’s what it’ll do:

  1. If this is the first time you’ve run Conan, it’ll generate a profile based on your system settings (i.e. your architecture, operating system, etc).
  2. It’ll then check if the package referred to by the package reference is available in the local cache. If you’ve only just installed Conan, it won’t be.
  3. Conan will then search the remotes. It’ll search conan-center, find that the reference matches up with a binary package on the server (which matches your system configuration), and pull it to your local cache. It will do the same for any (transitive) dependencies too.
  4. Conan will run a few of the methods defined in the packages conanfile.py, which basically ‘installs’ the packages as you’d expect, also doing the same for the dependencies. Exactly what happens at this stage depends on the packages you’re installing. For instance, the process is much simpler for pre-built binary packages (ones that have been built by the package maintainer for your particular system profile) than it is for packages which only ship the conanfile.py. We’ll learn about these methods and what they do later.

Basically, conan install installs a package – no surprises here. The data for the package is stored hierarchically in ~/.conan/data. If we output the directory tree, it’ll look something like this:

~/.conan/data/OpenSSL$ tree
.
└── 1.1.0g
    └── conan
        ├── stable
        │   ├── export
        │   │   ├── conanfile.py
        │   │   └── conanmanifest.txt
        │   ├── locks
        │   │   └── b3e8c6b6e5f8456a00d1a77a6a5a1aeb06b2ad48
        │   └── package
        │       └── b3e8c6b6e5f8456a00d1a77a6a5a1aeb06b2ad48
        │           ├── LICENSE
        │           ├── conaninfo.txt
        │           ├── conanmanifest.txt
        │           ├── include
        │           │   └── openssl
        │           │       ├── aes.h
        │           │       ├── applink.c
        │           │       ├── asn1.h
        │           │       ├── asn1_mac.h
				...

Congratualations. You just installed a Conan package on your system.

Packaging with Conan

To appreciate how Conan works as a package manager, it’s a good idea to take a look at how we create packages. Fortunatley, this is amazingly straightforward. As I said earlier, Conan is written in Python, and so is conanfile.py. When creating packages, you have all of the flexibility and power of Python, as well as having a suite of tools provided by the Conan team which facilitate a lot of common functions.

Let’s take a look!

Creating a Basic Package

We’re going to create a simple ‘Hello World’ Conan package. The only file we need is the following C code:

#include <stdio.h>

int main(int argc, char *argv[]) {
    printf("Hello World\n");
    return 0;
}

Call that file hello.c and store is somewhere safe. To create a skeleton directory for our Conan project, here’s what we run:

mkdir helloworld && cd helloworld
conan new HelloWorld/0.1 -t

Here, we create a new directory, change to it, and initialise a package called HelloWorld with version 0.1. The -t flag tells Conan to create a test directory for us to test the package. After this, we’re left with the following directory structure:

~/helloworld$ tree
.
├── conanfile.py
└── test_package
    ├── CMakeLists.txt
    ├── conanfile.py
    └── example.cpp

We’re primarily concerned with conanfile.py at the moment. Let’s take a look…

from conans import ConanFile, CMake, tools


class HelloworldConan(ConanFile):
    name = "HelloWorld"
    version = "0.1"
    license = "<Put the package license here>"
    url = "<Package recipe repository url here, for issues about the package>"
    description = "<Description of Helloworld here>"
    settings = "os", "compiler", "build_type", "arch"
    options = {"shared": [True, False]}
    default_options = "shared=False"
    generators = "cmake"

    def source(self):
        self.run("git clone https://github.com/memsharded/hello.git")
        self.run("cd hello && git checkout static_shared")
        # This small hack might be useful to guarantee proper /MT /MD linkage
        # in MSVC if the packaged project doesn't have variables to set it
        # properly
        tools.replace_in_file("hello/CMakeLists.txt", "PROJECT(MyHello)",
                              '''PROJECT(MyHello)
include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
conan_basic_setup()''')

    def build(self):
        cmake = CMake(self)
        cmake.configure(source_folder="hello")
        cmake.build()

        # Explicit way:
        # self.run('cmake %s/hello %s'
        #          % (self.source_folder, cmake.command_line))
        # self.run("cmake --build . %s" % cmake.build_config)

    def package(self):
        self.copy("*.h", dst="include", src="hello")
        self.copy("*hello.lib", dst="lib", keep_path=False)
        self.copy("*.dll", dst="bin", keep_path=False)
        self.copy("*.so", dst="lib", keep_path=False)
        self.copy("*.dylib", dst="lib", keep_path=False)
        self.copy("*.a", dst="lib", keep_path=False)

    def package_info(self):
        self.cpp_info.libs = ["hello"]

This is a lot to take in upon first glance. This is how a conanfile.py is structured – it contains a single class, subclassed from ConanFile, and has a couple of attributes and methods. At a high level, all you need to know is that the attributes provide a sort of metadata for the package and the methods in the file are either run or not run in different orders depending on exactly what Conan is doing.

Attributes are easiest to understand, so let’s cover those:

  • The name attribute is simply the name of the package.
  • The version attribute is the version of the package. This has to be a string – otherwise you’ll get a type error.
  • license, url, and description are just plain-old metadata.
  • settings is an interesting one. It lets you tell Conan what variables will ultimately change the final binary package that is built. The default as shown above is fairly sensible – if you change the OS, compiler, build type, or architecture, you’re certainly going to get a different binary at the end. Conan responds appropriately by giving the package a different package ID. So settings basically says to Conan: “these are the settings that, when changed, will give a different package – so you have to account for all of these when you’re thinking of a package ID”.
  • options is a bit of an extension to settings. It defines options which must be defined when the package is installed, and will affect the build in some way.
  • generators is a bit of a more complex discussion so I’ll omit any detail for now. In short, though, a generator quite literally generates an artifact that is used to utilise your package. For instance, the virtualenv generator creates a script to activate virtualenv that adds your package’s binaries to $PATH.

So what do these methods do? Let’s look at them in isolation first, then we can see how the big picture pulls together.

  • source() is used to retrieve your code. It could be pulled with git, downloaded via HTTP, doesn’t matter – it’s just a dedicated place for you to gather all the external resources you need.
  • build() is responsible for actually building your package. This could be as simple as just calling a shell command to run cmake or something.
  • package() is responsible for packaging up all of the artifacts from the build process by moving them into a package directory. This directory is what eventually gets compressed to a .tar.gz and is stored on the remote.
  • package_info() declares information about the package for the consumer. For instance, you need to specify the lib directories, etc. In the default conanfile.py, you can see an example of this.

With that in mind, let’s modify the default conanfile.py so that it properly packages our hello world program. To make the example more complete, suppose we have a GitHub repo at https://github.com/Hello/World which contains nothing but our helloworld.c file. Filling out the attributes first, we get:

class HelloworldConan(ConanFile):
    name = "HelloWorld"
    version = "0.1"
    url = "https://github.com/Hello/World"
    description = "A simple hello world"
    settings = "os", "compiler", "build_type", "arch"

I’ve left out some attributes intentionally here. They aren’t relevant for such a small program.

Now let’s create the required methods. First, we need a source() method whose job is to pull the code from GitHub. So we can write:

def source(self):
    self.run("git clone {}".format(url))

Easy enough – we just clone the URL we specified as an attribute. Next up is the build method, where we build our project. Thankfully there’s very little to do here. For larger projects, you might want to use a build system such as CMake (which Conan has generators for!), but for our trivial example we might as well just use clang alone. Our build method might look a little something like:

def build(self):
    self.run("clang -o helloworld {}/{}".format(self.source_folder, "helloworld.c"))

Here, self.source_folder refers to wherever the source was downloaded. This is handled automatically by Conan. Finally, we create a package info. Since we’ve only created a single artifact with no real structure, this is simple too.

def package(self):
    self.copy("*")

This just tells Conan to copy everything to a package directory. Here, everything is just one binary executable, so we’re good. Because our program is so simple, we’re done. We now have a full-fledged Conan package!

To test it out (if you used the -t flag when creating the skeleton, delete the test_package folder first), try running:

conan create . conan/testing

Then take a look inside ~/.conan/data. You should see your package, hierarchically structured as explained earlier.

What Happened?

conan create is something of a helper command – it runs a few of the methods, one after the other, to create and install the package locally. Specifically it runs the methods in this order:

config_options(), configure(), requirements(), package_id(), build_requirements(), build_id(), system_requirements(), source(), imports(), build(), package(), package_info()

We haven’t seen half of these methods! Truthfully, we’ve only scratched the surface of Conan in this post. If we filter out the above list to the ones we know…

source(), build(), package(), package_info()

Aha! So when we run conan create, we’ve actually just sourced our project, built in from scratch, and packaged it up. It’s now ready for distribution – you can upload it to a remote with conan upload, and then others can pull it to their local machine with conan install.

What happens when a user runs conan install with your package as a reference? That’s a discussion for another time, but the underlying idea is the same; Conan retrieves the package from the remote, and then runs a few functions, in order, to install the package on the user’s machine. The fundamental different here, though, is that if we’ve uploaded a binary package, the build() method won’t be called since the package is already built.

Conclusion

Conan is definitely worth looking into. I hope that this post has taught you the basics. Check out conan.io and the Conan docs to learn more if you’re interested!

I’ve been making slow but steady contributions to the Conan project in my spare time when I’m not busy doing anything else. The guys behind the scenes are awesome and they’re doing a great job at making Conan a seriously neat package manager, especially among engineering teams. Hats off to you all!

Written on August 28, 2018