Julia Packaging

Great, now you have a working knowledge of programming in Julia. When you use it to make awesome packages, you’ll no doubt want to publish them. So, today we’re going to be giving you an overview of packaging in Julia.

Some terminology

Before we start, there are some terms that you should be familiar with. For the sake of completeness, we’ll discuss them briefly, feel free to skip this section. When you code in Julia, you work in projects, which are composed of files with the .jl extension as well as a manifest. Artefacts are platform-specific binaries (that aren’t in Julia). At the risk of oversimplification, they can be thought of as the .bin files that tag along with everything nice.

When you bundle a project with all its dependencies, you get a package. And as we already know, packages themselves are the building blocks that supplement the standard library to add niche-specific functionality. And environments then, are defined by a list of packages and their respective versions. This is stored in a manifest stored in Manifest.toml.

Lights Camera Code

I mean action, lights camera action!… I mean lets look at all the things we talked about, but in actuality. Lets create a new environment:

(@v1.5) pkg> activate foo
 Activating new environment at `C:\Users\MLG\Desktop\foo\Project.toml`

Lets check Pkg status:

(foo) pkg> status
Status `C:\Users\MLG\Desktop\foo\Project.toml` (empty project)

Now, if you were to try to go to that directory, you’d find that it doesn’t even exist! That’s because a blank Project might as well be non-existent, so it is!! So lets add a package:

(foo) pkg> add Pluto
Resolving package versions...
Updating `C:\Users\MLG\Desktop\foo\Project.toml`
  [c3e4b0f8] + Pluto v0.12.17
Updating `C:\Users\MLG\Desktop\foo\Manifest.toml`
  [56f22d72] + Artifacts v1.3.0
  [9a962f9c] + DataAPI v1.4.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [fb4132e2] + FuzzyCompletions v0.4.0
  [cd3eb016] + HTTP v0.9.0
  [83e8ac13] + IniFile v0.5.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.1.3
  [739be429] + MbedTLS v1.0.3
  [c8ffd9c3] + MbedTLS_jll v2.16.8+1
  [99f44e22] + MsgPack v1.1.0
  [c3e4b0f8] + Pluto v0.12.17
  [3783bdb8] + TableTraits v1.0.0
  [bd369af6] + Tables v1.2.2
  [5c2747f8] + URIs v1.1.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode

Last time we glossed over this, but not this time. So, you see here, not only did Pkg install Pluto(its latest version at that) but also installed all it’s dependencies. So, now if you look at your Manifest.toml you’ll see it to be thus:

# This file is machine-generated - editing it directly is not advised

[[Artifacts]]
deps = ["Pkg"]
git-tree-sha1 = "c30985d8821e0cd73870b17b0ed0ce6dc44cb744"
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"
version = "1.3.0"

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[DataAPI]]
git-tree-sha1 = "ad84f52c0b8f05aa20839484dbaf01690b41ff84"
uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
version = "1.4.0"

[[DataValueInterfaces]]
git-tree-sha1 = "bfc1187b79289637fa0ef6d4436ebdfe6905cbd6"
uuid = "e2d170a0-9d28-54be-80f0-106bbe20a464"
version = "1.0.0"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[FuzzyCompletions]]
deps = ["REPL"]
git-tree-sha1 = "5ca3ddf3061771d25d1699ce53a80a39300811e3"
uuid = "fb4132e2-a121-4a70-b8a1-d5b831dcdcc2"
version = "0.4.0"

[[HTTP]]
deps = ["Base64", "Dates", "IniFile", "MbedTLS", "Sockets", "URIs"]
git-tree-sha1 = "9634200f8e16554cb1620dfb20501483b873df86"
uuid = "cd3eb016-35fb-5094-929b-558a96fad6f3"
version = "0.9.0"

[[IniFile]]
deps = ["Test"]
git-tree-sha1 = "098e4d2c533924c921f9f9847274f2ad89e018b8"
uuid = "83e8ac13-25f8-5344-8a64-a9f2b223428f"
version = "0.5.0"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[IteratorInterfaceExtensions]]
git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856"
uuid = "82899510-4779-5014-852e-03e436cf321d"
version = "1.0.0"

[[JLLWrappers]]
git-tree-sha1 = "c70593677bbf2c3ccab4f7500d0f4dacfff7b75c"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.1.3"

[[LibGit2]]
deps = ["Printf"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[MbedTLS]]
deps = ["Dates", "MbedTLS_jll", "Random", "Sockets"]
git-tree-sha1 = "1c38e51c3d08ef2278062ebceade0e46cefc96fe"
uuid = "739be429-bea8-5141-9913-cc70e7f3736d"
version = "1.0.3"

[[MbedTLS_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "0eef589dd1c26a3ac9d753fe1a8bcad63f956fa6"
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"
version = "2.16.8+1"

[[MsgPack]]
deps = ["Serialization"]
git-tree-sha1 = "a8cbf066b54d793b9a48c5daa5d586cf2b5bd43d"
uuid = "99f44e22-a591-53d1-9472-aa23ef4bd671"
version = "1.1.0"

[[Pkg]]
deps = ["Dates", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "UUIDs"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[Pluto]]
deps = ["Base64", "Dates", "Distributed", "FuzzyCompletions", "HTTP", "InteractiveUtils", "Logging", "Markdown", "MsgPack", "Pkg", "REPL", "Sockets", "Tables", "UUIDs"]
git-tree-sha1 = "41927b687ee3c553dd8dac28f4183326b054bd54"
uuid = "c3e4b0f8-55cb-11ea-2926-15256bba5781"
version = "0.12.17"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[TableTraits]]
deps = ["IteratorInterfaceExtensions"]
git-tree-sha1 = "b1ad568ba658d8cbb3b892ed5380a6f3e781a81e"
uuid = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c"
version = "1.0.0"

[[Tables]]
deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "LinearAlgebra", "TableTraits", "Test"]
git-tree-sha1 = "240d19b8762006ff04b967bdd833269ad642d550"
uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
version = "1.2.2"

[[Test]]
deps = ["Distributed", "InteractiveUtils", "Logging", "Random"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[URIs]]
git-tree-sha1 = "bc331715463c41d601cf8bfd38ca70a490af5c5b"
uuid = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4"
version = "1.1.0"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

Long and tedious indeed. But this is not the manifest we’ve been talking about. No, the manifest is in the same directory, Project.toml:

[deps]
Pluto = "c3e4b0f8-55cb-11ea-2926-15256bba5781"

Now this looks more readable. And this is all that your project is currently. But now you understand that whenever you clone a WIP project, which isn’t yet packaged, it’ll come with its own Project.toml which Julia will use to reproducibly build that project on your machine. To accomplish this, simply go into the project directory and:

(foo) pkg> activate .

Now that you know all about projects, lets move on to packages.

Packages

A package is really just a clothed project. More precisely,

A package is a project with a name, uuid and version entry in the Project.toml file, and a src/PackageName.jl file that defines the module PackageName. This file is executed when the package is loaded.

Without further ado lets create one and see for ourselves:

(fooProj) pkg> generate barPackage
 Generating  project barPackage:
    barPackage\Project.toml
    barPackage\src/barPackage.jl

So we see that we’ve got a new directory and some files. In particular, we have a new Project.toml while our original Manifest.toml and Project.toml are unchanged. Indeed, we’ve just created a package barPackage in our project fooProject. So lets import our project’s package into it:

julia> cd("barPackage")

(fooProj) pkg> activate .
 Activating environment at `C:\Users\icebear\Desktop\fooProj\barPackage\Project.toml`

julia> import barPackage
[ Info: Precompiling barPackage [317e392e-7f8c-4046-94eb-a62d137665d3]

julia> barPackage.greet()
Hello World!

And we even tested it out, without having to open any files, great. Now you can go on and publish this anywhere.

Sidenote

When you install Julia, you have only one project, named after the Julia version you installed, located at ~/.julia/environments/v*.*.

Although all the packages themselves are stored together (in .julia/packages directory), so that you avoid re-downloading package versions that you already have.

A novel thing about environments in Julia is that they’re stackable, as in you could be working in an environment foo, having some packages, and then overlay an environment bar on top of it to have packages from both the environments. Just a reminder, environments are independently updated, so updating foo won’t change bar at all.

develop

Finally, something to note is that Pkg has native support for multiple registries. We won’t go deeper into this, but in summary it means that you could, say be working with a public package, fork it into a private registry and then supply your peers with that forked version without having to jump through unnecessary hoops. In fact, forking a package is as simple as:

(foo) pkg> develop --local Pluto

This will setup a git clone of the Pluto package in your working directory, and then make the environment foo use that local version as opposed to the publicly available one. But remember that this stops the updating of this package’s dependency graph in the manifest, you have to trigger it manually hereon, by using resolve in the REPL. If you want to go back to the publicly available of a package you earlier cloned using develop, you’ll use:

(foo) pkg> free Pluto

As expected, you can import a package only once. If you want to make changes to (even a local) package, and then load the updated version, use Revise.jl. At the opposite end, if you want a package to stay untouched, and not be updated at all:

(foo) pkg> pin Pluto

This of course pertains only to that environment.

Orphans

If a package or artifact is not used by any Project, then julia labels it an orphan. Orphans that are more than a month old are deleted whenever Pkg garbage collects. You can also trigger it manually:

(foo) pkg> gc

This will delete all orphans present at that time.

References

• https://julialang.github.io/Pkg.jl/v1/artifacts/
• https://julialang.github.io/Pkg.jl/v1/creating-packages/