Sitemap

Signing in Four Public Software Package Registries: Quantity, Quality, and Influencing Factors

10 min readMay 10, 2024

--

This is a brief for the research paper “Signing in Four Public Software Package Registries: Quantity, Quality, and Influencing Factors,” published at IEEE S&P 2024. This work was led by Taylor R. Schorlemmer. The full paper is here. Taylor R. Schorlemmer also wrote this brief, which I have lightly edited.

Summary

Modern software relies heavily on third-party dependencies, which are often distributed through public software package registries. Signing is a security mechanism that can help ensure the integrity and authenticity of these dependencies. By signing a package, a developer can cryptographically bind the package to their identity and ensure that the package has not been tampered with. This helps downstream users understand the provenance of the package and make informed decisions about whether to trust it. Unfortunately, is not universally adopted, even in the ecosystems that support it.

This post describes the state of signing in four popular package registries: PyPI, Maven Central, Docker Hub, and Hugging Face. We find that the quantity and quality of signing varies widely across these registries. Furthermore, we identify factors that do (registry policies, dedicated tooling, and startup cost) and do not (cybersecurity events) influence the adoption of signing in these ecosystems.

Background

To understand this post, you need to know a bit about software package registries, software supply chains, and software signing.

Software Package Registries

Package registries are tools that help developers distribute and consume software packages. These registries exist over a wide range of software types, including programming languages (e.g., PyPI, Maven Central), operating systems (e.g., APT, RPM), containerization (e.g., Docker Hub, GitHub Container Registry), and machine learning (e.g., Hugging Face, PyTorch Hub).

Software Supply Chains and Attacks

Commercial and government software products incorporate open-source software packages. In a 2023 study of 1,703 commercial codebases across 17 sectors of industry, Synopsys found that 96% used open-source code, and 76% of the total application code was open-source. Open-source software packages depend on other packages, creating software supply chains. Malicious actors have begun to attack software supply chains, injecting malicious code into packages to gain access to downstream systems. These attacks have affected critical infrastructure and national security.

Many mitigations have been proposed for software supply chain attacks. Some approaches seek to increase confidence in a package’s behavior, e.g., measuring use of best practices, independent validation, and formal guarantees. Other approaches target the package’s provenance, e.g., Software Bill of Materials (SBOMs) and “vendoring” trusted copies of dependencies. The strongest guarantee of a package’s provenance is a cryptographic signature by its maintainer.

Software Signing

Signing is a security mechanism that can help ensure the integrity and authenticity of software packages. By using public key cryptography, a developer can cryptographically bind a package to their identity and ensure that the package has not been tampered with. The process goes like this:

  • First, a maintainer creates/maintains a package.
  • Next, the maintainer, or some other actor, generates a pair of cryptographic keys. Using the private key, they sign the package.
  • After signing, the package, public key, and signature are published to registries (these can be separate or the same).
  • Finally, a user fetches the package, public key, and signature to verify the package. A successful verification indicates that the package has not been tampered with since it was signed.

The next figure illustrates this process, as well as the (many) things that can go wrong:

Figure 1: Signing process and failures.
  1. Creation Failure: The signer fails to create a key-pair or sign the package. Sometimes maintainers just choose not to sign their packages.
  2. Bad Key: The signer may not use the correct key or the user uses the wrong key to verify the package. This can happen when keys have the same id.
  3. Bad Signature: The signer may not sign the package correctly (wrong artifact) or the package may be tampered with after signing.
  4. Publishing Failure: The signer may not publish the public key or signature.
  5. Key Revocation: The signer may revoke the key after signing the package. This may be due to a compromise or a change in maintainership.
  6. Key Expired: The key may expire before the package is verified. Sometimes keys are also used after they expire.
  7. Signature Expired: The signature may also expire before the package is verified.
  8. Discovery Failure: The user may not be able to fetch the public key or signature. This can happen when the registry does not support signing or the signer does not publish the key or signature.

Knowledge Gap

All research seeks to fill a gap in knowledge. For this study, prior work has noted that many packages are unsigned. However, we lack up-to-date measurements of software signing practices, and we do not know the general quality of existing signatures. Furthermore, we lack a deeper understanding of the factors that affect signature adoption. This knowledge would guide future efforts to incentivize software signing, so that the provenance of software supply chains can be improved.

Signing Adoption Theory

Motivated by behavioral economics, we developed a theory of signing adoption. In this theory, we posit that maintainers are influenced by incentives to create a signature and to correctly follow the signature creation process. These incentives could be intrinsic (e.g., altruism or reputation) or extrinsic (i.e., external factors that influence the maintainers decision). This theory is illustrated in Figure 2.

Figure 2: Incentives influence a maintainers decision to create a signature and to correctly follow the signature creation process.

For this study, we focus on extrinsic incentives because they are feasible to measure. We select four (although there are likely many more) extrinsic incentives that could influence a maintainer’s decision to sign a package:

  1. Registry Policies: The registry may require or incentivize signing.
  2. Dedicated Tooling: The registry may provide dedicated tooling to make signing easier.
  3. Cybersecurity Events: High-profile cybersecurity events may increase the adoption of signing.
  4. Startup Cost: The cost to start signing may be too high for some maintainers.

We formulated hypotheses as to their effects, as indicated in Table 1.

Table 1: What we expected to find — hypothesized relationship between extrinsic factors and signing adoption.
  • We expected that registry policies could be pro- or anti- signing, and that these policies would, respectively, increase or decrease signing adoption.
  • We expected that dedicated tooling would increase signing adoption.
  • We expected that major cybersecurity events would increase signing adoption.
  • We expected that starting to sign would be hard, so that this would disincentivize signing.

Approach — Let’s Measure!

We selected four popular package registries to study: PyPI, Maven Central, Docker Hub, and Hugging Face. These registries were chosen because they are widely used, represent a diverse set of software types, and support maintainer-generated signing.

As the next figure illustrates, using these registries, we:

  1. Collected a list of packages from each registry.
  2. Filtered for packages created between 2015 and 2023 with at least 5 versions.
  3. Measured the quantity and quality of signing in each registry.
  4. Evaluated signing adoption between registries. We considered adoption rates longitudinally (across time) so that we could assess the impact of the four external factors mentioned above.
Figure 3: Approach to measure and evaluate signature adoption.

Some Results (see paper for more)

We discuss a subset of the results from our paper. Please take a look if you’d like more details about our findings. This includes information about cryptographic algorithms, key lengths, influencing factors and statistical tests.

Quantity and Quality of Signing

Table 2 shows the quantity and quality of signing in each registry.

  • The top rows indicate the total number of packages, versions, and artifacts. For example, a single Maven package may have many versions, and within each version of a Maven package there are many distinct files, each of which may be signed. We call the base signable construct an “artifact” for consistency across registries.
  • The middle rows indicate signing quantity. All registries aside from Maven Central have less than 2% of artifacts signed in 2023. This trend is likely due to Maven Central’s semi-mandatory signing policy.
  • The bottom rows indicate signing quality. Failures are common for signed artifacts in all registries aside Docker Hub. This trend is likely due to Docker Hub’s dedicated signing tools.
Table 2: The number of assessed packages and artifacts from each registry, the percent with and without signatures, and the breakdown of signature status for signed artifacts. Each column is written like “X (Y)” — the X is in the past year, the (Y) is over the entire measurement period. For each measurement, we show the most recent year and the entire measurement period. “ — ”: not measurable (Hugging Face hides keys, only disclosing whether they think that validation succeeded).

Signing over Time

To shed a bit more light on the preceding table, which reported signing rates in aggregate, we took a look at signing over time. We examined quantity and quality.

Figure 4 tracks signing quantity over time. You can see that the only registry with a consistently high quantity of signed artifacts is Maven Central (blue line). Other registries have seen a relatively low quantity of signed artifacts, with some fluctuation over time. The Hugging Face data (green curve) is spikey — there were very few signed artifacts in that era (measured in the hundreds) so any deviation could lead to spikes.

Figure 4: Quantity of signed artifacts over time. Axes are time (monthly increments) and the percentage of signed artifacts per registry.

Figure 5 shows signing quality over time. This indicates that, aside from Docker Hub with perfect quality (orange line), the quality of signed artifacts in other registries remains sub-optimal. The spikes in Hugging Face (green line) are again due to the small number of signed artifacts.

Figure 5: Quality of signed artifacts over time. X-axis shows time (monthly increments). Y-axis shows percentage of signatures with good status.

Influencing Factors, or, What Might Make Someone Sign?

As mentioned above, we considered the effect of four factors: registry policies, dedicated tooling, cybersecurity events, and startup costs. Let’s examine each in turn. The first three, we checked with references to Figure 4 and 5 (visual tests) and some statistical tests on the underlying data. The last, we checked with a separate measurement on our data. Table 3 summarizes what we found:

Table 3: Summary of observed results
  1. Registry Policies: Registry policies appear to have an appreciable impact on signing quantity. We observe this in three different ways: (1) On March 22 2018, PyPI de-emphasized the use of PGP signatures by removing UI elements that encouraged signing. Visual and statistical tests show that the quantity of signatures on PyPI decreased after this change. This is visible in Figure 4. (2) After a provenance visibility-related update on September 5 2019, the quantity of signatures on Docker Hub increased. This is visible in Figure 4 and also supported by statistical tests. (3) Maven Central is the only registry with a consistently high proportion of signed artifacts. It is also the only registry with mandated signing. This indicates that mandating signing has a large impact on adoption.
  2. Dedicated Tooling: Dedicated tooling appears to have a noticeable effect on signing quality. Docker Hub has perfect signature quality. It is also the only registry with dedicated tooling. When creating a signature with Docker Content Trust (DCT), the tool ensures that the signature was created correctly. This suggests that tooling has an effect on the qualtity of signatures adopted.
  3. Cybersecurity Events: We were unable to find any instances where cybersecurity events had an effect on the quantity or quality of signatures. After checking for the effect of several publications, standards, and cyber attacks, we observed no significant change in signing. Notably, the Solarwinds hack and the Docker Hub hack did not align with any changes in signing.
  4. Startup Cost: To check the effect of startup cost, we compared the probability of artifacts being signed with the probability that artifacts were signed after the first signature was created for a package. These results are shown in Table 4. Note that for all registries, except for Maven Central where a mandate is in effect, the probability of signing increases by about 40% after the first signature. This suggests that startup costs disincentivize signatures, or in other words, “the first signature is the hardest”.
Table 4: The probability of an artifact having a signature. Raw probability describes the likelihood of any artifact in the registry having a signature. After 1st signature describes the probability that an artifact will be signed if one of the previous artifacts from the same package has been signed.

Caveats

We note two caveats for this work.

  1. We performed statistical analyses to test our various hypotheses. There are many comparisons that could be performed, but every additional test increases the risk of Type-1 errors (false positives). For this reason, we only tested the hypotheses that are supported by exploratory data analysis (visual inspection on the preliminary version of our data), and we applied a Bonferroni correction to our p-values.
  2. Where present, statistically significant results indicate correlation, not causation. Our research is informed by an underlying predictive theory. If the predicted results occur at a statistically significant level, they support the theory. However, our method is a quasi-experiment and we did not control all relevant variables. A controlled experiment would be necessary to substantiate the theory.

Conclusion

The current quantity and quality of signatures in package registries is lacking. Aside from Maven Central, all registries have less than 2% of artifacts signed. Aside from Docker Hub, at least 30% of signatures on all registries are not valid. Several factors may influence the state of signing. Among these are registry policies, dedicated tooling, and startup cost. Interestingly, cybersecurity events do not seem to have a notable effect on the adoption of signatures.

If you want to read more:

--

--

James Davis
James Davis

Written by James Davis

I am a professor in ECE@Purdue. My research assistants and I blog here about research findings and engineering tips.

No responses yet