Create deploy tokens instead of using passwords for deployment

[danielcompton]

Currently, a user's Clojars password is used to log in to Clojars, and to deploy packages.

If a users's password is leaked, an attacker can deploy packages, and log in to Clojars and change the Clojars user's email address. Occasionally Clojars passwords do get checked in to source control.

We should consider adding a concept of "deploy tokens" for Clojars which users could switch to instead of providing their password when deploying. New users would be required to use deploy tokens, and existing users could migrate over to deploy tokens, and disable the use of password deployment.

Our deploy tokens could have a prefix like CLOJARS_ which would then mean we could submit them to GitHub to be included in the token scanning system.

If we did this, we'd need to start keeping track of when password deployment was last used, so that people could safely disable it once they had moved away from providing their password. We'd also need to update any existing documentation for Clojars + third-party tooling that shows how to set your Clojars password for deployment.

Todo:

• Add tokens table & queries
• Add token auth to repo route
• Add last used tracking
• Add UI to manage tokens
• Add endpoint for GitHub to report scanning hits to
• Add outgoing email for ^
• Register token with GitHub scanner

[tobias]

The initial work on deploy tokens has been released. They are available at https://clojars.org/tokens when you are logged in. They can only be used for deploys, and should be used in place of a password (so you still need to provide a username).

Next up will be adding it to GitHub's scanner and responding to those calls. Once that is done we can discuss a date to require them for all deploys.

[tobias]

I've added an endpoint to listen for GitHub reports (f6fc331) that disables the token and sends out an email. I've emailed GitHub about enrolling in the program, but haven't yet heard back.

[hoebat]

With the deploy tokens, does it also need a gpg signing key to work?

I've passed the username and the token as the password but I'm still getting the need token message.

#req.deploy{:artifacts ["hara:base:jar:3.0.13-SNAPSHOT" "hara:base:pom:3.0.13-SNAPSHOT"], :metadata [], :repository {:authentication {:password "CLOJARS_438215b1659a4a6563911886c96d095ac734dca7c28eea9a43f42c------", :username "zcaudate"}, :content-type "default", :host "clojars.org", :id "clojars", :mirrored-repositories [], :protocol "https", :url "https://clojars.org/repo", :repository-manager? false}}

[tobias]

@zcaudate No, gpg signing isn't required. What tool are you using to deploy? I can see some successes in the logs for your username. Can you paste the output of your full deploy session?

[hoebat]

@tobias I'm using a custom deploy library that has it's own aether implementation (instead of pomegranate). I think I just need to add something else to the org.eclipse.aether.deployment.DeployRequest object.

The call to deploy is here:
https://github.com/zcaudate/hara/blob/master/src/hara/lib/aether.clj#L205

The request object is created here:
https://github.com/zcaudate/hara/blob/master/src/hara/lib/aether/request.clj#L191

This is a log from a deploy that I did just then. I've printed the request object but if there's any other information you need, let me know.

user=> (./deploy '[hara] {:tag :jep})                                                                                                                                        
                                                                                                                                                                             
---------------                                                                                                                                                              
DEPLOY PACKAGES
---------------

ITEMS (6) 

#req.deploy{:artifacts ["hara:base:jar:3.0.13-SNAPSHOT" "hara:base:pom:3.0.13-SNAPSHOT"], :metadata [], :repository {:authentication {:password "CLOJARS_438215b1659a4a6563911886c96d095ac734dca7c28eea9a43f42c7------", :username "zcaudate"}, :content-type "default", :host "clojars.org", :id "clojars", :mirrored-repositories [], :protocol "https", :url "https://clojars.org/repo", :repository-manager? false}}

  resolving     hara/base                                         <meta>                         
  downloading   hara/base                                         <meta>                         
  downloaded    hara/base                                         <meta>    [clojars]  (4.66s, 0 bytes)
  resolved      hara/base                                         <meta>    [clojars]  (4.67s, 0 bytes)
  deploying     hara/base:jar                                     3.0.13-20200707.034618-1                      
  deployed      hara/base:jar                                     3.0.13-20200707.034618-1 [clojars]  (2.31s, 91143 bytes)
  deploying     hara/base:pom                                     3.0.13-20200707.034618-1                      
  deployed      hara/base:pom                                     3.0.13-20200707.034618-1 [clojars]  (0.61s, 1222 bytes)
>> Failed to deploy artifacts: Could not transfer artifact hara:base:jar:3.0.13-20200707.034618-1 from/to clojars (https://clojars.org/repo): Unauthorized - a deploy token is required to deploy (see https://git.io/JfwjM) (401)
 1/6  hara/base             :errored                                                     8.30s    
#req.deploy{:artifacts ["hara:core.component:jar:3.0.13-SNAPSHOT" "hara:core.component:pom:3.0.13-SNAPSHOT"], :metadata [], :repository {:authentication {:password "CLOJARS_438215b1659a4a6563911886c96d095ac734dca7c28eea9a43f42c7------", :username "zcaudate"}, :content-type "default", :host "clojars.org", :id "clojars", :mirrored-repositories [], :protocol "https", :url "https://clojars.org/repo", :repository-manager? false}}

  resolving     hara/core.component                               <meta>                         
  downloading   hara/core.component                               <meta>                         
  downloaded    hara/core.component                               <meta>    [clojars]  (7.75s, 0 bytes)
  resolved      hara/core.component                               <meta>    [clojars]  (7.76s, 0 bytes)
  deploying     hara/core.component:jar                           3.0.13-20200707.034628-1                      
  deployed      hara/core.component:jar                           3.0.13-20200707.034628-1 [clojars]  (2.65s, 10240 bytes)
  deploying     hara/core.component:pom                           3.0.13-20200707.034628-1                      
  deployed      hara/core.component:pom                           3.0.13-20200707.034628-1 [clojars]  (1.23s, 1421 bytes)
>> Failed to deploy artifacts: Could not transfer artifact hara:core.component:jar:3.0.13-20200707.034628-1 from/to clojars (https://clojars.org/repo): Unauthorized - a deploy token is required to deploy (see https://git.io/JfwjM) (401)
 2/6  hara/core.component   :errored                                                     11.66s   
#req.deploy{:artifacts ["hara:io.concurrent:jar:3.0.13-SNAPSHOT" "hara:io.concurrent:pom:3.0.13-SNAPSHOT"], :metadata [], :repository {:authentication {:password "CLOJARS_438215b1659a4a6563911886c96d095ac734dca7c28eea9a43f42c7------", :username "zcaudate"}, :content-type "default", :host "clojars.org", :id "clojars", :mirrored-repositories [], :protocol "https", :url "https://clojars.org/repo", :repository-manager? false}}

  resolving     hara/io.concurrent                                <meta>                         
  downloading   hara/io.concurrent                                <meta>                         
  downloaded    hara/io.concurrent                                <meta>    [clojars]  (3.26s, 0 bytes)
  resolved      hara/io.concurrent                                <meta>    [clojars]  (3.26s, 0 bytes)
  deploying     hara/io.concurrent:jar                            3.0.13-20200707.034635-1                      
  deployed      hara/io.concurrent:jar                            3.0.13-20200707.034635-1 [clojars]  (1.33s, 6227 bytes)
  deploying     hara/io.concurrent:pom                            3.0.13-20200707.034635-1                      
  deployed      hara/io.concurrent:pom                            3.0.13-20200707.034635-1 [clojars]  (0.61s, 1434 bytes)
>> Failed to deploy artifacts: Could not transfer artifact hara:io.concurrent:jar:3.0.13-20200707.034635-1 from/to clojars (https://clojars.org/repo): Unauthorized - a deplo
y token is required to deploy (see https://git.io/JfwjM) (401)
 3/6  hara/io.concurrent    :errored                                                     5.22s    
#req.deploy{:artifacts ["hara:io.executor:jar:3.0.13-SNAPSHOT" "hara:io.executor:pom:3.0.13-SNAPSHOT"], :metadata [], :repository {:authentication {:password "CLOJARS_438215
b1659a4a6563911886c96d095ac734dca7c28eea9a43f42c7d9131", :username "zcaudate"}, :content-type "default", :host "clojars.org", :id "clojars", :mirrored-repositories [], :prot
ocol "https", :url "https://clojars.org/repo", :repository-manager? false}}

[hoebat]

fyi: code for the deploy tool is here: https://github.com/zcaudate/hara/blob/master/src/hara/deploy.clj

[tobias]

@zcaudate Thanks for the details. I suspect this is a bug in Clojars since I added new middleware to try and detect a case where a user is deploying without a token and to fail the deploy early, but it's not evident from reading the code. I just made a deploy that adds more logging to help narrow this down - would you mind trying to deploy again? It will fail, but should give me more data about what is happening.

[hoebat]

@tobias: I've done another deploy.

user=> (./deploy '[hara] {:tag :jep})                                                                                                                                        
                                                                                                                                                                             
---------------                                                                                                                                                              
DEPLOY PACKAGES                                                                                                                                                              
---------------                                                                                                                                                              
                                                                                                                                                                             
ITEMS (6)                                                                                                                                                                    
                                                                                                                                                                             
#req.deploy{:artifacts ["hara:base:jar:3.0.13-SNAPSHOT" "hara:base:pom:3.0.13-SNAPSHOT"], :metadata [], :repository {:authentication {:password "CLOJARS_438215b1659a4a656391
1886c96d095ac734dca7c28eea9a43f----------", :username "zcaudate"}, :content-type "default", :host "clojars.org", :id "clojars", :mirrored-repositories [], :protocol "https", 
:url "https://clojars.org/repo", :repository-manager? false}}                                                                                                                
                                                                                                                                                                             
  resolving     hara/base                                         <meta>                                                                                                     
  downloading   hara/base                                         <meta>                                                                                                     
  downloaded    hara/base                                         <meta>    [clojars]  (16.04s, 0 bytes)                                                                     
  resolved      hara/base                                         <meta>    [clojars]  (16.05s, 0 bytes)                                                                       deploying     hara/base:jar                                     3.0.13-20200707.135737-1                                                                                   
  deployed      hara/base:jar                                     3.0.13-20200707.135737-1 [clojars]  (4.60s, 91143 bytes)                                                   
  deploying     hara/base:pom                                     3.0.13-20200707.135737-1                      
  deployed      hara/base:pom                                     3.0.13-20200707.135737-1 [clojars]  (0.62s, 1222 bytes)
>> Failed to deploy artifacts: Could not transfer artifact hara:base:jar:3.0.13-20200707.135737-1 from/to clojars (https://clojars.org/repo): Unauthorized - a deploy token i
s required to deploy (see https://git.io/JfwjM) (401)
 1/6  hara/base             :errored                                                     21.48s   
#req.deploy{:artifacts ["hara:core.component:jar:3.0.13-SNAPSHOT" "hara:core.component:pom:3.0.13-SNAPSHOT"], :metadata [], :repository {:authentication {:password "CLOJARS_
438215b1659a4a6563911886c96d095ac734dca7c28eea9a43f----------", :username "zcaudate"}, :content-type "default", :host "clojars.org", :id "clojars", :mirrored-repositories [],
 :protocol "https", :url "https://clojars.org/repo", :repository-manager? false}}

  resolving     hara/core.component                               <meta>                         
  downloading   hara/core.component                               <meta>                         
  downloaded    hara/core.component                               <meta>    [clojars]  (3.66s, 0 bytes)
  resolved      hara/core.component                               <meta>    [clojars]  (3.66s, 0 bytes)
  deploying     hara/core.component:jar                           3.0.13-20200707.135746-1                      
  deployed      hara/core.component:jar                           3.0.13-20200707.135746-1 [clojars]  (1.22s, 10240 bytes)
  deploying     hara/core.component:pom                           3.0.13-20200707.135746-1                      
  deployed      hara/core.component:pom                           3.0.13-20200707.135746-1 [clojars]  (0.61s, 1421 bytes)
>> Failed to deploy artifacts: Could not transfer artifact hara:core.component:jar:3.0.13-20200707.135746-1 from/to clojars (https://clojars.org/repo): Unauthorized - a depl
oy token is required to deploy (see https://git.io/JfwjM) (401)
 2/6  hara/core.component   :errored                                                     5.53s    
#req.deploy{:artifacts ["hara:io.concurrent:jar:3.0.13-SNAPSHOT" "hara:io.concurrent:pom:3.0.13-SNAPSHOT"], :metadata [], :repository {:authentication {:password "CLOJARS_43
8215b1659a4a6563911886c96d095ac734dca7c28eea9a43f----------", :username "zcaudate"}, :content-type "default", :host "clojars.org", :id "clojars", :mirrored-repositories [], :
protocol "https", :url "https://clojars.org/repo", :repository-manager? false}}

  resolving     hara/io.concurrent                                <meta>                         
  downloading   hara/io.concurrent                                <meta>                         
  downloaded    hara/io.concurrent                                <meta>    [clojars]  (4.94s, 0 bytes)
  resolved      hara/io.concurrent                                <meta>    [clojars]  (4.95s, 0 bytes)
  deploying     hara/io.concurrent:jar                            3.0.13-20200707.135753-1                      
  deployed      hara/io.concurrent:jar                            3.0.13-20200707.135753-1 [clojars]  (1.23s, 6227 bytes)
  deploying     hara/io.concurrent:pom                            3.0.13-20200707.135753-1                      
  deployed      hara/io.concurrent:pom                            3.0.13-20200707.135753-1 [clojars]  (0.66s, 1434 bytes)
>> Failed to deploy artifacts: Could not transfer artifact hara:io.concurrent:jar:3.0.13-20200707.135753-1 from/to clojars (https://clojars.org/repo): Unauthorized - a deplo
y token is required to deploy (see https://git.io/JfwjM) (401)
 3/6  hara/io.concurrent    :errored

[tobias]

Hi @zcaudate - I'm still not able to see what the issue is. I can see that we are logging a nil username for your requests, which means that we either:

• can't find the authorization header
• are getting an exception when parsing the authorization header
• there is no username in the authorization header
• you aren't sending an authorization header

I've added some more logging to try to narrow it down - would you mind trying again?

[hoebat]

@tobias done.

It might be that the version of aether is old and I'm not sending auth information. I'll upgrade the version of aether and try again.

---------------                                                                                                                                                        
DEPLOY PACKAGES                                                                                                                                                        
---------------                                                                                                                                                        
                                                                                                                                                                       
ITEMS (6) 

#req.deploy{:artifacts ["hara:base:jar:3.0.13-SNAPSHOT" "hara:base:pom:3.0.13-SNAPSHOT"], :metadata [], :repository {:authentication {:password "", :username "zcaudate"}, :content-type "default", :host "clojars.org", :id "clojars", :mirrored-repositories [], :protocol "https", :url "https://clojars.org/repo", :repository-manager? false}}

  resolving     hara/base                                         <meta>                         
  downloading   hara/base                                         <meta>                         
  downloaded    hara/base                                         <meta>    [clojars]  (5.68s, 0 bytes)
  resolved      hara/base                                         <meta>    [clojars]  (5.68s, 0 bytes)
  deploying     hara/base:jar                                     3.0.13-20200708.130908-1                      
  deployed      hara/base:jar                                     3.0.13-20200708.130908-1 [clojars]  (3.03s, 91143 bytes)
  deploying     hara/base:pom                                     3.0.13-20200708.130908-1                      
  deployed      hara/base:pom                                     3.0.13-20200708.130908-1 [clojars]  (0.93s, 1222 bytes)
>> Failed to deploy artifacts: Could not transfer artifact hara:base:jar:3.0.13-20200708.130908-1 from/to clojars (https://clojars.org/repo): Unauthorized - a deploy t
oken is required to deploy (see https://git.io/JfwjM) (401)
 1/6  hara/base             :errored                                                     10.47s   
#req.deploy{:artifacts ["hara:core.component:jar:3.0.13-SNAPSHOT" "hara:core.component:pom:3.0.13-SNAPSHOT"], :metadata [], :repository {:authentication {:password "", :username "zcaudate"}, :content-type "default", :host "clojars.org", :id "clojars", :mirrored-repositories [], :protocol "https", :url "https://clojars.org/repo", :repository-manager? false}}

  resolving     hara/core.component                               <meta>                         
  downloading   hara/core.component                               <meta>                         
  downloaded    hara/core.component                               <meta>    [clojars]  (4.16s, 0 bytes)
  resolved      hara/core.component                               <meta>    [clojars]  (4.16s, 0 bytes)
  deploying     hara/core.component:jar                           3.0.13-20200708.130916-1                      
  deployed      hara/core.component:jar                           3.0.13-20200708.130916-1 [clojars]  (1.74s, 10623 bytes)
  deploying     hara/core.component:pom                           3.0.13-20200708.130916-1                      
  deployed      hara/core.component:pom                           3.0.13-20200708.130916-1 [clojars]  (1.02s, 1421 bytes)
>> Failed to deploy artifacts: Could not transfer artifact hara:core.component:jar:3.0.13-20200708.130916-1 from/to clojars (https://clojars.org/repo): Unauthorized - 
a deploy token is required to deploy (see https://git.io/JfwjM) (401)
 2/6  hara/core.component   :errored 

[hoebat]

I think I'm on the latest - 1.1.0 - it hasn't changed since 2016.

[tobias]

@zcaudate thanks for your patience on this, I think I have figured it out and released a fix.

It looks like your client first makes an unauthenticated request, then waits for the 401 response with a WWW-Authenticate header before sending credentials. This should be totally fine, but other clients used to deploy to clojars don't do this -- they include the Authorization header with every request -- so I wasn't aware that I broke the initially-unauthenticated flow.

I've fixed that issue and deployed it - would you mind trying yet again?