Dirk.Net

Update: Two hours after this post I discovered that the system clock on the remote server, just recently booked from my reliable hosting company, was off by about eight hours. On a server set up by them just a few days ago. Anyway, the clock issue might be the actual underlying reason for the failed authentication.  So check your clock and sync it. I guess the name of the lone horseman is Crystaltech. Duh.

Original post:

Ok, the title may be a bit harsh, but that’s how I felt at times over the last two days: Some evil forces slicing days out of my life without me getting much in return – besides learning a lot. 

So if you follow the instructions here and here  with perhaps a little sprinkling or CentOS you set up the most powerful web app deployment solution available today, a combination of your own git repository server (gitosis) and powerful deployment scripts (Capistrano). Then, after gitosis administration via gitosis-admin.git works like magic and cap deploy:setup runs without issues, you run cap deploy:cold.

And you need old git’s password?!! The gitosis user you’ve painstakingly set up without as password, i.e. the —disabled-password option (-r in CentOS), so who is positively devoid of a password, and you still get

 ** [myhost.com :: out]
 ** [myhost.com :: out] Permission denied, please try again.
 ** [myhost.com :: out] git@myhost.com’s password:
Password:
 ** [myhost.com :: out]
 ** [myhost.com :: out] Permission denied (publickey,gssapi-with-mic,password).
 ** [myhost.com :: out] fatal: The remote end hung up unexpectedly
    command finished

ad nauseam. And in your log (/var/log/secure for CentOS) you’ll have

Mar  1 04:27:25 HOST1 sshd[12653]: Postponed publickey for git from 210.11.11
.11 port 25409 ssh2
Mar  1 04:27:26 HOST1 sshd[12652]: Accepted publickey for git from 210.11.11.
11 port 25409 ssh2
Mar  1 04:27:30 HOST1 sshd[12657]: Accepted publickey for deploy from 210.11.1
1.11 port 25412 ssh2
Mar  1 04:28:15 HOST1 sshd[12663]: Failed password for git from 63.22.22.22
 port 34759 ssh2
Mar  1 04:28:16 HOST1 sshd[12663]: Failed password for git from 63.22.22.22
 port 34759 ssh2

where the 210.11.11.11 is the ip of your development machine, and 63.22.22.22 is the ip of your server running gitosis which is also the deployment target server (This ip-difference made me think of iptables, oh well.).

Now, before you mess with

• iptables
• ssh-agent
• sshd_config
• authorized_keys and .ssh permissions
• reinstalling and upgrading gitosis and other stuff

and anything else you might try, first, simply add

on :start do
    `ssh-add`
end

(mind the backquotes) to your deploy.rb file. And it might just start working. Here’s the solution, with the original suggestion here. I re-discovered it as I was calling it a night, closing one of the 132 Firefox tabs I had opened in this effort.

The issue is probably not Capistrano’s fault but has to do with the specific ssh setup on the server. I had no such problems with an Ubuntu slice as now on a dedicated CentOS, perhaps due to a more severe security lockdown on this “Enterprise” distro.  However, it didn’t make any difference for me if ssh-agent was running or if I had

default_run_options[:pty] = true

in my deploy.rb file or or or…. YMMV.

First and foremost, read this document and follow it meticulously. If you get this error:

unable to chdir or not a git archive
fatal: The remote end hung up unexpectedly

after running

cap deploy:cold

you might have to add the public key of the deploy user you set up with

set :user, “deploy”

in your capistrano deploy.rb file, to the /home/git/.ssh/authorized_keys on the REMOTE server.

Of course gitosis has a special process managing the keys and users from the LOCAL machine in the ~/gitosis/gitosis-admin directory, so you to have copy the public key you generated for the deploy user on your REMOTE server via

ssh-keygen -t rsa

to your LOCAL machine (via scp or cut and paste into a new deploy.pub file from the ssh shell for example) and then proceed as outlined here (under “Adding Users”):

cd ~/gitosis/gitosis-admin
cp ~/deploy.pub keydir/
git add keydir/deploy.pub

Add the user credentials to the gitosis.conf file:

[group myrailsteam]
writable = myrailsapp
members = deploy

Finally run

git commit -a -m “Granted deploy user access rights to myrailsapp”
git push

Now the cap deploy:cold should work (with one fewer errors that is ).

After extensive setup work to deploy your Ruby on Rails app with capistrano and after you put 

set :user, “deploy”

as a dedicated deploy user into your deploy.rb as many advise, you finally get to run

cap deploy:setup

Then this error stares back at you:

 servers: ["mydomain.com"]
*** [mydomain.com] could not open channel
    command finished
failed: nil on mydomain.com

Check you auth.log on your remote server

tail -f /var/log/auth.log

and look at the message:

Accepted publickey for myuser

This myuser, who lacks the necessary deployment permissions on your server, might be set in your ~/.ssh/config file, possibly after you set a custom port for ssh (sample custom port 30440 used from here on):

Host mydomain.com
User myuser
Port 30440
[…other settings…]

Some say this file (~/.ssh/config) is ignored by capistrano, but this does not seem to be the case, and capistrano will always make the ssh connection with the first user found in the ~/.ssh/config file, so it doesn’t help if you add

Host mydomain.com
User deploy
Port 30440
[…other settings…]

to the file or to add

set :admin_runner, “deploy“

to your deploy.rb, rather you can have only

Host mydomain.com
User deploy
Port 30440
[…other settings…]
in your ~/.ssh/config file for this particular host if you want to specify the user here.

As a side note, capistrano has no problems connecting on a custom ssh port if you set

ssh_options[:port] = 30440

so it might be time to clean out your ~/.ssh/config file for the hosts you deploy to with capistrano.

Specifying the port however won’t work with gitosis and git, i.e. you can’t have

set :repository, “ssh://git@mydomain.com:30440/myrepository.git”

in your deploy.rb file, as it will throw a

ssh: mydomain.com:30440: Name or service not known

, likely because of a gitosis issue. As a workaround you might just put

Host mydomain.com
Port 30440
[…other settings but no user specified…]

without any user settings into your ~/.ssh/config file for this particular host, and you’ll (hopefully) be able to deploy via a custom port with gitosis and capistrano.

Hope it helps.