March, 2018 – Blog & Insights

This guide will present a step-by-step guide to solving common connectivity problems using SSH tunnels. To read some useful comments and context, skip to the end.

Here we will use ssh local port forwarding to PULL the service port through the ssh connection. The ssh tunnel command (to be run from A) is:

Step 0: If box A can already hit box B:8080, then good for you. Otherwise, follow the steps below to make that happen.

Step 1: If box A can ssh into box B, read the following section. If not, go to Step 2.

Here we will use ssh local port forwarding to PULL the service port through the ssh connection. The ssh tunnel command (to be run from A) is:

user@A > ssh -vCNnTL A:8080:B:8080 user@B

This pulls the service port over to box A at port 8080, so that anyone connecting to A:8080 will transparently get their requests forwarded over the the actual server on B:8080.

Step 2: If box B can ssh into box A, read the following section. Otherwise, skip to step 3.

Now we will use ssh remote port forwarding to PUSH the service port through the ssh connection. The command (to be run from B) is

user@B > ssh -vCNnTR localhost:8080:B:8080 user@A

Now users on A hitting localhost:8080 will be able to connect to B:8080. Users not on either A or B will still be unable to connect.

To enable listening on A:8080, you have 2 options:

A) If you have sudo, add the following line to /etc/ssh/sshd_config and reload the sshd service:

GatewayPorts clientspecified

Then rerun the above command with “localhost” replaced by “A”:

user@B > ssh -vCNnTR A:8080:B:8080 user@A

B) Pretend “localhost (A)” is another box, and apply step 1, since A can generally ssh into itself:

user@A > ssh -vCNnTL A:8080:localhost:8080 user@localhost

Now we come to the situation where neither A nor B can ssh into the other.

Step 3: If there are any TCP ports that allow A, B to connect, continue reading. Otherwise, move on to step 4.

Suppose that B is able to connect to A:4040. Then the way to allow B to ssh into A is to turn A:4040 into an ssh port. This is doable by applying Step 1 on A itself, to pull the ssh service on 22 over to listen on A:4040:

user@A > ssh -vCNnTL A:4040:A:22 user@A

And then you can apply Step 2, specifying port 4040 for ssh itself:

user@B > ssh -vCNnTL localhost:8080:B:8080 user@A -p 4040

Similarly, if A is able to connect to B:4040, you’ll want to forward B:22 to B:4040 using Step 1, then apply Step 1 again as usual.

Step 4: Find a box C which has some sort of connectivity to both A and B, and continue reading. Otherwise, skip to step 10.

If A and B have essentially no connectivity, then the way to proceed is to route through another box.

Step 5: If C is able to hit B:8080, continue reading. Otherwise, skip to step 9.

Step 6: If A is able to SSH to C, continue reading. Otherwise, skip to step 7.

This is very similar to step 1 — we will pull the connection through the SSH tunnel using LOCAL forwarding.

user@A > ssh -vCNnTL A:8080:B:8080 user@C

Step 7: If C is able to SSH to A, continue reading. Otherwise, skip to step 8.

Again, this is analogous to step 2 — we will push the connection through the SSH tunnel using REMOTE ssh forwarding.

user@C: ssh -vCNnTR localhost:8080:B:8080 user@A

Just as before, we will need to add an additional forwarding step to listen on a public A interface rather than localhost on A:

user@A: ssh -vCNnTL A:8080:localhost:8080 user@localhost

Step 8: If neither C nor A can ssh to each other, but there are TCP ports open between the 2:

Apply the technique in step 3. Otherwise, skip to step 10.

Now we are in the situation where C can’t hit B:8080 directly.

Step 9: The general idea is to first connect C:8080 to B:8080 using one of Steps 1 or 2, and then do the same to connection A:8080 to C:8080.

Note that it doesn’t matter which order you set up these connections.

9a) If C can ssh to B and C can ssh to A. This is a very common scenario – maybe C is your local laptop which is connected to 2 separate VPNs.

First pull B:8080 to C:1337 and then push C:1337 to A:8080:

user@C > ssh -vCNnTL localhost:1337:B:8080 user@B
user@C > ssh -vCNnTR localhost:8080:localhost:1337 user@A
user@C > ssh user@A
user@A > ssh -vCNnTL A:8080:localhost:8080 user@localhost

9b) If C can ssh to B and A can ssh to C: Again, a fairly common scenario if you have a “super-private” network accessible only from an already private network.

Do two pulls in succession:

user@C > ssh -vCNnTL localhost:1337:B:8080 user@B
user@A > ssh -vCNnTL A:8080:localhost:1337 user@C

You can actually combine these into a single command:

user@A : ssh -vC -A -L A:8080:localhost:1337 user@c ‘ssh -vCNnTL localhost:1337:B:8080 user@B’

9c) If B can ssh to C and C can ssh to A: Double-push.

user@B > ssh -vCNnTR localhost:1337:B:8080 user@C
user@C > ssh -vCNnTR localhost:8080:localhost:1337 user@A
user@C > ssh user@A
user@A > ssh -vCNnTL A:8080:localhost:8080 user@A

Again these can be combined into a single command.

9d) If B can ssh to C and A can ssh to C: Push from B and then pull from A.

user@B > ssh -vCNnTR localhost:1337:B:8080 user@C
user@A > ssh -vCNnTL A:8080:localhost:1337 user@C

Any number of these may also need the trick from Step 3 to enable SSH access.

Step 10: If box C doesn’t have any TCP connectivity to either B or A,then having box C doesn’t really help the situation at all. You’ll need to find a different box C which actually has connectivity to both and return to step 4, or find a chain (C, D, etc.) through which you could eventually patch a connection through to B. In this case you’ll need a series of commands such as those in 9a)-d) to gradually patch B:8080 to C:1337, to D:42069, etc., until you finally end up at A:8080.

Addendum 1: If your service uses UDP rather than TCP (this includes dns, some video streaming protocols, and most video games), you may have to add a few steps to convert to TCP; see https://superuser.com/questions/53103/udp-traffic-through-ssh-tunnel for a guide.

Addendum 2: If your service host and port come with url signing (e.g. signed s3 urls), changing your application to hit A:8080 rather than B:8080 may cause the url signatures to fail. To remedy this, you can add a line in your /etc/hosts file to redirect B to A; since your computer checks this file before doing a DNS lookup, you can do completely transparent ssh tunneling while still respecting SSL and signed s3/gcs urls.

Addendum 3: My preferred tools for checking which TCP ports are open between 2 boxes are nc -l 4040 on the receiving side and curl B:4040 on the sending side. ping B, traceroute B, and route -n are also useful for diagnostic information but may not tell you the full story.

Addendum 4: SSH tunnels will not work if there is something already listening on that port, such as the SSH tunnel you created yesterday and forgot to remove. To easily check this, try ps -efjww | grep ssh or sudo netstat -nap | grep LISTEN.

Addendum 5: All the ssh flags are explained in the man page: man ssh. To give a brief overview: -v is verbose logging, -C is compression, -nNT together disable the interactive part of ssh and make it only tunnel, -A forwards over your ssh credentials, -f backgrounds ssh after connecting, and -L and -R are for local and remote forwarding respectively. -o StrictHostKeyChecking=no is also useful for disabling the known-hosts check.

FURTHER COMMENTS

SSH tunnels are useful as a quick-fix solution to networking issues, but are generally recognized as inferior solutions compared to long-term proper networking fixes. They tend to be difficult to maintain for an number of reasons: the setup does not require any configuration and leaves no trace other than a running process; they don’t automatically come up when restarting a box — except if manually added to the startup daemon; and they can easily be killed by temporary network outages.

However we’ve made good use of them here at Hive, for instance most recently when we needed to keep up production services during a network migration, but also occasionally when provisioning burst GPU resources from AWS and integrating them seamlessly into our hardware resource pool. They also can be very useful when developing locally or debugging production services, or for getting gmail access in China.

If you’re interested in different and more powerful ways to tunnel, I’m no networking expert — all I can do is point you in the direction of some interesting networking vocabulary.