EDIT: I posted a TL;DR version of this (including screen-shots!) as a LinkedIn article here: https://www.linkedin.com/pulse/making-changes-git-repository-access-codeready-workspaces-vic-cross. If you’ve come here from that article, thanks for your interest in the details of my experience!
As part of the setup of an environment for a customer workshop, I installed Red Hat CodeReady Workspaces (RHCRW) in an OpenShift cluster on IBM Z. My Z environment is in a testing lab without direct Internet connectivity and that creates some challenges — easily overcome, but it does mean that nothing is quite straightforward…
In setting up the environment I was guided by a fantastic workshop demo created by Filipe Miranda and Patrick Fruth, two of my IBM Z Acceleration Team colleagues. The demo involves using RHCRW to work on code held in a Git repository on our GitHub Enterprise. This was a challenge for me since both RHCRW and GHE lean heavily toward SSH-based access to Git when pushing to the repository is involved.
My super-secret lab
The network setup in my lab requires the use of the OpenSSH ProxyJump configuration to hop through a “bastion” system to reach any SSH server outside the isolated network. I already had a Red Hat Enterprise Linux (RHEL) system I use for accessing GitHub repositories over SSH via the command line, and on such a system it was easy for me to add the required configuration to my ~/.ssh/config file:
Host github.com
ProxyJump viccross@10.1.2.3
Host github.example.com
ProxyJump viccross@10.1.2.3I also use SSH-key-based authentication to ensure I don’t have to enter passwords, and SSH Agent Forwarding to avoid having my private key stored on multiple remote servers.
So knowing that it’s possible to create an SSH configuration underneath git to proxy the SSH connectivity, I was stuck on how to do the same thing in my RHCRW workspace. I could open a Terminal in my workspace, create my files, and make the test logon to GHE as expected (I did add the default SSH key from my workspace to my user-ID on the jump-host, to avoid giving the workspace access to my SSH private key). However, my repository access was still not working because my Terminal window was running in a different container to the Theia containers that held the Git functionality of the workspace.
A different approach
Pat Fruth came to my rescue, suggesting a capability that allows settings, in the form of files or environment variables, to be injected into all the containers that make up a workspace. It is done by creating a secret, with appropriate labels, in the OpenShift project where your workspace will run. The process is documented (for CodeReady Workspaces 2.9) at https://access.redhat.com/documentation/en-us/red_hat_codeready_workspaces/2.9/html-single/end-user_guide/index#mounting-a-secret-as-a-file-or-an-environment-variable-into-a-workspace-container_crw.
Using my existing approach of modifying my user SSH config file as a starting point, I set about trying to reproduce my non-CRW configuration exactly. The RHCRW documentation described a process to push a file into home directory config areas using a {prod-home} syntax in the secret… which didn’t work — no config file was created. I couldn’t find any documentation on the {prod-home} syntax, but thought it might have been just as likely that the ~/.ssh directory doesn’t already exist in the containers.
If at first you don’t succeed…
I decided to try injecting the file at the system-level instead. RHEL includes an /etc/ssh/ssh_config.d/ directory that looks perfect for the task, since I didn’t want to replace any existing configuration. So I created a secret to inject a 00-localproxy.conf file into the directory. I recreated the workspace, and found that my new configuration file was created successfully! The Git repository was still not being pulled, though, and testing logon to GHE was failing as if the ProxyJump setting was not present…
To investigate, I looked at the documentation for the ssh command. By the man page, the SSH client will look at ~/.ssh/config or /etc/ssh/ssh_config for client configuration options — no mention of /etc/ssh/ssh_config.d/. I looked at my local RHEL system, and found that the directory is activated through an Include statement in the /etc/ssh/ssh_config file. Looking at that file on a Terminal in my RHCRW workspace, I saw that it had been replaced by some other configuration settings applied by CRW and the Include statement was not present. Foiled!
…try, try again
Since it now looked like modifying the SSH environment transparently to Git wasn’t going to be an option, I looked at ways to affect the way git was invoked to supply options to the SSH client. For a few versions, the git command has supported the use of the GIT_SSH environment variable to supply an alternative command to perform the SSH connection. Rather than providing alternative options or parameters though, GIT_SSH needs to point to a program (or script) that is invoked by git. To use this, I would have to write a script that invoked ssh with the right parameters and then inject that script into the workspace, along with injecting the environment variable as well. This approach just seemed to create more questions for me though: where to put the script (is /usr/local/bin in the PATH in these containers)? Is SELinux going to block me at some point?
I then found a more recent addition to OpenSSH, the environment variable GIT_SSH_COMMAND. Rather than pointing to a program or script like GIT_SSH, GIT_SSH_COMMAND provides git with a command string to use to invoke SSH. This gave me exactly what I needed! I created the following secret in my RHCRW workspaces project:
apiVersion: v1
kind: Secret
metadata:
name: ssh-jump-command
labels:
app.kubernetes.io/component: workspace-secret
app.kubernetes.io/part-of: che.eclipse.org
annotations:
che.eclipse.org/automount-workspace-secret: 'true'
che.eclipse.org/env-name: GIT_SSH_COMMAND
che.eclipse.org/mount-as: env
data:
mykey: ssh -J viccross@10.1.2.3
… or at least I tried to add the secret this way: I got an error about the format of the key I had provided. In the examples I’d seen about adding files, the file data had to be Base64 encoded. The environment variable examples didn’t mention that, but it is still a requirement. So I jumped to a command prompt and did the following:
$ echo ssh -J viccross@10.1.2.3
c3NoIC1KIHZpY2Nyb3NzQDEwLjEuMi4zCg==
…and dutifully pasted the resulting Base64 into my secret, which now successfully was created.
Guess what happened when I recreated my RHCRW workspace…
Getting it right in the end
The eagle-eyed amongst you will already know that, once again, it didn’t work. First thing I checked was whether the environment variable was present. I got the following when I checked:
$ set | grep GIT_SSH
GIT_SSH_COMMAND=$'ssh -J viccross@10.1.2.3\n'
Hmm, well I was glad that the process had worked and my environment variable was created, but why didn’t it look right? The clue is the \n: I had forgotten that echo adds an end-of-line to its output, and base64 had dutifully included that into its encoded output. This in turn meant that the environment variable had the line-end character embedded, and any SSH transaction initiated by git was doomed.
To fix this, I had to remove that line-end from the variable. I ran echo with the -n switch to suppress it:
$ echo -n ssh -J viccross@10.1.2.3 c3NoIC1KIHZpY2Nyb3NzQDEwLjEuMi4z
I pasted this string into my secret to update it, then recreated my workspace… and when I logged on I was greeted with a message that Git had successfully cloned the repository!
The final secret, which worked, looked like this:
apiVersion: v1
kind: Secret
metadata:
name: ssh-jump-command
labels:
app.kubernetes.io/component: workspace-secret
app.kubernetes.io/part-of: che.eclipse.org
annotations:
che.eclipse.org/automount-workspace-secret: 'true'
che.eclipse.org/env-name: GIT_SSH_COMMAND
che.eclipse.org/mount-as: env
data:
mykey: c3NoIC1KIHZpY2Nyb3NzQDEwLjEuMi4z
To break it down for you:
- kind is Secret, which identifies a secret;
- name gives the secret a friendly name among the rest of the secrets created in the workspace;
- the two labels are required by CRW, and identify this secret as something that has to be considered for mounting into the workspace containers. These have to be included verbatim, in order for the function to work (they are the defaults though; there is another setting at the RHCRW level that specifies the names of the labels that must be used. I’m a “use the defaults if there’s no reason to change them” kind of guy, so I didn’t seek this out);
- the annotation che.eclipse.org/automount-workspace-secret controls whether the secret will actually be mounted or not. The value of true says that it will — this can be used at the secret level to override a setting that generally excludes mounting;
- che.eclipse.org/mount-as is set here to env, which means treating this secret as details for an environment variable to be created. The value file would tell the system to treat it as a file to be created (and a couple of the other settings would be different too);
- Since we’ve said it’s an environment variable, the system will expect che.eclipse.org/env-name to contain the name of the environment variable to be created;
- the data setting contains the Base64-encoded value the environment variable will contain (it’s not documented if the mykey name is required or just a placeholder).
There is a syntax for including multiple environment variables in a single secret, should you need to do that. However you can’t create both files and environment variables in the same secret (in fact when creating files, it’s one file per secret).
Onward to working with CodeReady Workspaces
I have little experience with modern IDEs. My most advanced setup of the last couple of years was using Eclipse with the SVN and Bugzilla plugins to manage some Perl code I was running. I have both VS Code and Atom on my system (and of course vim), but haven’t had much opportunity to customise them. So I’m probably a ripe target to be won over by capabilities that help me in development.
I’m looking forward to what I might be able to use RHCRW for, but also what I might be able to build to show off the Z and LinuxONE platforms.
Crossed Wires 