File permissions for the `datalab` user getting corrupted, which blocks the `datalab connect` command from working.

[YauKeith]

Hey,

I am running an error (as shown in the below screenshot)

I've tried running the suggested solution found on a previous thread but nothing seems to work.

[ojarjur]

That is the generic error that is reported if there are any issues creating an SSH tunnel to your instance.

Things to check:

1. Is your instance running? The datalab connect command restarts the instance if it is not running, but it is still good to check this.
2. Do the firewall rules for your GCP network look right? There should be a single firewall rule named datalab-network-allow-ssh (if you are using the datalab-network network) that allows incoming traffic to port 22.
3. Can you SSH directly to the VM without using datalab connect? I.E. try running gcloud compute ssh <INSTANCE_NAME> and see if that works.
4. Can you connect from a different machine? For instance, you could try running the datalab connect ... command from inside of Cloud Shell.
5. If the above all fail, then can you connect to the instance using the SSH button in the Cloud Console?

[YauKeith]

Hey Omar, Thanks for replying to my issue. Replying directly to the questions you posed. 1. The instance is indeed running. I've tried stopping and restarting before connecting but still failed. 2. Checked the firewall too, using " gcloud compute firewall-rules list", datalab-network-allow-ssh allows "tcp:22" 3. The system is able to run the command "gcloud compute ssh <instance name>" but I'm not sure what am I to do beyond this step to help validate it. 4. I am able to connect to a new datalab I created but that has the same issue occurring after about a week. 5. I am able to connect via the SSH button , but similar to step 3 I am not sure what else I am to do beyond clicking the button.

…

On Thu, May 31, 2018 at 6:13 AM, Omar Jarjur ***@***.***> wrote: That is the generic error that is reported if there are any issues creating an SSH tunnel to your instance. Things to check: 1. Is your instance running? The datalab connect command restarts the instance if it is not running, but it is still good to check this. 2. Do the firewall rules for your GCP network look right? There should be a single firewall rule named datalab-network-allow-ssh (if you are using the datalab-network network) that allows incoming traffic to port 22. 3. Can you SSH directly to the VM without using datalab connect? I.E. try running gcloud compute ssh <INSTANCE_NAME> and see if that works. 4. Can you connect from a different machine? For instance, you could try running the datalab connect ... command from inside of Cloud Shell. 5. If the above all fail, then can you connect to the instance using the SSH button in the Cloud Console? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2014 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AlO31tHnxcZ_v3WINb9jOnvCWDZyNfTuks5t3xmhgaJpZM4UPjAf> .

-- *Keith Yau* Analytics

-- ------------------------------------------------------- essenceglobal.com <https://www.essenceglobal.com> Facebook <http://www.facebook.com/essenceglobal> • Twitter <http://www.twitter.com/essenceglobal> • YouTube <http://www.youtube.com/essenceglobal> • Instagram <http://www.instagram.com/essence_global>

[ojarjur]

@YauKeith Since the SSH connections are being created, the issue must be something inside of the Datalab instance itself.

Here are some follow up things to check:

1. What version of the Cloud SDK do you have installed? You can see this by running gcloud version.
2. Run gcloud compute ssh datalab@<INSTANCE> to confirm that there are no issues with the datalab user on the VM.
3. Once you've SSH'ed in, run docker ps -a to list the running Docker containers. If there is not a datalab container, then something may be preventing it from running inside of your VM.

[YauKeith]

Hey Omar, 1. GCloud SDK version is : Google Cloud SDK 202.0.0 2. Running this code returns the same error as before. (pasted below) Permission denied (publickey). ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. 3. I connected using the SSH button and ran the code, it returned 2 containers as shown in the screenshot. [image: ssh.PNG]

…

On Fri, Jun 1, 2018 at 1:11 AM Omar Jarjur ***@***.***> wrote: @YauKeith <https://github.com/YauKeith> Since the SSH connections are being created, the issue must be something inside of the Datalab instance itself. Here are some follow up things to check: 1. What version of the Cloud SDK do you have installed? You can see this by running gcloud version. 2. Run gcloud compute ssh datalab@<INSTANCE> to confirm that there are no issues with the datalab user on the VM. 3. Once you've SSH'ed in, run docker ps -a to list the running Docker containers. If there is not a datalab container, then something may be preventing it from running inside of your VM. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2014 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AlO31niz5wVFBkkOVnLpDJezQFF004aGks5t4CRWgaJpZM4UPjAf> .

-- *Keith Yau* Senior Analytics Executive 158 Cecil St #11-01 • Singapore • 069545

-- ------------------------------------------------------- essenceglobal.com <https://www.essenceglobal.com> Facebook <http://www.facebook.com/essenceglobal> • Twitter <http://www.twitter.com/essenceglobal> • YouTube <http://www.youtube.com/essenceglobal> • Instagram <http://www.instagram.com/essence_global>

[ojarjur]

@YauKeith OK, if you can't SSH in as the datalab user, then that explains the problem you are seeing.

Next, we need to figure out what is wrong with the datalab user in your VM.

SSH in using the gcloud tool (without the datalab@ prefix to the instance name), and then check the following:

1. What are the permissions for the /home/datalab directory? Does the datalab user have permission to write to it? Alternatively, you can just grant the user permission by running sudo chown -R datalab:datalab /home/datalab
2. Can you change to the datalab user after SSH'ing in? Try running the gcloud compute ssh <INSTANCE> comand, and then running sudo su datalab.
3. What are the permissions for the /home/datalab/.ssh directory? It should also be owned by the datalab user.

[YauKeith]

Hey Omar, I think I might be a little confused with the steps for the troubleshooting this time round, but here is what I did and what was returned. - Within the gcloud shell, I typed " gcloud compute ssh <instance name>". 1. Typed "/home/datalab ", I was returned with "bash: /home/datalab: Is a directory" 1a. Thereafter, I typed the "sudo chown -R datalab:datalab /home/datalab" and nothing happened. 2. Running "sudo su datalab" changed the line to "datalab@instance /home/instance" 3. And running "/home/datalab/.ssh" returned with "bash: /home/datalab/.ssh: Is a directory" Not sure if I'm doing anything wrong here or is this the ex[ected response from the system.

…

On Sat, Jun 2, 2018 at 5:42 AM Omar Jarjur ***@***.***> wrote: @YauKeith <https://github.com/YauKeith> OK, if you can't SSH in as the datalab user, then that explains the problem you are seeing. Next, we need to figure out what is wrong with the datalab user in your VM. SSH in using the gcloud tool (without the datalab@ prefix to the instance name), and then check the following: 1. What are the permissions for the /home/datalab directory? Does the datalab user have permission to write to it? Alternatively, you can just grant the user permission by running sudo chown -R datalab:datalab /home/datalab 2. Can you change to the datalab user after SSH'ing in? Try running the gcloud compute ssh <INSTANCE> comand, and then running sudo su datalab. 3. What are the permissions for the /home/datalab/.ssh directory? It should also be owned by the datalab user. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2014 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AlO31pTK0kh05bRf9Y3rOoW3fChMFTEYks5t4bUogaJpZM4UPjAf> .

-- *Keith Yau* Senior Analytics Executive

-- ------------------------------------------------------- essenceglobal.com <https://www.essenceglobal.com> Facebook <http://www.facebook.com/essenceglobal> • Twitter <http://www.twitter.com/essenceglobal> • YouTube <http://www.youtube.com/essenceglobal> • Instagram <http://www.instagram.com/essence_global>

[ojarjur]

@YauKeith Sorry my previous comment omitted too many details.

Here is my line of thought regarding your issue:

1. The fact that you can SSH in to the VM as the default user (which will be named based on your authenticated account in gcloud), but not as the user datalab indicates that there is something wrong with the datalab account on your VM.
2. Since the datalab connect ... command relies on SSH'ing in as the datalab user, this explains the problem you are seeing.
3. In the past, when we've seen this sort of issue the cause has been related to the datalab user's home directory in the VM. That directory is /home/datalab. The issue we've seen in the past is that directory having the wrong permissions (e.g. being owned by root instead of datalab).
4. The sudo chown -R datalab:datalab /home/datalab command is meant to fix such issues. It is not normally expected to return any results (i.e. it is silent on success).

Now that you've run that command, does the gcloud compute ssh datalab@<INSTANCE> still fail, or does it succeed?

[YauKeith]

Hey Omar, Good news & Bad news on the issue. Good news is I am able to ssh into the VM after the sudo chown code yesterday, but bad news is that it failed again when I tried to directly connect to the VM. Using the sudo chown code seem to fix the issue again. I am honestly happy with the fix (albeit the small detour) but out of curiosity is there are other possible troubleshooting procedures I can look at?

…

On Tue, Jun 5, 2018 at 1:28 AM Omar Jarjur ***@***.***> wrote: @YauKeith <https://github.com/YauKeith> Sorry my previous comment omitted too many details. Here is my line of thought regarding your issue: 1. The fact that you can SSH in to the VM as the default user (which will be named based on your authenticated account in gcloud), but *not* as the user datalab indicates that there is something wrong with the datalab account on your VM. 2. Since the datalab connect ... command relies on SSH'ing in as the datalab user, this explains the problem you are seeing. 3. In the past, when we've seen this sort of issue the cause has been related to the datalab user's home directory in the VM. That directory is /home/datalab. The issue we've seen in the past is that directory having the wrong permissions (e.g. being owned by root instead of datalab). 4. The sudo chown -R datalab:datalab /home/datalab command is meant to fix such issues. It is not normally expected to return any results (i.e. it is silent on success). Now that you've run that command, does the gcloud compute ssh datalab@ <INSTANCE> still fail, or does it succeed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2014 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AlO31rewO5Qw9XZ9jbVYa17PS0nToksxks5t5W5NgaJpZM4UPjAf> .

-- *Keith Yau* Senior Analytics Executive

-- ------------------------------------------------------- essenceglobal.com <https://www.essenceglobal.com> Facebook <http://www.facebook.com/essenceglobal> • Twitter <http://www.twitter.com/essenceglobal> • YouTube <http://www.youtube.com/essenceglobal> • Instagram <http://www.instagram.com/essence_global>

[ojarjur]

@YauKeith the next step should be to figure out what is messing up the permissions on that directory.

The first thing I would do is (after the next time it fails and before running the chown command) run ls -al /home/datalab and ls -al /home/datalab/.ssh.

That will tell us what the messed up permissions are, and hopefully give some hint into what is changing the permissions.

[YauKeith]

Hey Omar, I haven't had any trouble with accessing the Datalab post the last we discussed. I guess the permission is granted and have stuck. Thanks for all the help over the week!

…

On Wed, Jun 6, 2018 at 3:10 AM Omar Jarjur ***@***.***> wrote: @YauKeith <https://github.com/YauKeith> the next step should be to figure out what is messing up the permissions on that directory. The first thing I would do is (after the next time it fails and before running the chown command) run ls -al /home/datalab and ls -al /home/datalab/.ssh. That will tell us what the messed up permissions are, and hopefully give some hint into what is changing the permissions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2014 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AlO31uxyfaSh5CWNB9piF0gBnyi26099ks5t5teegaJpZM4UPjAf> .

-- *Keith Yau* Senior Analytics Executive 158 Cecil St #11-01 • Singapore • 069545

-- ------------------------------------------------------- essenceglobal.com <https://www.essenceglobal.com> Facebook <http://www.facebook.com/essenceglobal> • Twitter <http://www.twitter.com/essenceglobal> • YouTube <http://www.youtube.com/essenceglobal> • Instagram <http://www.instagram.com/essence_global>

[ojarjur]

@YauKeith Glad to hear it is working for you now.

[usul83]

Hi, since the cause of this was never found (and I'm having problems) here is the output from
ls -al /home/datalab:
drwxr-xr-x 4 owner owner 4096 Jun 1 08:25 .
drwxr-xr-x 9 root root 4096 Aug 14 14:46 ..
-rw-r--r-- 1 owner owner 127 May 11 00:37 .bash_logout
-rw-r--r-- 1 owner owner 204 May 11 00:37 .bash_profile
-rw-r--r-- 1 owner owner 551 May 11 00:37 .bashrc
drwx------ 2 owner owner 4096 Jun 1 08:25 .docker
drwx------ 2 datalab datalab 4096 Jun 1 08:25 .ssh

and ls -al /home/datalab/.ssh:
drwx------ 2 datalab datalab 4096 Jun 1 08:25 .
drwxr-xr-x 4 owner owner 4096 Jun 1 08:25 ..
-rw------- 1 datalab datalab 4887 Aug 17 09:24 authorized_keys

running the chmod command also works for me, but the issue keeps reappearing

[ojarjur]

@usul83 That is very strange; it indicates that there is a user account in your instance named owner.

Nothing in our code base uses that account name.

Similarly, nothing running inside of Datalab should be able to change that (i.e. your notebook code should not be able to cause this).

Are you running the stock version of datalab installed with the Cloud SDK or a modified version? If it is the stock version, what version is it? You can get this information from running gcloud version.

Additionally, what version of the tool was used to create your instance? You can find this by running datalab connect --diagnose-me and then looking for the two lines in the output after it says Instance <YOUR_INSTANCE_NAME> was created with the following Cloud SDK component versions:.

[usul83]

It's standard datalab 20180503 and Cloud SDK: 201.0.0 .

I've noticed that this problem tends to occur after an idle timeout.

I recall having some problems connecting to my instance and resorting to creating a new one with the same disk, not sure if that explains the owner.

I've also often had the same error message and solved the problem by deleting the SSH key.