Questions[gsutil](server)

I have internet connection on the ESXi server and I want to copy everything to GCP storage. But Gcloud couldn't authenticate. It looks like it can't reach the gloud login servers.

I'm not if the esxi provider has blocked GCP IP's but is there a way I can verify? Do we know which IP or DNS address it tries to connect?

Could this be a different issue? I tried disable ipv6 and was hoping it would work.

./gcloud init --skip-diagnostics

    ERROR: There was a problem with web authentication.
    ERROR: (gcloud.auth.login) Could not reach the login server. A potential cause of this could be because you are behind a proxy. Please set the environment variables HTTPS_PROXY and HTTP_PROXY to the address of the proxy in the format "protocol://address:port" (without quotes) and try again.
    Example: HTTPS_PROXY=https://192.168.0.1:8080

I have a bucket with a file whose size is larger than 11 GB. I note that I can access all bucket's data from the standard google cloud console. Unfortunately, this console has very limited configurations since, I believe, it was primarily thought of as a tool to manage files and small operations within google cloud.

To better analyze my data I create a VM instance with 32 GB RAM. The problem is: once I am inside the VM instance I can no longer access the data from my bucket. After a bit of research I figure out that I can copy files from my bucket to my instance using:

gsutil -m cp -r gs://<my-bucket>/* .

I thought I have found a solution, but the operation stop after reaching 39%:

- [1/2 files][  4.4 GiB/ 11.2 GiB]  39% Done     0.0 B/s 

I don't know what configuration my VM must have to complete this operation successfully or if the problem is with the VM at all.

Now, I just start to use google cloud and I'm in no way sure that I'm doing the right thing here. Is this the standard procedure for use bucket data within a VM instance? How can I copy large files from bucket to my VM instance?

Trying to use gsutil today I got the following error:

+ gsutil cp -nra public-read a gs://b

/usr/local/lib/python2.7/dist-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
  CryptographyDeprecationWarning,
Traceback (most recent call last):
  File "/usr/lib/google-cloud-sdk/bin/bootstrapping/gsutil.py", line 13, in <module>
    import bootstrapping
  File "/usr/lib/google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 44, in <module>
    from googlecloudsdk.core.credentials import store as c_store
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/credentials/store.py", line 40, in <module>
    from googlecloudsdk.core.credentials import creds as c_creds
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/credentials/creds.py", line 34, in <module>
    from googlecloudsdk.core.credentials import devshell as c_devshell
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/credentials/devshell.py", line 33, in <module>
    from google.oauth2 import credentials
  File "/usr/lib/google-cloud-sdk/lib/third_party/google/oauth2/credentials.py", line 43, in <module>
    from google.oauth2 import _client
  File "/usr/lib/google-cloud-sdk/lib/third_party/google/oauth2/_client.py", line 35, in <module>
    from google.auth import jwt
  File "/usr/lib/google-cloud-sdk/lib/third_party/google/auth/jwt.py", line 57, in <module>
    from google.auth import _service_account_info
  File "/usr/lib/google-cloud-sdk/lib/third_party/google/auth/_service_account_info.py", line 22, in <module>
    from google.auth import crypt
  File "/usr/lib/google-cloud-sdk/lib/third_party/google/auth/crypt/__init__.py", line 43, in <module>
    from google.auth.crypt import rsa
  File "/usr/lib/google-cloud-sdk/lib/third_party/google/auth/crypt/rsa.py", line 20, in <module>
    from google.auth.crypt import _cryptography_rsa
  File "/usr/lib/google-cloud-sdk/lib/third_party/google/auth/crypt/_cryptography_rsa.py", line 25, in <module>
    from cryptography.hazmat.primitives import serialization
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/primitives/serialization/__init__.py", line 22, in <module>
    from cryptography.hazmat.primitives.serialization.ssh import (
  File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/primitives/serialization/ssh.py", line 27, in <module>
    from bcrypt import kdf as _bcrypt_kdf
  File "/usr/local/lib/python2.7/dist-packages/bcrypt/__init__.py", line 57
    def gensalt(rounds: int = 12, prefix: bytes = b"2b") -> bytes:
SyntaxError: invalid syntax

I'm running gsutil with Python 2 (export CLOUDSDK_GSUTIL_PYTHON=python2) and it seems like the tool or the OS installed bcrypt 3.2.0 over night which dropped support for Python 2.

This breaks the gsutil tool.

I have a VM running on Google cloud that I would like to start and stop.

I am able to start/stop it from another VM in the same project using the command

sudo gcloud compute instances stop my-daily-data-vm --zone us-central1-c

Now, I would like to start/stop it from my Windows desktop. (FYI I do have gsutil installed on it). Is there a way I can do this?

Thank you :)

I get the following when I attempt to list the buckets for the account I'm logged in with:

$ gsutil ls -l gs://bucket
AccessDeniedException: 403 Forbidden

$ gcloud config list
Your active configuration is: [default]

[compute]
region = us-central1
zone = us-central1-b
[core]
account = [email protected]
disable_usage_reporting = True
project = bar-proj

.. which correctly reflects that I've logged in ( as I expected. )

I can see three buckets at cloud.google.com, and at least one ( let's call it console-bucket, as it was created directly from the console ) has at least these permissions:

project owners-[projectnumber] - owner
user    [email protected]          - owner

Another one ( lets call it gsutil-bucket ) has at least this permission:

project owners-[projectnumber] - owner

As you might guess, it was created using gsutil ( with the very same permissions that fail to list the bucket. )

It is possible that there could be a proxy issue, but I would find that strange because the gsutil mb command successfully created a bucket that I can see in the console.

Additionally, [email protected] has the role of owner to bar-project, which was configured and displays under project permissions.

... What am I missing?

What call is gsutil trying to make, to what IP, and what port? ( I expect that the port is 443, but if I'm wrong, and it's also not 80, that would explain why it's forbidden ... )