How to access Cloud Compute instance Google as a service through SSH?

I'm trying to access a Compute instance through SSH in my bitbucket-pipeline, but for some reason, anything I try to do doesn't work, so I'm obviously missing something. I created a service account in my Google Cloud project. I gave this service following IAM permissions:

• Compute OS Admin Login
• Compute OS Login

In my pipeline I have a following code:

apk add --no-cache openssh-client python2-dev
wget -O ~/gc-sdk.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-261.0.0-linux-x86_64.tar.gz
tar xvzf ~/gc-sdk.tar.gz -C ~/
export PATH="$PATH:/root/google-cloud-sdk/bin"
~/google-cloud-sdk/install.sh -q
GCLOUD_KEY_FILE=/root/key.json
echo "${GCR_JSON_KEY}" > ${GCLOUD_KEY_FILE}
gcloud auth activate-service-account --key-file ${GCLOUD_KEY_FILE}

It is tested and works. I'm authorized as a service, which I can test by being able to pull docker images from Container Registry in Google Cloud (with proper permissions set for this service).

In my Compute Engine's metadata I have:

enable-oslogin  TRUE

And finally in my bitbucket-pipeline I have an SSH key configured, that is also added to my Compute Engine's metadata (I think this shouldn't be required if oslogin is enabled?).

However if I try to run gcloud compute ssh --project myproject myinstance I'm getting:

Permission denied (publickey).
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]

What step is missing here for me to be able to just SSH into my Compute Engine instance as a service?

Added: I discovered that it actually works when I give my service the Editor role for the whole project. It's quite over-permissive though, but at least now I know that the issue is with permissions. So what are the least permissive IAM settings for this to work? I tried few from Compute Engine category but I wasn't able to figure it out.