Some notes on using the Google Cloud storage.
Install client Library#
We need to install the gcloud storage client to use the storage API. For Python, use pip to install the client:
pip install google-cloud-storage
test in local env#
When we run the gcloud storage client in local PC and want to connect to our project, we need to do the authentication:
gcloud auth application-default login
Follow the instructions and finish the authentication. Then we can use the client locally for testing.
use the correct bucket name#
Note that bucket names refer to the top level folders when you click Cloud Storage --> Buckets.
Under each bucket, you can define sub-folder structures.
However, when we want to get a bucket, these sub-folders should not be part of the bucket name!!
check whether a file exists?#
from google.cloud import storage
storage_client = storage.Client()
bucket_name = "my-bucket"
bucket = storage_client.bucket(bucket_name)
file_path = "path/to/your_file/under/the/bucket"
blob = bucket.blob(file_path)
print(blob.exists())
ref:
- check whether a file exists in gcloud: https://stackoverflow.com/q/13525482/6064933
Save Python dict in gcloud storage?#
We need to convert the dict to string using JSON package and then use the method provided by blob to upload it. Here is a working code example:
import json
from google.cloud import storage
storage_client = storage.Client()
bucket_name = "my-bucket"
bucket = storage_client.bucket(bucket_name)
file_path = "path/to/your_file/under/the/bucket"
blob = bucket.blob(file_path)
content = {"message": "hello world", 'code': 0}
blob.upload_from_string(json.dumps(content))
ref:
download file and convert to dict#
We can download the file as string and then convert it to the desired type:
import json
from google.cloud import storage
storage_client = storage.Client()
bucket_name = "my-bucket"
bucket = storage_client.bucket(bucket_name)
file_path = "path/to/your_file/under/the/bucket"
blob = bucket.blob(file_path)
content = blob.download_as_string()
content_real = json.loads(content)
ref: