keboola / storage-driver-bigquery
Keboola BigQuery driver
Requires
- php: ^8.1
- ext-json: *
- google/apiclient: ^2.12.1
- google/apiclient-services: 0.282.0
- google/cloud-bigquery: ^1.23
- google/cloud-bigquery-analyticshub: ^0.1.0
- google/cloud-billing: ^1.4
- google/cloud-resource-manager: ^0.6.1
- google/cloud-service-usage: ^0.2.7
- google/protobuf: ^3.21
- keboola/db-import-export: >=2.7.0
- keboola/php-file-storage-utils: ^0.2.5
- keboola/retry: ^0.5.1
- keboola/storage-driver-common: ^6.9
- keboola/table-backend-utils: >=2.8.0
- psr/log: ^1.1|^2.0|^3.0
- react/async: ^3.0
- symfony/polyfill-php80: ^1.26
Requires (Dev)
- brianium/paratest: ^6.10
- google/cloud-bigquery-connection: ^1.5
- keboola/coding-standard: ^15.0
- keboola/phpunit-retry-annotations: ^0.5.0
- php-parallel-lint/php-parallel-lint: ^1.3
- phpstan/phpstan: ^1.8
- phpstan/phpstan-phpunit: ^1.1
- phpstan/phpstan-symfony: ^1.2
- phpunit/phpunit: ^9.5
- symfony/finder: ^5.4
- symfony/lock: ^6.3
- dev-main
- v7.8.7
- v7.8.6
- v7.8.5
- v7.8.4
- v7.8.3
- v7.8.2
- v7.8.1
- v7.8.0
- v7.7.2
- v7.7.1
- v7.7.0
- v7.6.0
- v7.5.0
- v7.4.3
- v7.4.2
- v7.4.1
- v7.4.0
- v7.3.0
- v7.2.4
- v7.2.3
- v7.2.2
- v7.2.1
- v7.2.0
- v7.1.1
- v7.1.0
- v7.0.1
- v7.0.0
- v6.0.1
- v6.0.0
- v5.5.1
- v5.5.0
- v5.4.0
- v5.3.4
- v5.3.3
- v5.3.2
- v5.3.1
- v5.3.0
- v5.2.17
- v5.2.16
- v5.2.15
- v5.2.14
- v5.2.13
- v5.2.12
- v5.2.11
- v5.2.10
- v5.2.9
- v5.2.8
- v5.2.7
- v5.2.6
- v5.2.5
- v5.2.4
- v5.2.3
- v5.2.2
- v5.2.1
- v5.2.0
- v5.1.3
- v5.1.2
- v5.1.1
- v5.1.0
- v5.0.0
- v4.0.1
- v4.0.0
- v3.4.1
- v3.4.0
- v3.3.0
- v3.2.0
- v3.1.1
- v3.1.0
- v3.0.5
- v3.0.4
- v3.0.3
- v3.0.2
- v3.0.1
- v3.0.0
- v2.14.0
- v2.13.1
- v2.13.0
- v2.12.0
- v2.11.0
- v2.10.0
- v2.9.0
- v2.8.0
- v2.7.0
- v2.6.0
- v2.5.0
- v2.4.0
- v2.3.0
- v2.2.0
- v2.1.0
- v2.0.0
- v1.2.0
- v1.1.0
- v1.0.0
- v0.1.1
- v0.1
- dev-jirka-fix-node
- dev-jirka-CT-1805-force-nullable-columns
- dev-ct-1482-view-in-dest-project
- dev-ct-1482-manual-add-subscriber
- dev-zajca-fix-throwexception
- dev-tf-test-bq-monorepo
- dev-big-245-fix-creating-job-query
- dev-big-123-filter-out-ws-dataset-from-other-ws
- dev-BIG-113-array_conversion
- dev-roman-create-exchanger-when-creating-project
This package is auto-updated.
Last update: 2024-10-24 07:25:17 UTC
README
Keboola high level storage backend driver for Big Query
Install dependencies.
docker compose run --rm dev composer install
Setup Big Query
Install Google Cloud client (via Brew), initialize it and log in to generate default credentials.
To prepare the backend use Terraform template. Create a sub folder in the KBC Team Dev (id: 431160969986) folder and fill the folder into the terraform command.
- get missing pieces (organization_id and billing_id) from Connection repository.
- (optional) move
bq-storage-backend-init.tf
out of project directory so new files would be out of git - Run
terraform init
- Run
terraform apply -var folder_id=[folder_id] -var billing_account_id=[billing_id] -var backend_prefix=<your prefix, eg. kbc-js> -var file_storage_backend_region=<desired region>
(e.g. us-central1 region)- Optionally set
terraform.tfvars
file with variables there is predefinedterraform.tfvars.dist
file
- Optionally set
- New key files was created
principal_key.json
andbig_query_key.json
- Set envs from the
principal_key.json
andbig_query_key.json
by runningphp SetBQVars.php
and stop. OR continue with following steps with manual extraction. - open
principal_key.json
set content ofprivate_key
as variableBQ_SECRET
and remove (the whole entry) it from json file- note: simply cut&paste it whole even with the quotes and new lines -> your .env will be like
BQ_SECRET="-----BEGIN PRIVATE KEY-----XXXXZQ==\n-----END PRIVATE KEY-----\n"
- note: simply cut&paste it whole even with the quotes and new lines -> your .env will be like
- remove line breaks from the rest of key file (without
private_key
entry) and set this string as variableBQ_PRINCIPAL
to.env
- You can convert the key to string with
awk -v RS= '{$1=$1}1' principal_key.json
- You can convert the key to string with
- remove line breaks from the
big_query_key.json
key file and set this string as variableBQ_KEY_FILE
to.env
- You can convert the key to string with
awk -v RS= '{$1=$1}1' big_query_key.json
- You can convert the key to string with
At the end, your .env
file should look like...
# the id is printed by terraform at the end and it is just the numbers after `folders/` BQ_PRINCIPAL=<the content of the principal_key.json key file as single line without private_key entry> BQ_SECRET=<private_key from principal_key.json key file (taken from BQ_PRINCIPAL)> BQ_FOLDER_ID=<TF output file_storage_bucket_id : the id of the created folder, just the number, without /folders prefix> BQ_BUCKET_NAME=<TF output file_storage_bucket_id : bucket id created in main project> # choose different BQ_STACK_PREFIX than you Terraform prefix otherwise project created by Terraform will be deleted . e.g. local :) BQ_STACK_PREFIX=local BQ_KEY_FILE=<big_query_key.json file owned by main service acc>
All done. Now you can try composer loadGcs
script and run tests.
docker compose run --rm dev composer loadGcs
Build docker images
docker compose build
Xdebug
To run with xdebug use dev-xdebug
container instead of dev
Tests
Run tests with following command.
# This will run all tests docker compose run --rm dev composer tests # This will run all tests in parallel docker compose run --rm dev composer paratest # This will run import tests in parallel docker compose run --rm dev composer paratest-import # This will run export tests in parallel docker compose run --rm dev composer paratest-export # This will run all tests in parallel excluding import and export docker compose run --rm dev composer paratest-other
To disable retry copy phpunit-retry.xml.dist
cp phpunit-retry.xml.dist phpunit-retry.xml
Code quality check
#run all bellow but not tests docker compose run --rm dev composer check #phplint docker compose run --rm dev composer phplint #phpcs docker compose run --rm dev composer phpcs #phpcbf docker compose run --rm dev composer phpcbf #phpstan docker compose run --rm dev composer phpstan
Full CI workflow
This command will run all checks and run tests
docker compose run --rm dev composer ci
Using
Project ID: A globally unique identifier for your project. This lib creating project id as combinations of stackPrefix
and projectId
from CreateProjectCommand
A project ID is a unique string used to differentiate your project from all others in Google Cloud. You can use the Google Cloud console to generate a project ID, or you can choose your own. You can only modify the project ID when you're creating the project.
Project ID requirements:
- Must be 6 to 30 characters in length.
- Can only contain lowercase letters, numbers, and hyphens.
- Must start with a letter.
- Cannot end with a hyphen.
- Cannot be in use or previously used; this includes deleted projects.
- Cannot contain restricted strings, such as
google
andssl
.
License
MIT licensed, see LICENSE file.