keboola / storage-backend
Requires
- php: ^8.1
- ext-json: *
- ext-odbc: *
- ext-pdo: *
- aws/aws-sdk-php: ^3
- doctrine/dbal: ^3.3
- google/cloud-bigquery: ^1.23
- google/cloud-storage: ^1.27
- google/protobuf: ^3.21
- keboola/common-exceptions: ^1
- keboola/csv-options: ^1
- keboola/php-csv-db-import: ^6
- keboola/php-file-storage-utils: ^0.2.2
- keboola/php-temp: ^2.0
- keboola/php-utils: ^4.1
- keboola/retry: ^0.5.0
- microsoft/azure-storage-blob: ^1.4
- symfony/console: ^5.2|^6
- symfony/filesystem: ^5.2|^6
- symfony/process: ^4.4|^5.0|^6.0
Requires (Dev)
- keboola/coding-standard: ^15
- keboola/phpunit-retry-annotations: ^0.3.0
- php-parallel-lint/php-parallel-lint: ^1.3
- phpstan/extension-installer: ^1.1
- phpstan/phpdoc-parser: ^1.6
- phpstan/phpstan: ^1.4
- phpstan/phpstan-phpunit: ^1
- phpstan/phpstan-symfony: ^1.1
- phpunit/phpunit: ^9
- react/async: ^4||^3
- squizlabs/php_codesniffer: ^3
- symfony/finder: ^5.4
- symplify/monorepo-builder: 11.1.30.72
- dev-main
- dev-jirka-ct-1424-remove-dependency-on-native-types-feature
- dev-jirka-ct-1424-drop-fractional-seconds
- dev-odin-KAB-312
- dev-zajca-fix-distinct-on-nonnative-tables-2
- dev-zajca-ct-1642
- dev-zajca-ct-950-ignore-columns
- dev-zajca-fix-wrong-tests
- dev-zajca-tag.php
- dev-big-160-update-common
- dev-zajca-big-171
- dev-revert-84-zajca-ct-1118
- dev-zajca-ct-1118
- dev-BIG-126-external-buckets-2
- dev-BIG-126-external-buckets
- dev-big-153-roman-improve-type-hint
- dev-zajca-BIG-157
- dev-zajca-new-err-code
- dev-zajca-big-142
- dev-zajca-fix-zero-length
- dev-zajca-kbc-1003
- dev-zajca-CT-666-snflk-null
- dev-php81
- dev-roman-finish-release
- dev-ct-835-fixx-export-null
- dev-ct-843-fix-numeric-value-is-empty-string
- dev-CT-843-null-import
- dev-PST-631_SNFLK-add-missing-types
- dev-add-ie-lib-repo
- dev-roman-improve-docker
This package is auto-updated.
Last update: 2024-11-19 21:03:53 UTC
README
Storage backend related tools monorepo
Packages included in this repo:
Adding new repository
If you want to add a new repository to monorep, please follow the instructions.
Provisioning of resources
Some libraries need to install backend drivers that are stored on S3 for this purpose we create a cloudformation template. This cloudformation template is managed manually on the corresponding AWS stack for CI. When adding a new library to the monorepo, we can use the base role for building and use as in main.yml on line 32, or add the new role to the cloudformation template and assign the necessary resources to it.
- name: Configure AWS Credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: ${{ env.AWS_CREDENTIALS_ROLE_TO_ASSUME }} aws-region: ${{ env.AWS_CREDENTIALS_REGION }}
How to use monorepo and packages
We maintaining the monorepo with the symplify/monorepo-builder.
To download drivers stored on S3 use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (Export them or use --build-arg option in docker compose build command) from the AWS account "Dev - Main Legacy" -> create user namesurname-kbc-drivers-download and add policy "DriversDownload").
The base docker service is dev
To install dependencies use:
docker compose build dev docker compose run dev composer install docker compose run dev vendor/bin/monorepo-builder
The individual package services in the monorepo are then prefixed with the word dev-
.
So if we want to install composer for example for the php-table-backend-utils
package we use the command:
docker compose build dev-php-table-backend-utils docker compose run dev-php-table-backend-utils composer install
and run for example unit tests like this:
docker compose run dev-php-table-backend-utils composer tests-unit
By separating the packages and installing each of them separately when we run the previous command. Because the other packages in monorep are installed from packagist.org, so individual customizations are not shared between packages.
For example, php-table-backend-utils
required on php-datypes
, but if I make a change to php-datatypes
and
call docker compose run dev-php-table-backend-utils composer tests-unit
the changes will not be reflected in the php-table-backend-utils
tests.
Running tests locally
To be able to run tests to include all changes and packages requested via composer we can use global phpunit.
docker compose run dev vendor/bin/phpunit packages/php-table-backend-utils/tests/Unit/
However, by creating a new CLI interpreter in PHPStorm we will achieve that the changes will be reflected in the whole monorepo.
So for each lib we will create our own interpreter. Also add env variables for each interpreter separately when creating a new interpreter.
For example for lib php-table-backend-utils
as seen in the picture.
- Then you need to configure Test Framework in
PHP->Test Frameworks
- Click on + and select PHPUnit by Remote Interpreter
- And in path to script select the path to
keboola/storage-backend/vendor/autoload.php
- Click ok
Release
If you want to release a new version of a package, you need to
- merge the PR to
main
- on main branch create and push tag
<package>/<newVersion>
. So e.g.php-datatypes/7.7.0
if you want to releasephp-datatypes
in version7.7.0