keboola / openlineage-generator
Generates jobs data for a service with OpenLineage API
1.1.0
2023-10-30 08:47 UTC
Requires
- php: ^8.1
- guzzlehttp/guzzle: ^7.0
- psr/log: ^1.1
Requires (Dev)
- keboola/coding-standard: ^15.0
- php-parallel-lint/php-parallel-lint: ^1.3
- phpstan/phpstan: ^1.4
README
This library will generate jobs metadata obtained from Job Queue API open-api-lineage endpoint for a data lineage service supporting OpenLineage API (e.g. Marquez).
Installation
The library is available as composer package. To start using this library in your project follow these steps:
Install package:
composer require keboola/openlineage-generator
Add autoloader in your bootstrap script:
require 'vendor/autoload.php';
Read more in Composer documentation
Usage
$openLineageWriter = new OpenLineageWriter( $queueClient, // keboola job queue client $openLineageClient, // http client for open lineage api $logger, $createdTimeFrom, //DateTimeImmutablee object from which jobs will be imported $config->getOpenLineageEndpoint(), // url of open lineage api $config->getJobNameAsConfig() // boolean, if true, job name will be used as config name ); $openLineageWriter->write();
Development
Clone this repository and init the workspace with following command:
git clone https://github.com/keboola/openlineage-generator
cd openlineage-generator
docker-compose build
docker-compose run --rm dev composer install --no-scripts
Run the CI checks using this command:
docker-compose run --rm dev composer ci
Integration
For information about deployment and integration with KBC, please refer to the deployment section of developers documentation
License
MIT licensed, see LICENSE file.