Concepts

blackline_project.yaml

The blackline_project.yaml file defines the configuration for the Blackline project. It contains project-level settings and is located in the root directory of the project.

The project-level settings available are defined by the ProjectConfig model. The configuration below shows a project with the default settings.

blackline_project.yaml

name: <name> # Required
config-version: 1
version: 0.1.0

default-profile: default

catalogue-path: ./catalogue
adapters-path: ./adapter

Catalogue

A catalogue is defines but a set folder structure and respective yaml file definitions for each of the organizations, systems, resources, and datasets. The structure is designed to be flexible and scalable, and can be adapted to suit a wide range of applications and use cases. Each folder can have multiple nested children.

The catalogue folder structure consists of a series of nested directories, starting with a top-level directory catalogue. Inside this directory, you can create subdirectories for different organizations, each of which contains subdirectories for specific systems and resources. Within each resource directory, you can create subdirectories for datasets and collections, as needed.

Here's an overview of the folder structure:

catalogue/
└── <organization_name>/
    ├── <system_name>/
    │   ├── <resource_name>/
    │   │   ├── <dataset_name>/
    │   │   │   ├── <collection_name>.yaml
    │   │   │   ├── ...
    │   │   │   └── dataset.yaml
    │   │   ├── ...
    │   │   └── resource.yml
    │   ├── ...
    │   └── system.yaml
    ├── ...
    └── organization.yaml

Folder Structure Details

catalogue/ The catalogue/ directory is the top-level directory for the catalogue, and should contain all of the subdirectories for the different organizations.

<organization_name>/ Each organization should have its own subdirectory within the catalogue/ directory. The name of the subdirectory should be the name of the organization. The organization.yml file should be located in the top-level catalogue/ directory, and should contain information about the organization as a whole.

<system_name>/ Within each organization directory, you can create subdirectories for different systems that the organization uses. The name of the subdirectory should be the name of the system. The system.yml file should be located in each system directory, and should contain information about the system.

<resource_name>/ Within each system directory, you can create subdirectories for different resources that the system uses. The name of the subdirectory should be the name of the resource. The resource.yml file should be located in each resource directory, and should contain information about the resource.

<dataset_name>/ Within each resource directory, you can create subdirectories for different datasets that the resource uses. The name of the subdirectory should be the name of the dataset. Each dataset directory should also contain a dataset.yml file with information about the dataset itself. The name of the file should be dataset.yaml.

<collection_name>.yaml Each <dataset_name> directory should contain a <collection_name>.yml file with information about the collection of data. The name of the file should be the name of the collection, with the .yaml extension.

Adapters

The folder structure for adapters mirrors to the catalogue folder structure but includes a dataset_name.yaml file that defines the connection for the dataset's adapter.

The adapters folder structure consists of a series of nested directories, starting with a top-level directory called adapters. Inside this directory, you can create subdirectories for different organizations, each of which contains subdirectories for specific systems and resources. Within each resource directory, you can create a YAML file that defines the connection for the dataset's adapter. The names for the organization, system, and resource MUST match.

Here's an overview of the folder structure:

adapters/
└── <organization_name>/
    ├── <system_name>/
    │   ├── <resource_name>/
    │   │   ├── <dataset_name>.yaml
    │   │   └── ...
    │   └── ...
    └── ...

Folder Structure Details

adapters/ The adapters/ directory is the top-level directory for the adapters, and should contain all of the subdirectories for the different organizations.

<organization_name>/ Each organization should have its own subdirectory within the adapters/ directory. The name of the subdirectory should be the name of the organization.

<system_name>/ Within each organization directory, you can create subdirectories for different systems that the organization uses. The name of the subdirectory should be the name of the system.

<resource_name>/ Within each system directory, you can create subdirectories for different resources that the system uses. The name of the subdirectory should be the name of the resource. Within each resource directory, you can create a <dataset_name>.yaml file that defines the connection for the dataset's adapter. The name of the file should be the name of the dataset, with the .yaml extension.

The contents of the dataset_name.yaml file should include the necessary connection information for the adapter, such as the type of adapter, the database URL, username, and any other required parameters.