Resource

A resource is a crucial element within a system that is typically associated with a specific business function. It represents a single entity that can either store or process data. Each resource is designed to perform a specific function within a system. A resource can be a microservice or a single product from a cloud provider.

A resource is defined in the catalogue/\<organization name\>/\<system name\>/\<resource name/>/resource.yml file. The resource.yml file accepts all arguments defined in the Resource model with the expection of:

1. The key is created using the catalogue folder structure.
2. The datasets dictionary is created using the catalogue folder strucutre.

Example

# resource.yaml

resource:
  key: resource_key
  tags:
  - foo
  - bar
  name: <resource name>
  description: <resource description>
  resource_type: Service
  data_responsibility_title: Processor
  privacy_declarations:
  - name: Privacy Policy
    data_categories:
    - user.device.cookie_id
    - user.contact
    data_use: improve.system
    data_subjects:
    - Customer
    data_qualifier: Aggregated
  dependencies:
  - system_key
  joint_controller:
    name: Dave
    address: Museumplein 10, 1071 DJ Amsterdam, Netherlands
    email: dave@organization.com
    phone: 020 573 2911
  third_country_transfers:
  - USA
  - CAN
  administrating_department: engineering
  data_protection_impact_assessment:
    is_required: true
    status: Complete
    link: https://example.org/analytics_system_data_protection_impact_assessment

Models

blackline.models.catalogue.Resource

Bases: BlacklineModel

The System resource model.

Describes an application and includes a list of PrivacyDeclaration resources.

Parameters:

Name	Type	Description	Default
resource_type	Optional[ResourceTypeEnum]	A required value to describe the type of system being modeled	None
data_responsibility_title	Optional[DataResponsibilityTitle]		None
privacy_declarations	Optional[list[PrivacyDeclaration]]		None
dependencies	Optional[list[Key]]	A list of keys to model dependencies.	None
joint_controller	Optional[ContactDetails]		None
third_country_transfers	Optional[list[str]]	An optional array to identify any countries where data is transited to. For consistency purposes, these fields are required to follow the Alpha-3 code set in ISO 3166-1.	None
administrating_department	Optional[str]	The department or group that owns the resource	None
data_protection_impact_assessment	Optional[DataProtectionImpactAssessment]		None
children	Optional[dict[str, Dataset]]	Dataset dict	None
stem			required
children_stem			required
children_cls			required
_sort_privacy_declarations	classmethod		required
_no_self_reference	classmethod		required
_check_valid_country_code	classmethod		required

Source code in BAR /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

class Resource(BlacklineModel):
    """
    The System resource model.

    Describes an application and includes a list of PrivacyDeclaration resources.
    """

    resource_type: Optional[ResourceTypeEnum] = Field(
        description="A required value to describe the type of system being modeled",
    )
    data_responsibility_title: Optional[DataResponsibilityTitle] = Field(
        default=DataResponsibilityTitle.CONTROLLER,
        description=DataResponsibilityTitle.__doc__,
    )
    privacy_declarations: Optional[list[PrivacyDeclaration]] = Field(
        description=PrivacyDeclaration.__doc__,
    )
    dependencies: Optional[list[Key]] = Field(
        description="A list of keys to model dependencies."
    )
    joint_controller: Optional[ContactDetails] = Field(
        description=ContactDetails.__doc__,
    )
    third_country_transfers: Optional[list[str]] = Field(
        description="An optional array to identify any countries where data is transited to. For consistency purposes, these fields are required to follow the Alpha-3 code set in ISO 3166-1.",
    )
    administrating_department: Optional[str] = Field(
        description="The department or group that owns the resource",
    )
    data_protection_impact_assessment: Optional[DataProtectionImpactAssessment] = Field(
        default=DataProtectionImpactAssessment(),
        description=DataProtectionImpactAssessment.__doc__,
    )

    children: Optional[dict[str, Dataset]] = Field(description="Dataset dict", alias="datasets")  # type: ignore[assignment]

    stem = "resource"
    children_stem = "datasets"
    children_cls = Dataset

    _sort_privacy_declarations: classmethod = validator(
        "privacy_declarations", allow_reuse=True
    )(sort_list_objects_by_name)

    _no_self_reference: classmethod = validator(
        "dependencies", allow_reuse=True, each_item=True
    )(no_self_reference)

    _check_valid_country_code: classmethod = country_code_validator

    class Config:
        "Class for the System config"
        use_enum_values = True

Config

Class for the System config

Source code in BAR /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

class Config:
    "Class for the System config"
    use_enum_values = True

blackline.models.catalogue.BlacklineModel

Bases: BaseModel

The base model for all Resources.

Parameters:

Name	Type	Description	Default
key	Key	A unique key used to identify this resource.	required
tags	Optional[list[str]]	A list of tags for this resource.	None
name	Optional[str]		None
description	Optional[str]		None
children	Optional[dict[str, Type[BlacklineModel]]]	The children resources.	None
stem	str	The stem of the resource.	required
children_stem	Optional[str]		required
children_cls	Optional[type[BlacklineModel]]		required

Source code in BAR /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

class BlacklineModel(BaseModel):
    """The base model for all Resources."""

    key: Key = Field(description="A unique key used to identify this resource.")
    tags: Optional[list[str]] = Field(description="A list of tags for this resource.")
    name: Optional[str] = name_field
    description: Optional[str] = description_field
    children: Optional[dict[str, Type[BlacklineModel]]] = Field(
        None, description="The children resources."
    )
    stem: ClassVar[str] = Field(description="The stem of the resource.")
    children_stem: ClassVar[Optional[str]] = None
    children_cls: ClassVar[Optional[type[BlacklineModel]]] = None

    class Config:
        "Config for the BlacklineModel"
        extra = "forbid"
        orm_mode = True

    def __getitem__(self, key: str) -> Type[BlacklineModel]:
        parts = key.split(".")
        key = ".".join([self.key, parts[0]])
        if self.children is None:
            raise KeyError(f"No children for {self.key}")
        model = self.children[key]

        for part in parts[1:]:
            model = model[part]  # type: ignore[index]
        return model

    @classmethod
    def parse_dir(cls, path: Path, key_prefix: Optional[str] = None):
        """
        Parse a directory of YAML files into a dictionary of Dataset objects.

        Args:
            path: The path to the directory of YAML files.
            path: Path

        Returns:
            A dictionary of Dataset objects.
        """
        key = ".".join([key_prefix, path.name]) if key_prefix is not None else path.name
        children = cls.parse_children(path=path, key_prefix=key)
        filepath = cls.find_definition_file(path=path)
        return cls.parse_yaml(path=filepath, key=key, children=children)

    @classmethod
    def parse_children(
        cls, path: Path, key_prefix: Optional[str] = None
    ) -> dict[str, Type[BlacklineModel]]:
        """
        Parse a directory of YAML files into a dictionary of Dataset objects.

        Args:
            path: The path to the directory of YAML files.
            path: Path

        Returns:
            A dictionary of Dataset objects.
        """
        children: dict[str, Type[BlacklineModel]] = {}
        if cls.children_cls is None:
            return children
        for child_path in path.iterdir():
            if child_path.is_dir():
                child = cls.children_cls.parse_dir(
                    path=child_path, key_prefix=key_prefix
                )
                children[child.key] = child
        return children

    @classmethod
    def find_definition_file(cls, path: Path) -> Path:
        file = list(path.glob(f"{cls.stem}.yml")) + list(path.glob(f"{cls.stem}.yaml"))
        file_len = len(list(file))
        if file_len == 0:
            raise FileNotFoundError(
                f"No {cls.stem} file found in directory: {path.absolute()}"
            )
        if file_len > 1:
            raise ValueError(
                f"Multiple {cls.stem} files found in directory: {path.absolute()}, only include one of resource.yaml or resource.yml"
            )
        return file[0]

    @classmethod
    def parse_yaml(
        cls,
        path: Path,
        key: str,
        children: Optional[dict[str, Type[BlacklineModel]]] = {},
    ):
        """
        Parse a yaml file into a the children_cls object.

        Args:
            path: Path location of the yaml file.
            key: Key to identify the dataset.

        Returns:
            Dataset object.
        """
        with open(path, "r") as f:
            info = yaml.safe_load(f)[cls.stem][0]
            info["key"] = key
            if cls.stem == "dataset":
                return cls.parse_obj(info)
            info[cls.children_stem] = children
            return cls.parse_obj(info)

Config

Config for the BlacklineModel

Source code in BAR /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

class Config:
    "Config for the BlacklineModel"
    extra = "forbid"
    orm_mode = True

parse_children(path, key_prefix=None) classmethod

Parse a directory of YAML files into a dictionary of Dataset objects.

Parameters:

Name	Type	Description	Default
path	Path	The path to the directory of YAML files.	required
path	Path	Path	required

Returns:

Type	Description
dict[str, Type[BlacklineModel]]	A dictionary of Dataset objects.

Source code in /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

@classmethod
def parse_children(
    cls, path: Path, key_prefix: Optional[str] = None
) -> dict[str, Type[BlacklineModel]]:
    """
    Parse a directory of YAML files into a dictionary of Dataset objects.

    Args:
        path: The path to the directory of YAML files.
        path: Path

    Returns:
        A dictionary of Dataset objects.
    """
    children: dict[str, Type[BlacklineModel]] = {}
    if cls.children_cls is None:
        return children
    for child_path in path.iterdir():
        if child_path.is_dir():
            child = cls.children_cls.parse_dir(
                path=child_path, key_prefix=key_prefix
            )
            children[child.key] = child
    return children

parse_dir(path, key_prefix=None) classmethod

Parse a directory of YAML files into a dictionary of Dataset objects.

Parameters:

Name	Type	Description	Default
path	Path	The path to the directory of YAML files.	required
path	Path	Path	required

Returns:

Type	Description
	A dictionary of Dataset objects.

Source code in /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

@classmethod
def parse_dir(cls, path: Path, key_prefix: Optional[str] = None):
    """
    Parse a directory of YAML files into a dictionary of Dataset objects.

    Args:
        path: The path to the directory of YAML files.
        path: Path

    Returns:
        A dictionary of Dataset objects.
    """
    key = ".".join([key_prefix, path.name]) if key_prefix is not None else path.name
    children = cls.parse_children(path=path, key_prefix=key)
    filepath = cls.find_definition_file(path=path)
    return cls.parse_yaml(path=filepath, key=key, children=children)

parse_yaml(path, key, children={}) classmethod

Parse a yaml file into a the children_cls object.

Parameters:

Name	Type	Description	Default
path	Path	Path location of the yaml file.	required
key	str	Key to identify the dataset.	required

Returns:

Type	Description
	Dataset object.

Source code in /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

@classmethod
def parse_yaml(
    cls,
    path: Path,
    key: str,
    children: Optional[dict[str, Type[BlacklineModel]]] = {},
):
    """
    Parse a yaml file into a the children_cls object.

    Args:
        path: Path location of the yaml file.
        key: Key to identify the dataset.

    Returns:
        Dataset object.
    """
    with open(path, "r") as f:
        info = yaml.safe_load(f)[cls.stem][0]
        info["key"] = key
        if cls.stem == "dataset":
            return cls.parse_obj(info)
        info[cls.children_stem] = children
        return cls.parse_obj(info)

blackline.models.catalogue.ResourceTypeEnum

Bases: str, Enum

The model for resource types.

Source code in BAR /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

class ResourceTypeEnum(str, Enum):
    """The model for resource types."""

    APPLICATION = "Application"
    SERVICE = "Service"
    DATABASE = "Database"
    DATALAKE = "Datalake"
    DATA_WAREHOUSE = "Data Warehouse"

APPLICATION = 'Application' class-attribute instance-attribute

DATABASE = 'Database' class-attribute instance-attribute

DATALAKE = 'Datalake' class-attribute instance-attribute

DATA_WAREHOUSE = 'Data Warehouse' class-attribute instance-attribute

SERVICE = 'Service' class-attribute instance-attribute

blackline.models.catalogue.DataResponsibilityTitle

Bases: str, Enum

The model defining the responsibility or role over the system that processes personal data.

Used to identify whether the organization is a Controller, Processor, or Sub-Processor of the data

Source code in BAR /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

class DataResponsibilityTitle(str, Enum):
    """
    The model defining the responsibility or role over
    the system that processes personal data.

    Used to identify whether the organization is a
    Controller, Processor, or Sub-Processor of the data
    """

    CONTROLLER = "Controller"
    PROCESSOR = "Processor"
    SUB_PROCESSOR = "Sub-Processor"

blackline.models.catalogue.PrivacyDeclaration

Bases: BaseModel

The PrivacyDeclaration resource model.

States a function of a system, and describes how it relates to the privacy data types.

Parameters:

Name	Type	Description	Default
name	str	The name of the privacy declaration on the system.	required
data_categories	list[Key]	An array of data categories describing a system in a privacy declaration.	required
data_use	Key	The Data Use describing a system in a privacy declaration.	required
data_qualifier	Optional[Key]		None
data_subjects	list[Key]	An array of data subjects describing a system in a privacy declaration.	required
dataset_references	Optional[list[Key]]	Referenced Dataset keys used by the system.	None

Source code in BAR /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

class PrivacyDeclaration(BaseModel):
    """
    The PrivacyDeclaration resource model.

    States a function of a system, and describes how it relates
    to the privacy data types.
    """

    name: str = Field(
        description="The name of the privacy declaration on the system.",
    )
    data_categories: list[Key] = Field(
        description="An array of data categories describing a system in a privacy declaration.",
    )
    data_use: Key = Field(
        description="The Data Use describing a system in a privacy declaration.",
    )
    data_qualifier: Optional[Key] = Field(
        default=Key(
            "aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified"
        ),
        description="The key of the data qualifier describing a system in a privacy declaration.",
    )
    data_subjects: list[Key] = Field(
        description="An array of data subjects describing a system in a privacy declaration.",
    )
    dataset_references: Optional[list[Key]] = Field(
        description="Referenced Dataset keys used by the system.",
    )

blackline.models.validation.Key

Bases: ConstrainedStr

A Key type that creates a custom constrained string.

Parameters:

Name	Type	Description	Default
regex	Pattern[str]		required

Source code in BAR /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/validation.py

class Key(ConstrainedStr):
    """
    A Key type that creates a custom constrained string.
    """

    regex: Pattern[str] = re.compile(r"^[a-zA-Z0-9_.-]+$")

    # This overrides the default method to throw the custom ValidationError
    @classmethod
    def validate(cls, value: str) -> str:
        if not cls.regex.match(value):
            raise ValueError(
                "Key must only contain alphanumeric characters, '.', '_' or '-'."
            )

        return value

blackline.models.catalogue.ContactDetails

Bases: BaseModel

The contact details information model.

Used to capture contact information for controllers, used as part of exporting a data map / ROPA.

This model is nested under an Organization and potentially under a system/dataset.

Parameters:

Name	Type	Description	Default
name	Optional[str]	An individual name used as part of publishing contact information.	None
address	Optional[str]	An individual address used as part of publishing contact information.	None
email	Optional[str]	An individual email used as part of publishing contact information.	None
phone	Optional[str]	An individual phone number used as part of publishing contact information.	None

Source code in BAR /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

class ContactDetails(BaseModel):
    """
    The contact details information model.

    Used to capture contact information for controllers, used
    as part of exporting a data map / ROPA.

    This model is nested under an Organization and
    potentially under a system/dataset.
    """

    name: Optional[str] = Field(
        description="An individual name used as part of publishing contact information.",
    )
    address: Optional[str] = Field(
        description="An individual address used as part of publishing contact information.",
    )
    email: Optional[str] = Field(
        description="An individual email used as part of publishing contact information.",
    )
    phone: Optional[str] = Field(
        description="An individual phone number used as part of publishing contact information.",
    )

blackline.models.catalogue.DataProtectionImpactAssessment

Bases: BaseModel

The DataProtectionImpactAssessment (DPIA) resource model.

Contains information in regard to the data protection impact assessment exported on a data map or Record of Processing Activities (RoPA).

A legal requirement under GDPR for any project that introduces a high risk to personal information.

Parameters:

Name	Type	Description	Default
is_required	bool	A boolean value determining if a data protection impact assessment is required. Defaults to False.	False
status	Optional[ImpactAssessmentStatusEnum]	The optional status of a Data Protection Impact Assessment. Returned on an exported data map or RoPA.	None
link	Optional[AnyUrl]	The optional link to the Data Protection Impact Assessment. Returned on an exported data map or RoPA.	None

Source code in BAR /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

class DataProtectionImpactAssessment(BaseModel):
    """
    The DataProtectionImpactAssessment (DPIA) resource model.

    Contains information in regard to the data protection
    impact assessment exported on a data map or Record of
    Processing Activities (RoPA).

    A legal requirement under GDPR for any project that
    introduces a high risk to personal information.
    """

    is_required: bool = Field(
        default=False,
        description="A boolean value determining if a data protection impact assessment is required. Defaults to False.",
    )
    status: Optional[ImpactAssessmentStatusEnum] = Field(
        default=None,
        description="The optional status of a Data Protection Impact Assessment. Returned on an exported data map or RoPA.",
    )
    link: Optional[AnyUrl] = Field(
        default=None,
        description="The optional link to the Data Protection Impact Assessment. Returned on an exported data map or RoPA.",
    )

blackline.models.catalogue.ImpactAssessmentStatusEnum

Bases: str, Enum

The model for impact assessment statuses.

Source code in BAR /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/blackline/models/catalogue.py

class ImpactAssessmentStatusEnum(str, Enum):
    """The model for impact assessment statuses."""

    NOT_STARTED = "Not Started"
    IN_PROGRESS = "In Progress"
    WAITING_FOR_APPROVAL = "Waiting for Approval"
    COMPLETE = "Complete"

COMPLETE = 'Complete' class-attribute instance-attribute

IN_PROGRESS = 'In Progress' class-attribute instance-attribute

NOT_STARTED = 'Not Started' class-attribute instance-attribute

WAITING_FOR_APPROVAL = 'Waiting for Approval' class-attribute instance-attribute