Configuration

This project uses YAML configuration files in the config/ directory to manage settings across different environments. This approach keeps configuration separate from code, making it easy to adjust behavior without modifying Python source files.

File Layout

config/
├── config.yml              # Main project settings (committed to version control)
├── secrets_template.yml    # Template for credentials (committed)
└── secrets.yml             # Your actual credentials (git-ignored, create from template)

Main Configuration — config.yml

The configuration file has two parts:

1. Shared settings — always loaded regardless of environment (e.g. project).
2. Environment-specific sections — under the environments key. Add, rename, or remove environments by editing this block — no Python changes required.

project:
  name: "arcgis-enterprise-clone-content"
  title: "ArcGIS Enterprise Clone Content"
  description: "Clone content between ArcGIS portals."

environments:

  default:
    logging:
      level: DEBUG
    data:
      input: "data/raw/input_data.csv"
      output: "data/processed/processed.gdb/output_data"
    migration:
      source_env: "source"          # key in secrets.yml for the source portal
      destination_env: "destination" # key in secrets.yml for the destination portal

  source:
    logging:
      level: DEBUG

  destination:
    logging:
      level: INFO

When the configuration is loaded, the active environment's section is deep-merged onto the shared settings, so you only need to specify what differs per environment. The default block provides base values that all named environments inherit.

Custom portal environment names

The available environments are introspected from the keys under environments in config.yml. If your organisation names its ArcGIS portals differently (e.g. prod_portal and dr_portal), add matching blocks here and update migration.source_env and migration.destination_env accordingly.

Secrets — secrets.yml

Credentials and API keys live in a separate file that is never committed to version control.

1. Copy secrets_template.yml to secrets.yml.
2. Fill in your actual values.

source:
  profile: ""        # ArcGIS named profile (takes precedence over url/username/password if set)
  url: "https://source-arcgis-enterprise.example.com/arcgis"
  username: "your_source_username"
  password: "your_source_password"

destination:
  profile: ""
  url: "https://destination-arcgis-enterprise.example.com/arcgis"
  username: "your_destination_username"
  password: "your_destination_password"

Profile vs. URL authentication

If profile is set to a non-empty string, url, username, and password are ignored. Set up a named profile via ArcGIS Pro or the ArcGIS API for Python.

Warning

secrets.yml is listed in .gitignore. Never commit real credentials.

Switching Environments

The active environment defaults to source. There are two ways to switch:

Option 1 — Environment Variable

Set the PROJECT_ENV variable before running any script or notebook:

=== "PowerShell"

```powershell
$env:PROJECT_ENV = "destination"
python -m scripts.make_data
```

=== "Bash / macOS"

```bash
export PROJECT_ENV=destination
python -m scripts.make_data
```

Option 2 — Edit the Constant

Open src/arcgis_cloning/config.py and change the default value:

ENVIRONMENT: str = os.environ.get("PROJECT_ENV", "source")  # change to "destination" or a custom portal name

Using Configuration in Python

In Scripts

from arcgis_cloning.config import config, secrets, ENVIRONMENT

# dot-notation access
log_level = config.logging.level
input_path = config.data.input

# dict-style access
output_path = config["data"]["output"]

# secrets
gis_url = secrets.esri.gis_url

# check current environment
print(f"Running in {ENVIRONMENT} mode")

Legacy-style imports also work for backward compatibility:

from arcgis_cloning.config import LOG_LEVEL, INPUT_DATA, OUTPUT_DATA

In Notebooks

import yaml
from pathlib import Path

config_path = Path.cwd().parent / "config" / "config.yml"
with open(config_path, encoding="utf-8") as f:
    cfg = yaml.safe_load(f)

# access the source environment settings
env = cfg["environments"]["source"]
print(env["logging"]["level"])

Or import the config module directly:

import sys, pathlib
sys.path.insert(0, str(pathlib.Path.cwd().parent / "src"))

from arcgis_cloning.config import config
print(config.logging.level)

API Reference

arcgis_cloning.config

Configuration loader for the project.

Reads settings from YAML configuration files in the config/ directory using a singleton pattern so the files are parsed once and reused across modules.

The YAML config supports environment-specific sections defined under the environments key in config.yml. A special default sub-section provides fallback values for any key that is not overridden in a named environment. The merge order is:

1. Top-level keys in config.yml
2. environments.default (if present) — overrides top-level defaults
3. environments.<active-env> — overrides both of the above

Add, rename, or remove environments by editing that YAML block — no Python changes required. Change the :pydata:ENVIRONMENT constant below — or set the PROJECT_ENV environment variable — to select the active environment.

Usage::

from arcgis_cloning.config import config, secrets, ENVIRONMENT, load_config, load_secrets

# dot-notation access (singleton loaded for the default 'source' portal)
log_level = config.logging.level

# dict-style access
input_path = config["data"]["input"]

# Per-portal clone pattern — load source and destination explicitly:
source_cfg = load_config(environment="source")
source_sec = load_secrets()
dest_cfg   = load_config(environment="destination")
dest_sec   = load_secrets()

# Access portal credentials — profile takes precedence over url/username/password
# Profile (ArcGIS named credential store):
if source_sec.source.profile:
    source_gis = GIS(profile=source_sec.source.profile)
else:
    source_gis = GIS(source_sec.source.url, source_sec.source.username, source_sec.source.password)

if dest_sec.destination.profile:
    dest_gis = GIS(profile=dest_sec.destination.profile)
else:
    dest_gis = GIS(dest_sec.destination.url, dest_sec.destination.username, dest_sec.destination.password)

# check current environment
print(f"Running in {ENVIRONMENT} mode")

ENVIRONMENT = os.environ.get('PROJECT_ENV', 'source') module-attribute

ConfigNode

Immutable, attribute-accessible wrapper around nested dictionaries.

Supports both dot-notation (cfg.logging.level) and dict-style (cfg["logging"]["level"]) access for convenience.

Source code in src/arcgis_cloning/config.py

class ConfigNode:
    """Immutable, attribute-accessible wrapper around nested dictionaries.

    Supports both dot-notation (``cfg.logging.level``) and dict-style
    (``cfg["logging"]["level"]``) access for convenience.
    """

    def __init__(self, data: dict[str, Any] | None = None) -> None:
        data = data or {}
        for key, value in data.items():
            if isinstance(value, dict):
                value = ConfigNode(value)
            # store on the instance __dict__ so attribute access works
            object.__setattr__(self, key, value)

    # dict-style access -------------------------------------------------------
    def __getitem__(self, key: str) -> Any:
        try:
            return getattr(self, key)
        except AttributeError:
            raise KeyError(key)

    def __contains__(self, key: str) -> bool:
        return key in self.__dict__

    def __iter__(self) -> Iterator[str]:
        return iter(self.__dict__)

    # convenience --------------------------------------------------------------
    def get(self, key: str, default: Any = None) -> Any:
        """Return the value for *key* if present, else *default*."""
        return self.__dict__.get(key, default)

    def to_dict(self) -> dict[str, Any]:
        """Recursively convert back to a plain dictionary."""
        out: dict[str, Any] = {}
        for key, value in self.__dict__.items():
            out[key] = value.to_dict() if isinstance(value, ConfigNode) else value
        return out

    def __repr__(self) -> str:
        return f"{self.__class__.__name__}({self.to_dict()!r})"

get(key, default=None)

Return the value for key if present, else default.

Source code in src/arcgis_cloning/config.py

def get(self, key: str, default: Any = None) -> Any:
    """Return the value for *key* if present, else *default*."""
    return self.__dict__.get(key, default)

to_dict()

Recursively convert back to a plain dictionary.

Source code in src/arcgis_cloning/config.py

def to_dict(self) -> dict[str, Any]:
    """Recursively convert back to a plain dictionary."""
    out: dict[str, Any] = {}
    for key, value in self.__dict__.items():
        out[key] = value.to_dict() if isinstance(value, ConfigNode) else value
    return out

get_available_environments(config_path=None)

Return the named environment keys defined in config.yml.

The reserved default key is excluded from the result because it is a fallback section, not a selectable environment.

Parameters

config_path : Path or str, optional Explicit path to a YAML file. Defaults to config/config.yml.

Returns

list[str] Sorted list of environment keys found under the environments section, excluding default (e.g. ["dev", "prod", "test"]).

Source code in src/arcgis_cloning/config.py

def get_available_environments(
    config_path: Path | str | None = None,
) -> list[str]:
    """Return the named environment keys defined in ``config.yml``.

    The reserved ``default`` key is excluded from the result because it is a
    fallback section, not a selectable environment.

    Parameters
    ----------
    config_path : Path or str, optional
        Explicit path to a YAML file.  Defaults to ``config/config.yml``.

    Returns
    -------
    list[str]
        Sorted list of environment keys found under the ``environments``
        section, excluding ``default`` (e.g. ``["dev", "prod", "test"]``).
    """
    path = Path(config_path) if config_path else CONFIG_DIR / _CONFIG_FILE
    raw = _load_yaml(path)
    return sorted(
        k for k in raw.get("environments", {}).keys() if k != "default"
    )

load_config(config_path=None, environment=None)

Load the main project configuration for a given environment.

Configuration is built up in three layers:

1. Top-level keys (e.g. project) — always included.
2. environments.default — deep-merged on top of the top-level keys, providing shared fallback values for all environments.
3. environments.<env> — deep-merged last, overriding both of the above with environment-specific values.

Any key present in default but absent from the active environment section is inherited from default. Keys present in the active environment always win.

Available environments are introspected from the environments key in config.yml (excluding the reserved default key) — add or remove named sections there to define your own.

Parameters

config_path : Path or str, optional Explicit path to a YAML file. Defaults to config/config.yml relative to the project root. environment : str, optional One of the keys under environments in config.yml. Defaults to the module-level :pydata:ENVIRONMENT constant.

Returns

ConfigNode A recursively accessible configuration object.

Raises

ValueError If the requested environment is not defined in config.yml.

Source code in src/arcgis_cloning/config.py

def load_config(
    config_path: Path | str | None = None,
    environment: str | None = None,
) -> ConfigNode:
    """Load the main project configuration for a given environment.

    Configuration is built up in three layers:

    1. **Top-level keys** (e.g. ``project``) — always included.
    2. **``environments.default``** — deep-merged on top of the top-level
       keys, providing shared fallback values for all environments.
    3. **``environments.<env>``** — deep-merged last, overriding both of the
       above with environment-specific values.

    Any key present in ``default`` but absent from the active environment
    section is inherited from ``default``.  Keys present in the active
    environment always win.

    Available environments are introspected from the ``environments`` key in
    ``config.yml`` (excluding the reserved ``default`` key) — add or remove
    named sections there to define your own.

    Parameters
    ----------
    config_path : Path or str, optional
        Explicit path to a YAML file.  Defaults to ``config/config.yml``
        relative to the project root.
    environment : str, optional
        One of the keys under ``environments`` in ``config.yml``.
        Defaults to the module-level :pydata:`ENVIRONMENT` constant.

    Returns
    -------
    ConfigNode
        A recursively accessible configuration object.

    Raises
    ------
    ValueError
        If the requested environment is not defined in ``config.yml``.
    """
    env = environment or ENVIRONMENT

    path = Path(config_path) if config_path else CONFIG_DIR / _CONFIG_FILE
    raw = _load_yaml(path)

    # pull out the environments block and the active env section
    environments = raw.pop("environments", {})

    # extract the default section (if any) before validation
    default_settings = environments.pop("default", None) or {}

    if env not in environments:
        available = ", ".join(sorted(environments.keys())) or "(none)"
        raise ValueError(
            f"Invalid environment '{env}'. "
            f"Available environments in config.yml: {available}"
        )

    env_settings = environments[env] or {}

    # three-way merge: top-level → default → env-specific
    merged = _deep_merge(raw, default_settings)
    merged = _deep_merge(merged, env_settings)
    return ConfigNode(merged)

load_secrets(secrets_path=None)

Load project secrets.

Parameters

secrets_path : Path or str, optional Explicit path to a YAML file. Defaults to config/secrets.yml relative to the project root.

Returns

ConfigNode A recursively accessible secrets object.

Raises

FileNotFoundError If the secrets file does not exist. Copy config/secrets_template.yml to config/secrets.yml and fill in your values.

Source code in src/arcgis_cloning/config.py

def load_secrets(
    secrets_path: Path | str | None = None,
) -> ConfigNode:
    """Load project secrets.

    Parameters
    ----------
    secrets_path : Path or str, optional
        Explicit path to a YAML file.  Defaults to ``config/secrets.yml``
        relative to the project root.

    Returns
    -------
    ConfigNode
        A recursively accessible secrets object.

    Raises
    ------
    FileNotFoundError
        If the secrets file does not exist. Copy
        ``config/secrets_template.yml`` to ``config/secrets.yml`` and
        fill in your values.
    """
    path = Path(secrets_path) if secrets_path else CONFIG_DIR / _SECRETS_FILE
    return ConfigNode(_load_yaml(path))