bfgn.configuration package

Submodules

bfgn.configuration.check_config_options_not_obsolete module

bfgn.configuration.check_config_options_not_obsolete.check_config_options_not_obsolete()[source]

bfgn.configuration.configs module

class bfgn.configuration.configs.Config(raw_files=None, data_build=None, data_samples=None, model_training=None, architecture=None, model_reporting=None, callback_general=None, callback_tensorboard=None, callback_early_stopping=None, callback_reduced_learning_rate=None)[source]

Bases: object

Handles the reading and formatting of raw data files, the building and training of models and architectures, and the reporting of training and validation results.

raw_files = None

RawFiles config section.

Type

sections.RawFiles

data_build = None

DataBuild config section.

Type

sections.DataBuild

data_samples = None

DataSamples config section.

Type

sections.DataSamples

model_training = None

ModelTraining config section.

Type

sections.ModelTraining

architecture = None

Architecture config section.

Type

sections.Architecture

model_reporting = None

ModelReporting config section.

Type

sections.ModelReporting

callback_general = None

CallbacksGeneral config section.

Type

sections.CallbacksGeneral

callback_tensorboard = None

Tensorboard config section.

Type

sections.Tensorboard

callback_early_stopping = None

EarlyStopping config section.

Type

sections.EarlyStopping

callback_reduced_learning_rate = None

CallBackReducedLearningRate config section.

Type

sections.CallBackReducedLearningRate

get_config_as_dict()[source]

Get configuration options as a nested dictionary with delineated sections.

Return type

dict

Returns

Configuration options as a nested dictionary with delineated sections.

get_config_errors(include_sections=None, exclude_sections=None)[source]

Get configuration option errors by checking the validity of each config section.

Parameters

  • include_sections (Optional[List[str]]) – Config sections that should be included. All config sections are included if None and exclude_sections is not specified. Cannot specify both include_sections and exclude_sections.

  • exclude_sections (Optional[List[str]]) – Config sections that should be excluded. All config sections are included if None and exclude_sections is not specified. Cannot specify both include_sections and exclude_sections.

Return type

list

Returns

List of errors associated with the current configuration.

get_human_readable_config_errors(include_sections=None, exclude_sections=None)[source]

Generates a human-readable string of configuration option errors.

Parameters

  • include_sections (Optional[List[str]]) – Config sections that should be included. All config sections are included if None and exclude_sections is not specified. Cannot specify both include_sections and exclude_sections.

  • exclude_sections (Optional[List[str]]) – Config sections that should be excluded. All config sections are included if None and exclude_sections is not specified. Cannot specify both include_sections and exclude_sections.

Return type

str

Returns

Human-readable string of configuration option errors.

bfgn.configuration.configs.create_config_from_file(filepath)[source]

Creates a Config object from a YAML file.

Parameters

filepath (str) – Filepath to existing YAML file.

Return type

Config

Returns

Config object with parsed YAML file attributes.

bfgn.configuration.configs.create_config_template(architecture_name, filepath=None)[source]

Creates a template version of a Config for a given architecture, with required and optional parameters highlighted, and default values for other parameters. Config is returned but can optionally be written to YAML file.

Parameters

  • architecture_name (str) – Name of available architecture.

  • filepath (Optional[str]) – Filepath to which template YAML file is saved, if desired.

Return type

Config

Returns

Template version of a Config.

bfgn.configuration.configs.save_config_to_file(config, filepath, include_sections=None)[source]

Saves/serializes a Config object to a YAML file.

Parameters

  • config (Config) – Config object.

  • filepath (str) – Filepath to which YAML file is saved.

  • include_sections (Optional[List[str]]) – Config sections that should be included. All config sections are included if None.

Return type

None

Returns

None

bfgn.configuration.configs.get_config_differences(config_a, config_b)[source]
Return type

Dict[~KT, ~VT]

bfgn.configuration.sections module

class bfgn.configuration.sections.BaseConfigSection[source]

Bases: object

Base Configuration Section from which all Configuration Sections inherit. Handles shared functionality like getting, setting, and cleaning configuration options.

classmethod get_config_name_as_snake_case()[source]
Return type

str

get_config_options_as_dict()[source]
Return type

Dict[str, Dict[str, <built-in function any>]]

get_option_keys()[source]
Return type

List[str]

set_config_options(config_options, highlight_required)[source]
Return type

None

check_config_validity()[source]
Return type

List[str]

class bfgn.configuration.sections.RawFiles[source]

Bases: bfgn.configuration.sections.BaseConfigSection

Raw file configuration, information necessary to locate and parse the raw files.

feature_files = 'REQUIRED'

List of filepaths to raw feature rasters. Format is a list of lists, where the outer grouping is sites, and the inner grouping is different data files for that particular site. E.G.: [[site_1_file_1, site_1_file_2],[site_2_file_1, site_2_file_2]]. File order and type between sites is expected to match. Files must all be the same projection and resolution, but need not be aligned to the same extent - if they are not, the common inner-area will be utilized.

Type

list

response_files = 'REQUIRED'

List of filepaths to raw response rasters or vectors (shp or kml supported currently). Format is a list of lists, where the outer grouping is sites, and the inner grouping is different data files for that particular site. E.G.: [[site_1_file_1, site_1_file_2],[site_2_file_1, site_2_file_2]]. File order and type between sites is expected to match. Files must all be the same projection and resolution, but need not be aligned to the same extent - if they are not, the common inner-area will be utilized.

Type

list

boundary_files = 'OPTIONAL'

Optional list of filepaths to boundaries. Data is built or sampled within the boundaries.

Type

list

feature_data_type = 'REQUIRED'

Data type from each input feature band. R for Real, C for Categorical. All C bands will be one-hot encoded. Provided as a list of lists corresponding to each band from each file in the raw input files list.

Type

list

response_data_type = 'REQUIRED'

Data type from each input feature band. R for Real, C for Categorical. All C bands will be one-hot encoded. Provided as a list of lists corresponding to each band from each file in the raw input files list.

Type

list

feature_nodata_value = -9999

Value that denotes missing data in feature files.

Type

float

response_nodata_value = -9999

Value that denotes missing data in response files.

Type

float

boundary_bad_value = 'OPTIONAL'

Value that denotes out-of-bounds areas in boundary files. For example, raster files may have non-negative values to represent in-bounds areas and -9999 to represent out-of-bounds areas. This value must be 0 when working with shapefiles where vectors/polygons denote in-bounds areas.

Type

float

ignore_projections = False

Should projection differences between feature and response files be ignored? This option should only be true if the user is confident that projections are identical despite encodings.

Type

bool

response_vector_property_name = 'Class'

This is the property (or column) of the response vector to use for categorization or regression. This value is only used if the response file(s) is/are in vector format.

Type

str

class bfgn.configuration.sections.DataBuild[source]

Bases: bfgn.configuration.sections.BaseConfigSection

Data build configuration, information necessary to structure and format the built data files.

dir_out = '.'

Directory to which built data files are saved.

Type

str

log_level = 'INFO'

Experiment log level. One of ERROR, WARNING, INFO, or DEBUG.

Type

str

filename_prefix_out = ''

Optional prefix for built data filenames, useful for organizing or tracking built data files from different build strategies.

Type

str

network_category = 'FCN'

Either CNN for convolutional neural network or FCN for fully convolutional network.

Type

str

random_seed = 1

Random seed for reproducible data generation.

Type

int

max_samples = 'REQUIRED'

Maximum number of built data samples to draw from the raw data files. Sampling stops when the raw data files are fully crawled or the maximum samples are reached.

Type

int

max_built_data_gb = 10.0

The maximum size of any given memmap array created in GB.

Type

float

number_folds = 10

Number of training data folds.

Type

int

validation_fold = 0

Index of fold to use for validation.

Type

int

test_fold = 'OPTIONAL'

Index of fold to use for testing.

Type

int

window_radius = 'REQUIRED'

Window radius determines the full image size as 2x the window radius

Type

int

loss_window_radius = 'REQUIRED'

Loss window radius determines the internal image window to use for loss calculations during model training.

Type

int

feature_mean_centering = False

Should features be mean centered?

Type

bool

feature_nodata_maximum_fraction = 0.0

Only include built data samples with a lower proportion of missing feature data values.

Type

float

response_nodata_maximum_fraction = 0.0

Only include built data samples with a lower proportion of missing response data values.

Type

float

response_min_value = 'OPTIONAL'

Response values below this minimum are converted to missing data. Currently applied to all response values uniformly.

Type

float

response_max_value = 'OPTIONAL'

Response values above this maximum are converted to missing data. Currently applied to all response values uniformly.

Type

float

response_background_values = 'OPTIONAL'

Built data samples containing only these responses are discarded and not included in the final built data files.

Type

int

sparse_read = False

Flag indicating if data should be read in ‘sparse’ mode. If the response input data is sparse within a large raster, this will dramatically speed up the data build. This is facilitated through an initial pass through the response and boundary files. If file is large and densely populated, this in unecessary and slows down the read.

Type

bool

class bfgn.configuration.sections.DataSamples[source]

Bases: bfgn.configuration.sections.BaseConfigSection

Data sample configuration, information necessary to parse built data files and pass data to models during training.

apply_random_transformations = False

Should random transformations, including rotations and flips, be applied to sample images.

Type

bool

batch_size = 100

The sample batch size for images passed to the model.

Type

int

feature_scaler_names = 'REQUIRED'

Names of the scalers which are applied to each feature file.

Type

list

response_scaler_names = 'REQUIRED'

Names of the scalers which are applied to each response file.

Type

list

feature_nodata_encoding = -10.0

The encoding for missing data values passed to the model, given that neural networks are sensitive to nans.

Type

float

class bfgn.configuration.sections.ModelTraining[source]

Bases: bfgn.configuration.sections.BaseConfigSection

Model training configuration, information necessary to train models from start to finish.

dir_out = '.'

Directory to which new model files are saved and from which existing model files are loaded.

Type

str

log_level = 'INFO'

Experiment log level. One of ERROR, WARNING, INFO, or DEBUG.

Type

str

verbosity = 1

Verbosity value for keras library. Either 0 for silent or 1 for verbose.

Type

int

assert_gpu = False

Assert, i.e., fail if GPUs are required and not available.

Type

bool

architecture_name = 'REQUIRED'

Architecture name from existing options.

Type

str

loss_metric = 'REQUIRED'

Loss metric to use for model training.

Type

str

max_epochs = 100

Maximum number of epochs to run model training.

Type

int

optimizer = 'adam'

Optimizer to use during model training. See Keras documentation for more information.

Type

str

weighted = False

Should underrepresented classes be overweighted during model training

Type

bool

class bfgn.configuration.sections.ModelReporting[source]

Bases: bfgn.configuration.sections.BaseConfigSection

Model reporting configuration, information necessary to generate reports for evaluating model performance.

max_pages_per_figure = 1

The max number of pages per figure in the model report.

Type

int

max_samples_per_page = 20

The max number of samples per page in supported figures in the model report

Type

int

max_features_per_page = 10

The max number of features per page in supported figures in the model report

Type

int

max_responses_per_page = 10

The max number of responses per page in supported figures in the model report

Type

int

network_progression_max_pages = 1

The max number of pages for the network progression figure in the model report. Note that the network progression figure is particularly expensive, both for computation and memory.

Type

int

network_progression_max_filters = 10

The max number of filters for the network progression figure in the model report. Note that the network progression figure is particularly expensive, both for computation and memory.

Type

int

network_progression_show_full = True

Whether to plot the full network progression plot, with fully-visible filters.

Type

bool

network_progression_show_compact = True

Whether to plot the compact network progression plot, with partially-visible filters.

Type

bool

class bfgn.configuration.sections.CallbackGeneral[source]

Bases: bfgn.configuration.sections.BaseConfigSection

checkpoint_periods = 5

Number of periods of model training between model state and history saves.

Type

int

use_terminate_on_nan = True

Terminate model training if nans are observed.

Type

bool

class bfgn.configuration.sections.CallbackTensorboard[source]

Bases: bfgn.configuration.sections.BaseConfigSection

use_callback = True

Use the Tensorboard callback to enable Tensorboard by saving necessary data.

Type

bool

update_freq = 'epoch'

See Keras documentation.

Type

str

histogram_freq = 0

See Keras documentation.

Type

int

write_graph = True

See Keras documentation.

Type

bool

write_grads = False

See Keras documentation.

Type

bool

write_images = True

See Keras documentation.

Type

bool

class bfgn.configuration.sections.CallbackEarlyStopping[source]

Bases: bfgn.configuration.sections.BaseConfigSection

use_callback = True

See Keras documentation.

Type

bool

loss_metric = 'val_loss'

Loss metric to monitor for early stopping. See Keras documentation.

Type

str

min_delta = 0.0001

See Keras documentation.

Type

float

patience = 10

See Keras documentation.

Type

int

class bfgn.configuration.sections.CallbackReducedLearningRate[source]

Bases: bfgn.configuration.sections.BaseConfigSection

use_callback = True

See Keras documentation.

Type

bool

factor = 0.5

See Keras documentation.

Type

float

loss_metric = 'val_loss'

Loss metric to monitor for reducing learning rate. See Keras documentation.

Type

str

min_delta = 0.0001

See Keras documentation.

Type

float

patience = 5

See Keras documentation.

Type

int

bfgn.configuration.sections.get_config_sections()[source]
Return type

List[Type[BaseConfigSection]]

bfgn.configuration.sections.check_input_file_validity(f_file_list, r_file_list, b_file_list)[source]
Return type

List[str]

Module contents