- [ ] Standard format in which individual tasks (e.g. semantic segmentation, fusion, etc...) can be executed.

- [ ] Make common code shared among teams for increased re-usability.

- [ ] All tasks can be executed through a common fit and predict modules with similar APIs.

- [ ] Have tools infer sensible defaults whenever possible.

- [ ] 

watch
├── datasets                                    # generatic re-usable dataset templates
│   ├── video.py            
│   ├── segmentation.py      
│   ├── detection.py         
│   ├── classification.py    
│   └── ...                                
├── tasks                                       # Individual tasks for each group
│   ├── <fusion_task> 
│   │   └── ...        
│   ├── <rutgers_task>    
│   │   ├── <rutgers_task>_dataset.py           # method specific datasets
│   │   ├── models                              # method specific models
│   │   │   ├── <rutgers_task>_model.py
│   │   ├── utils                               # method specific utilities
│   │   │   ├── <rutgers_task>_specific_util.py
│   │   ├── fit.py
│   │   ├── predict.py
│   │   └── ...        
│   ├── <u_maryland_task>      
│   │   └── ...        
│   ├── <u_conn_task>         
│   │   └── ...        
│   ├── <u_kentucky_task>
│   │   └── ...        
│   └── ...        
├── models                                      # generaic or common models. This can also contain all models.
│   ├── resnet.py            
│   ├── unet.py      
│   ├── segnet.py         
│   ├── deeplabv3.py   
│   ├── aspp.py
│   ├── swin.py
│   └── ... 
├── utils                                       # generic re-usable utilities
│   ├── util_raster.py      
│   ├── util_girder.py         
│   ├── util_visualization.py    
│   └── ...    
├── gis                                         # 
├── demo                                        # Demos
├── validation                                  # 
└── ...

The name of each task, can be chosen by the owners of that task. Code in each task folder should contain at minimum two scripts:

• fit.py for training a model from a specified dataset, and
• predict.py for predicting on a specified dataset given a specified model.

Other code in each task folder can be arbitrary, and task-developers should be able to expect that their task folder to be somewhat sandboxed, and other developers will not introduce conflicts.

This structure allows for training and evaluating tasks independently, or evaluate models jointly (meaning concatenated features) through the fusion module. When a method stores the best performing model, it saves a "deployed.zip" zipfile which contains dataset hyperparameters, model weights, and method specific configurations.

To train a model, we expect that the task-specific fit script will use a command line interface somewhat like this:

python -m geowatch.tasks.<task_name>.fit --train_dataset=<path-to-kwcoco> --vali_dataset=<path-to-kwcoco> <additional hyperparam config>

The <additional hyperparam config> could be additional command line parameters (e.g. --lr=3e-4, --batch_size=4) or a path to a config file (e.g. --config=<path-to-yml>), which might also contain the train and validation dataset paths (although we strongly recommend that passing paths to the training / validation / testing datasets be specifiable via the command line). Note the best-of-both-worlds can be obtained by using scriptconfig (https://pypi.org/project/scriptconfig/) for configuration management.

At a minimum, this fit task should produce a trained state-dict for a particular model. Ideally the task will use torch-liberator (https://pypi.org/project/torch-liberator/) to package that state-dict with the model code itself, and a json file containing relevant metadata into a standalone deploy zipfile.

NOTE: if your method does not require learning parameters of a model, it is fine to omit the "fit" script and just provide "predict".

To predict with a model, we expect that there will be a task-specific predict script. This should take model weights to predict with, and a kwcoco dataset to predict on.

python -m geowatch.tasks.<task_name>.predict --deployed=<path-to-deploy-zipfile> --dataset=<path-to-kwcoco> <additional prediction config>

The output of this script should be a modified version of the input kwcoco file with additional annotations / auxiliary channels predictions. Again <additional prediction config> can be a config file, or additional command line arguments (again we suggest using scriptconfig).

Evaluation will be handled by using the predict API in conjunction with an external evaluation tool.

Example invocations of fit and predict scripts may look like this:

    python -m geowatch.tasks.rutgers.fit --train_dataset=drop0-train.kwcoco.json --config=train_config_v1.yml

    python -m geowatch.tasks.rutgers.predict --deployed=model_v1.zip --dataset=drop0-test.kwcoco.json

    python -m geowatch.tasks.invariants.fit --train_dataset=drop0-train.kwcoco.json --vali_dataset=drop0-train.kwcoco.json --model=custom_arch_v1 --init=<path/to/pretrained/state.pt> --lr=1e-3 --workers=8 --workdir=$HOME/work/smart --name=myexpt_v1
    python -m geowatch.tasks.invariants.predict --deployed=$HOME/work/smart/myexpt_v1/deployed.zip --dataset=drop0-test.kwcoco.json --output=drop0-test-predictions.kwcoco.json

    python -m geowatch.tasks.fusion.fit --config fusion_fit_config_v5.yml
    python -m geowatch.tasks.fusion.predict --config fusion_predict_config_v5.yml