Power plant data

The World Resources Institute has a great open dataset

Power plant data

      import pandas as pd
      import matplotlib.pyplot as plt
      sns.set_color_codes("muted")
      sns.set(font_scale=2)
      
      power_plants = pd.read_csv("talk_data/global_power_plant_database.csv")
      f, ax = plt.subplots(figsize=(12, 8))
      
      plants = power_plants.groupby('primary_fuel')\
                            .sum()\
                            .sort_values('estimated_generation_gwh', ascending=False)\
                            .reset_index()
        
      sns.barplot(x="estimated_generation_gwh", y="primary_fuel", data=plants,
                  label="Total", color="b")
      
      ax.set(ylabel="", xlabel="Estimated Generation 2018 in GWh")
      sns.despine(left=True, bottom=True)
    

Data: https://datasets.wri.org/dataset/globalpowerplantdatabase

Geospatial Vector

e.g. *.geojson, *.shp, ...

Source: AutoGIS, University of Helsink, 'These materials are partly based on Shapely-documentation and Westra E. (2013), Chapter 3.'

Getting Sentinel2 data

One option: Google Earth Engine

https://earthengine.google.com/

            conda update -c conda-forge earthengine-api
            
            # or 
            
            pip install earthengine-api --upgrade
          

            import ee

            def obtain_sentinel_imgs(collection='COPERNICUS/S2', 
                date_range, area, max_cc_pct=None):
            """
            Selection of median, cloud-free image from a 
            collection of images in the Sentinel 2 dataset

            Parameters
            ----------
            collection: str
            Name of the collection
            date_range: list
            Must be of the following form ['%Y-%m-%d','%Y-%m-%d']
            area: ee.geometry.Geometry
            Area of interest
            reduction: str
            Operation to be carried out over the aggregated image
            max_cc_pct: float
            Maximum cloud cover percentage to be returned

            Returns
            -------
            sentinel_median: ee.image.Image

            """

            col = (ee
            .ImageCollection(collection)
            .filterBounds(area)
            .filterDate(date_range[0], date_range[1])
            )

            if max_cc_pct is not None:
            col = col.filter(ee.Filter.lt(
                'CLOUDY_PIXEL_PERCENTAGE',
                max_cc_pct))

            return col
          

Working with imagery data

xarray

          array = xr.open_mfdataset(f'../data/imgs_eu/my_plant.nc', combine='by_coords')
        

My plan to
locate solar PV

1. Get image data set of the UK
2. Use Open Street Map labels for solar panels to create a training set
3. Use neural networks for semantic segmentation on all images to find the rest of the panels
4. Write up and relax

Semantic segmentation?

Image source: CSAIL

Ok, but what about the labels?

Enter Open Street Map!

At the time, 15000 labels

Now... 240,000+ !

          # # https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_API_by_Example

          solar_query = """
          [out:json];
          area["name"="United Kingdom"]->.searchArea;
          (
          nwr(area.searchArea)["generator:source"="solar"]['generator:output:electricity'='yes'];
          );
          out body;
          >;
          out skel qt;
          """
        

        overpass_url = "http://overpass-api.de/api/interpreter"
        
        import requests
        osm_pv = requests.get(overpass_url, params={'data': solar_query})
        df = pd.DataFrame(osm_pv.json()['elements'])
        

Setting up a pipeline?

Enter RasterVision

An open source framework for deep learning on satellite and aerial imagery.

Thank you Azavea 👏

How to use RasterVision

See my experiment script here: https://github.com/Rabscuttler/raster-vision/tree/master/code

Setup

• Docker image
• nvidia-docker requires Linux for GPUs
• Dependencies are a pain to manage otherwise
• GPUs mandatory - model runs can easily take 12hrs+ depending on experiment

Experiments

RasterVision takes on a lot of the work that would be reasonably boilerplate between geospatial analysis projects.

            import rastervision as rv

            ##########################################
            # Experiment
            ##########################################

            class SolarExperimentSet(rv.ExperimentSet):
            def exp_main(self, test=False):

              # experiment goes here
          

Experiment Steps

• SETUP
• TASK
• BACKEND
• DATASET (TRAINING & VALIDATION)
• ANALYZE
• EXPERIMENT

Setup

                  # Connect docker filepath mounted to my data directory
                  base_uri = join(raw_uri, '/labels')

                  # Experiment label, used to label config files
                  exp_id = 'pv-detection-1'

                  # Number of times passing a batch of images through the model
                  num_steps = 1e5
                  batch_size = 8
                  # Specify whether or not to make debug chips (a zipped sample of png chips
                  # that you can examine to help debug the chipping process)
                  debug = True

                  # Split the data into training and validation sets:
                  # Randomize the order of all scene ids
                  random.seed(5678)
                  scene_ids = sorted(scene_ids)
                  random.shuffle(scene_ids)

                  # Figure out how many scenes make up 80% of the whole set
                  num_train_ids = round(len(scene_ids) * 0.8)
                  # Split the scene ids into training and validation lists
                  train_ids = scene_ids[0:num_train_ids]
                  val_ids = scene_ids[num_train_ids:]
          

Task

              # ------------- TASK -------------

              task = rv.TaskConfig.builder(rv.SEMANTIC_SEGMENTATION) \
                .with_chip_size(300) \
                .with_classes({
                'pv': (1, 'yellow'),
                'background': (2, 'black')
              }) \
              .with_chip_options(
                chips_per_scene=50,
                debug_chip_probability=0.1,
                negative_survival_probability=1.0,
                target_classes=[1],
                target_count_threshold=1000) \
              .build()
        

Backend

            # # ------------- BACKEND -------------
            backend = rv.BackendConfig.builder(rv.TF_DEEPLAB) \
              .with_task(task) \
              .with_debug(debug) \
              .with_batch_size(num_steps) \
              .with_num_steps(batch_size) \
              .with_model_defaults(rv.MOBILENET_V2) \
              .build()

            # Make scenes to pass to the DataSetConfig builder.
            def make_scene(id):
              ###

            # Create lists of train and test scene configs
            train_scenes = [make_scene(id) for id in train_ids]
            val_scenes = [make_scene(id) for id in val_ids]
            
            

Dataset

            # ------------- DATASET -------------
            # Construct a DataSet config using the lists of train and
            # validation scenes
            dataset = rv.DatasetConfig.builder() \
              .with_train_scenes(train_scenes) \
              .with_validation_scenes(val_scenes) \
              .build()
          

Analyze

                # ------------- ANALYZE -------------
                # We will need to convert this imagery from uint16 to uint8
                # in order to use it. We specified that this conversion should take place
                # when we built the train raster source but that process will require
                # dataset-level statistics. To get these stats we need to create an
                # analyzer.
                analyzer = rv.AnalyzerConfig.builder(rv.STATS_ANALYZER) \
                  .build()
          

Experiment

          # ------------- EXPERIMENT -------------
          experiment = rv.ExperimentConfig.builder() \
            .with_id(exp_id) \
            .with_task(task) \
            .with_backend(backend) \
            .with_analyzer(analyzer) \
            .with_dataset(dataset) \
            .with_root_uri('/opt/data') \
            .build()

          return experiment


          if __name__ == '__name__':
            rv.main()
        

Run it!

Start up the docker container

          docker run --runtime=nvidia --rm -it -p 6006:6006 \
            -v ${RV_QUICKSTART_CODE_DIR}:/opt/src/code \
            -v ${RV_QUICKSTART_EXP_DIR}:/opt/data \
            quay.io/azavea/raster-vision:gpu-latest /bin/bash
        

Fire away

        rastervision run local -p find_the_solar_pv.py