Merge pull request #259 from iiasa/transport/2024-W49

Transport improvements from 2024-W49

Author: khaeru

doc/api/data-sources.rst

@@ -63,16 +63,16 @@ The data:
 
 The approach to handling proprietary data is the same as in :mod:`.project.advance` and :mod:`.project.ssp`:
 
-- Copies of the data are stored in the (private) :mod:`message_data` repository using Git LFS.
+- Copies of the data are stored in the (private) `message-static-data` repository using Git LFS.
   This respository is accessible only to users who have a license for the data.
 - :mod:`message_ix_models` contains only a ‘fuzzed’ version of the data (same structure, random values) for testing purposes.
 - Non-IIASA users must obtain their own license to access and use the data; obtain the data themselves; and place it on the system where they use :mod:`message_ix_models`.
 
 The module :mod:`message_ix_models.tools.iea.web` attempts to detect and support both the providers/formats described below.
 The code supports using data from any of the above locations and formats, in multiple ways:
 
-- Use :func:`.tools.iea.web.load_data` to load data as :class:`pandas.DataFrame` and apply further pandas processing.
-- Use :class:`.IEA_EWEB` via :func:`.tools.exo_data.prepare_computer` to use the data in :mod:`genno` structured calculations.
+- Use :class:`.IEA_EWEB` via :func:`.exo_data.prepare_computer` to use the data in :mod:`genno` structured calculations.
+- Use :func:`.iea.web.load_data` to load data as :class:`pandas.DataFrame` and apply further processing using pandas.
 
 The **documentation** for the `2023 edition <https://iea.blob.core.windows.net/assets/0acb1453-1221-421b-9131-632ce71a4c1a/WORLDBAL_Documentation.pdf>`__ of the IEA source/format is publicly available.
 
@@ -82,8 +82,8 @@ Structure
 The data have the following conceptual dimensions, each enumerated by a different list of codes:
 
 - ``FLOW``, ``PRODUCT``: for both of these, the lists of codes appearing in the data are the same from 2021 and 2023 inclusive.
-- ``COUNTRY``: The data provided by IEA directly contain codes that are all caps, abbreviated country names, for instance "DOMINICANR".
-  The data provided by the OECD contain ISO 3166-1 alpha-3 codes, for instance "DOM".
+- ``COUNTRY``: The data provided by IEA directly contain codes that are all caps, abbreviated country names, for instance 'DOMINICANR'.
+  The data provided by the OECD contain ISO 3166-1 alpha-3 codes, for instance 'DOM'.
   In both cases, there are additional labels denoting country groupings; these are defined in the documentation linked above.
 
   Changes visible in these lists include:
@@ -98,8 +98,9 @@ The data have the following conceptual dimensions, each enumerated by a differen
     - New codes: GNQ, MDG, MKD, RWA, SWZ, UGA.
     - Removed: EQGUINEA, GREENLAND, MALI, MBURKINAFA, MCHAD, MMADAGASCA, MMAURITANI, MPALESTINE, MRWANDA, MUGANDA, NORTHMACED.
 
-- TIME: always a year.
-- MEASURE: unit of measurement, either "TJ" or "ktoe".
+  See the :py:`transform=...` source keyword argument and :meth:`.IEA_EWEB.transform` for different methods of handling this dimension.
+- ``TIME``: always a year.
+- ``UNIT_MEASURE`` (not labeled): unit of measurement, either 'TJ' or 'ktoe'.
 
 :mod:`message_ix_models` is packaged with SDMX structure data (stored in :file:`message_ix_models/data/sdmx/`) comprising code lists extracted from the raw data for the COUNTRY, FLOW, and PRODUCT dimensions.
 These can be used with other package utilities, for instance:

doc/api/util.rst

@@ -10,10 +10,12 @@ Submodules:
 
    click
    context
+   genno
    importlib
    _logging
    node
    pooch
+   pycountry
    scenarioinfo
    sdmx
 
@@ -158,6 +160,13 @@ Commonly used:
 
       If no base scenario can be loaded, :func:`.bare.create_res` is called to generate a base scenario.
 
+:mod:`.util.genno`
+==================
+
+.. currentmodule:: message_ix_models.util.genno
+
+.. automodule:: message_ix_models.util.genno
+   :members:
 
 :mod:`.util.importlib`
 ======================

doc/conf.py

@@ -74,7 +74,9 @@
 .. role:: strike
 .. role:: underline
 
+.. |c| replace:: :math:`c`
 .. |n| replace:: :math:`n`
+.. |t| replace:: :math:`t`
 .. |y| replace:: :math:`y`
 .. |y0| replace:: :math:`y_0`
 

doc/project/ssp.rst

@@ -54,3 +54,20 @@ Transport
    :members:
 
    Use :program:`mix-models ssp transport --help in.xlsx out.xlsx` to invoke :func:`.main`.
+   Data are read from PATH_IN, in :file:`.xlsx` or :file:`.csv` format.
+   If :file:`.xlsx`, the data are first temporarily converted to :file:`.csv`.
+   Data are written to PATH_OUT; if not given, this defaults to the same path and suffix as PATH_IN, with "_out" added to the stem.
+
+   For example:
+
+   .. code-block:: shell
+
+      mix-models ssp transport --method=B \
+        SSP_SSP2_v2.1_baseline.xlsx
+
+   …produces a file :file:`SSP_SSP2_v2.1_baseline_out.xlsx` in the same directory.
+
+   As of 2025-01-25:
+
+   - Method 'B' (that is, :func:`.prepare_method_B`; see its documentation) is the preferred method.
+   - The code is tested on :file:`.xlsx` files in the (internal) directories under `SharePoint > ECE > Documents > SharedSocioeconomicPathways2023 > Scenario_Vetting <https://iiasahub.sharepoint.com/sites/eceprog/Shared%20Documents/Forms/AllItems.aspx?csf=1&web=1&e=APKv0Z&CID=23fa0a51%2Dc303%2D4381%2D8c6d%2D143305cbc5a1&FolderCTID=0x012000AA9481BF7BE9264E85B14105F7F082FF&id=%2Fsites%2Feceprog%2FShared%20Documents%2FSharedSocioEconomicPathways2023%2FScenario%5FVetting&viewid=956acd8a%2De1e7%2D4ae9%2Dab1b%2D0506911bae11>`_, for example :file:`v2.1_Internal_version_Dec13_2024/Reporting_output/SSP_SSP2_v2.1_baseline.xlsx`.

doc/transport/index.rst

@@ -142,7 +142,7 @@ Run the entire workflow
    The options result in the following behaviour:
 
    - :program:`--platform=ixmp-dev`: store MESSAGEix-Transport scenarios on the :mod:`ixmp` platform named "ixmp-dev".
-   - :program:`--base=auto`: identify the base scenario URLs using :func:`.base_scenario_url` / the file :file:`base-scenario-url.json`, according to other config settings.
+   - :program:`--base=auto`: identify the base scenario URLs using :func:`.base_scenario_url` / the file :ref:`CL_TRANSPORT_SCENARIO`, according to other config settings.
    - :program:`--model-extra="ci nightly"`: append the string " ci nightly" to the model name of any created Scenario.
      This avoids accidentally producing new versions of ‘production’ (model name, scenario name) combinations.
    - :program:`--from=""`: start from the very first step in the workflow—load the identified base scenario—and perform all subsequent workflow steps, up to and including…

doc/transport/input.rst

@@ -20,8 +20,8 @@ This value is retrieved from the :attr:`.model.Config.regions` setting on a :cla
 
 .. _transport-config:
 
-Configuration
-=============
+Configuration and model structure
+=================================
 
 General (:file:`config.yaml`, required)
 ---------------------------------------
@@ -50,13 +50,45 @@ This file gives code lists for other MESSAGE concepts/sets/dimensions.
 
 → View :source:`message_ix_models/data/transport/set.yaml` on GitHub
 
+.. _CL_TRANSPORT_SCENARIO:
+
+Code list ``CL_TRANSPORT_SCENARIO``
+-----------------------------------
+
+This code list, stored in the file :file:`message_ix_models/data/sdmx/IIASA_ECE_CL_TRANSPORT_SCENARIO(1.0.0).xml`, contains an SDMX code list for distinct MESSAGEix-Transport scenarios.
+The codes have IDs like ``LED-SSP1`` that give a short identifier used in :mod:`.transport.workflow` and elsewhere, and names that give a complete, human-readable description.
+Every code has all of following annotations:
+
+``SSP-URN``
+   Complete URN of a code in ``ICONICS:SSP(2024)`` or another code list for the SSP used for sociodemographic input data and to control other settings in :mod:`.transport.build`.
+
+   Example annotation text: ``'urn:sdmx:org.sdmx.infomodel.codelist.Code=ICONICS:SSP(2024).1'``
+
+``is-LED-Scenario``
+   Example annotation text: ``True``
+
+   :func:`repr` of Python :any:`True` or :any:`False`, the former indicating that "Low Energy Demand (LED)" settings should be used.
+   See also :attr:`Config.project <.transport.config.Config.project>`.
+
+``EDITS-activity-id``
+   Example annotation text: ``'HA'``
+
+   For :doc:`/project/edits`, the identity of an ITF PASTA scenario providing exogenous transport activity.
+
+``base-scenario-URL``
+   Example annotation text: ``'ixmp://ixmp-dev/SSP_SSP1_v1.1/baseline_DEFAULT_step_13'``
+
+   URL of a base scenario used to build the corresponding MESSAGEix-Transport scenario.
+
+
 .. _transport-data-files:
 
 Input data files
 ================
 
 :data:`.transport.files.FILES` gives a list of all data files.
-Through :func:`.transport.build.main` (ultimately, :func:`.transport.build.add_exogenous_data`), each of these files is connected to a :class:`genno.Computer` used for model-building, and its contents appear at the key given in the list below.
+Through :func:`.transport.build.main` (ultimately, :func:`.transport.build.add_exogenous_data`), each of these files is connected to a :class:`genno.Computer` used for model-building.
+Its content are available at the corresponding key, which is used as an input for further model-building computations.
 
 .. admonition:: Example
 
@@ -67,9 +99,22 @@ Through :func:`.transport.build.main` (ultimately, :func:`.transport.build.add_e
 Not all files are currently or always used in model-building computations.
 Some submodules of :mod:`~.model.transport` use additional data files via other mechanisms.
 Most of the files have a header comment including a precise description of the quantity, source of the data, and units of measurement.
-In some cases extended information is below (where a header comment would be too long).
+In some cases—where a header comment would be too long—extended information is below.
 The :program:`git` history of files, or the GitHub "blame" view can also be used to inspect the edit history of each file, line by line.
 
+.. _transport-input-emi-intensity:
+
+:file:`emi-intensity.csv` → ``emissions intensity:t-c-e:transport``
+-------------------------------------------------------------------
+
+Measure
+   Emissions intensity of energy use
+Units
+   g (of emissions species) / MJ (of input energy)
+
+See the file :source:`on GitHub <message_ix_models/data/transport/emi-intensity.csv>` for inline comments and commit history.
+Currently only used in :mod:`.ssp.transport`.
+
 :file:`ldv-activity.csv` → ``ldv activity:n:exo``
 -------------------------------------------------
 

doc/whatsnew.rst

@@ -1,8 +1,20 @@
 What's new
 **********
 
-.. Next release
-.. ============
+Next release
+============
+
+- Update :class:`.IEA_EWEB` to support :py:`transform="B"` / :func:`.transform_B` (:issue:`230`, :pull:`259`).
+- Update :doc:`/transport/index` (:pull:`259`).
+
+  - Adjust constraints on :py:`t="conm_ar"`.
+  - Recompute :attr:`.minimum_activity` for transport technologies.
+  - Adjust freight activity, freight and passenger mode shares for some regions.
+  - Drop :file:`base-scenario-url.json`; store base scenario URLs in :ref:`CL_TRANSPORT_SCENARIO`.
+  - Simplify and consolidate tests.
+
+- Add :func:`.prepare_method_B` to :mod:`.ssp.transport` (:pull:`259`).
+- New utility :class:`.sdmx.AnnotationsMixIn` (:pull:`259`).
 
 v2025.1.10
 ==========

message_ix_models/data/emission.yaml

@@ -13,6 +13,7 @@ CO2:
 
 N2O:
   name: Nitrous oxide
+  units: kt
 
 NH3:
   name: Ammonia