Skip to content
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
version: 2
updates:
# Enable version updates for Github Actions
- package-ecosystem: "github-actions"
# Look for `/.github/workflows` and `/action.yml` or `.yaml`
directory: "/"
# Check for updates once a week
schedule:
interval: "weekly"

4 changes: 2 additions & 2 deletions .github/workflows/linting.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ jobs:

steps:
- name: Checkout IMAS-Python sources
uses: actions/checkout@v4
uses: actions/checkout@v6

- name: Set up Python
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
# until saxonche is available in 3.13
# https://saxonica.plan.io/issues/6561
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@ jobs:
name: Build distribution
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
# until saxonche is available in 3.13
# https://saxonica.plan.io/issues/6561
Expand All @@ -25,7 +25,7 @@ jobs:
- name: Build a binary wheel and a source tarball
run: python3 -m build .
- name: Store the distribution packages
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: python-package-distributions
path: dist/
Expand All @@ -43,7 +43,7 @@ jobs:
id-token: write # IMPORTANT: mandatory for trusted publishing
steps:
- name: Download all the dists
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
name: python-package-distributions
path: dist/
Expand All @@ -63,7 +63,7 @@ jobs:
id-token: write # IMPORTANT: mandatory for trusted publishing
steps:
- name: Download all the dists
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
name: python-package-distributions
path: dist/
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/test_with_pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ jobs:

steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v6

- name: Set up Python ${{ matrix.python-version }}

uses: actions/setup-python@v4
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
- name: Display Python version
Expand All @@ -37,13 +37,13 @@ jobs:
python -m pytest -n=auto --cov=imas --cov-report=term-missing --cov-report=xml:coverage.xml --cov-report=html:htmlcov --junit-xml=junit.xml

- name: Upload coverage report ${{ matrix.python-version }}
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: coverage-report-${{ matrix.python-version }}
path: htmlcov

- name: Upload test report ${{ matrix.python-version }}
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: test-report-${{ matrix.python-version }}
path: junit.xml
53 changes: 0 additions & 53 deletions .github/workflows/verify_sphinx_doc.yml

This file was deleted.

2 changes: 1 addition & 1 deletion .readthedocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ python:
sphinx:
builder: html
configuration: docs/source/conf.py
fail_on_warning: false
fail_on_warning: true
3 changes: 3 additions & 0 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@
author = "ITER Organization"
src_host = "https://github.com/iterorganization/"

# Warn about missing references
nitpicky = True

# Parse urls here for convenience, to be re-used
# ITER docs
iter_projects = "https://github.com/iterorganization/"
Expand Down
33 changes: 27 additions & 6 deletions docs/source/identifiers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,16 @@ enumerated list of options for defining, for example:
- These may have alternative naming conventions supported through aliases
(e.g., "235U" and "U_235" for Uranium 235).

Identifiers are a list of possible valid labels. Each label has up to four
representations:
Identifiers are a list of possible valid options. Each option has three representations
that are stored in an IDS:

1. An index (integer)
2. A name (short string)
3. A description (long string)
4. List of aliases (list of short strings)

.. seealso::
`Data Dictionary documentation for identifiers
<https://imas-data-dictionary.readthedocs.io/en/latest/identifiers.html>`__


Identifiers in IMAS-Python
Expand All @@ -32,6 +35,20 @@ constructed on-demand from the loaded Data Dictionary definitions.
All identifier enums can be accessed through ``imas.identifiers``. A list of
the available identifiers is stored as ``imas.identifiers.identifiers``.

Each identifier option provides the following attributes:

- ``name``: the name of the option.
- ``index``: the integer index value of the option.
- ``description``: a longer string describing the option.
- ``aliases``: a list of aliases that can be used instead of the name.
- ``units``: optional information about the units of the quantities that are affected by
the identifier option. Take, for example, the `poloidal plan coordinate identifier
<https://imas-data-dictionary.readthedocs.io/en/stable/generated/identifier/poloidal_plane_coordinates_identifier.html>`__
which affects the units of ``grid/dim1`` and ``grid/dim2``.

.. versionadded:: 2.1.0 ``aliases`` for identifiers.
.. versionadded:: 2.3.0 ``units`` metadata.

.. code-block:: python
:caption: Accessing identifiers

Expand All @@ -47,6 +64,9 @@ the available identifiers is stored as ``imas.identifiers.identifiers``.
print(csid.total.index)
print(csid.total.description)

# Search identifier options by their index value
print(csid(1))

# Access identifiers with aliases (when available)
mid = imas.identifiers.materials_identifier
print(mid["235U"].name) # Access by canonical name
Expand All @@ -57,12 +77,12 @@ the available identifiers is stored as ``imas.identifiers.identifiers``.
assert mid["235U"].name is mid.U_235.name

# Item access is also possible
print(identifiers["edge_source_identifier"])
print(imas.identifiers["edge_source_identifier"])

# You can use imas.util.inspect to list all options
imas.util.inspect(identifiers.ggd_identifier)
imas.util.inspect(imas.identifiers.ggd_identifier)
# And also to get more details of a specific option
imas.util.inspect(identifiers.ggd_identifier.SN)
imas.util.inspect(imas.identifiers.ggd_identifier.SN)

# When an IDS node is an identifier, you can use
# metadata.identifier_enum to get the identifier
Expand Down Expand Up @@ -186,6 +206,7 @@ material_identifier["235U"].
mat.names[0] = mid.U_235.name # enum value via alias
mat.names[0] = mid["U_235"].name # enum value via alias


Compare identifiers
-------------------

Expand Down
40 changes: 25 additions & 15 deletions docs/source/intro.rst
Original file line number Diff line number Diff line change
Expand Up @@ -86,17 +86,6 @@ get an error message if this is not possible:
Load and store an IDS to disk with IMAS-Core
''''''''''''''''''''''''''''''''''''''''''''

.. note::

- This functionality requires the IMAS-Core, until this library is openly available
on GitHub you may need to fetch it from `git.iter.org <https://git.iter.org/>`_
(requires to have an ITER account). Using IMAS-Core also enable slicing methods
:py:meth:`~imas.db_entry.DBEntry.get_slice`,
:py:meth:`~imas.db_entry.DBEntry.put_slice` and
:py:meth:`~imas.db_entry.DBEntry.get_sample` (with IMAS-Core>=5.4).
- If you can't have access to it, you can save IDS to disk with the built-in
netCDF backend :ref:`Load and store an IDS to disk with netCDF`

To store an IDS to disk, we need to indicate the following URI to the
IMAS-Core: ``imas:<backend>?path=<path_to_folder>`` or using the legacy query keys
``imas:<backend>?user=<user>;database=<database>;version=<version>;pulse=<pulse>;run=<run>``
Expand All @@ -115,11 +104,9 @@ In IMAS-Python you do this as follows:
>>> # now store the core_profiles IDS we just populated
>>> dbentry.put(core_profiles)

.. image:: imas_structure.png

To load an IDS from disk, you need to specify the same information as
when storing the IDS (see above). Once the data entry is opened, you
can use ``<IDS>.get()`` to load IDS data from disk:
can use ``dbentry.get()`` to load IDS data from disk:

.. code-block:: python

Expand All @@ -146,11 +133,34 @@ In IMAS-Python you do this as follows:

To load an IDS from disk, you need to specify the same file information as
when storing the IDS. Once the data entry is opened, you
can use ``<IDS>.get()`` to load IDS data from disk:
can use ``dbentry.get()`` to load IDS data from disk:

.. code-block:: python

>>> # Now load the core_profiles IDS back from disk
>>> dbentry2 = imas.DBEntry("mypulsefile.nc","r")
>>> core_profiles2 = dbentry2.get("core_profiles")
>>> print(core_profiles2.ids_properties.comment.value)


Data Entry API overview
'''''''''''''''''''''''

See the documentation of :py:class:`imas.DBEntry <imas.db_entry.DBEntry>` for more
details on reading and writing IDSs to disk. Useful functions include:

- :py:meth:`~imas.db_entry.DBEntry.put` and :py:meth:`~imas.db_entry.DBEntry.put_slice`
to write a full IDS or write append a time slice to existing data.
- :py:meth:`~imas.db_entry.DBEntry.get`, :py:meth:`~imas.db_entry.DBEntry.get_slice` and
:py:meth:`~imas.db_entry.DBEntry.get_sample` to read all time slices, a single time
slice, or a sample of time slices from disk. ``get_slice()`` and ``get_sample()`` can
also interpolate data to a requested point in time.

All three ``get()`` methods have a ``lazy`` mode, which will only load data from disk
when you need it. This can greatly speed up data access in some scenarios. See
:ref:`Lazy loading` for more details.
- :py:meth:`~imas.db_entry.DBEntry.list_all_occurrences` to query whether there are any
occurrences of a certain IDS stored on disk.
- :py:meth:`~imas.db_entry.DBEntry.list_filled_paths` to query which Data Dictionary
paths have data filled inside a specific IDS.

30 changes: 5 additions & 25 deletions docs/source/multi-dd.rst
Original file line number Diff line number Diff line change
Expand Up @@ -290,24 +290,17 @@ you may see the following error:
Dictionary version of the DBEntry (3.42.0). This is not supported when using the
UDA backend.

There are three possible workarounds. The first two require passing an additional option
There are two possible workarounds. The first one requires passing an additional option
in the IMAS UDA URI: please see the `imas-core documentation
<https://imas-core.readthedocs.io/en/stable/user_guide/backends_guide.html#query-keys-specific-for-the-uda-backend>`__
for more details on these URI options.
for more details on URI options.

1. Use UDA fetch to bypass the cache problem. You can do this by appending ``&fetch=1``
to the URI when you create the :py:class:`~imas.db_entry.DBEntry`.

Note that this will download the entire IDS files from the remote server, this may
not be desired if you only want to read a single time slice.
2. Disable the UDA cache. You can do this by appending ``&cache_mode=none`` to the URI
when you create the :py:class:`~imas.db_entry.DBEntry`.

Note that this may make the ``get()`` (a lot) slower, since a separate request needs
to be sent to the remote UDA server for every data variable. However, this may still
be the best performing option if you are only interested in a subset of all the data
in an IDS (and use :ref:`lazy loading`).
3. Explicitly provide the data dictionary version when you create the
2. Explicitly provide the data dictionary version when you create the
:py:class:`~imas.db_entry.DBEntry`, setting it to match the Data Dictionary version
of the data you want to load. To obtain the version of the data on the remote server
from the field `ids_properties.put_version.data_dictionary` via a _lazy_ ``get()``
Expand All @@ -316,7 +309,7 @@ for more details on these URI options.
Note that you may need to call ``imas.convert_ids`` to convert the IDS to your
desired Data Dictionary version.

All three possible workarounds are shown in the examples below:
The two possible workarounds are shown in the examples below:

.. md-tab-set::

Expand Down Expand Up @@ -346,20 +339,7 @@ All three possible workarounds are shown in the examples below:
with imas.DBEntry(URI, "r") as entry:
cp = entry.get("core_profiles")

.. md-tab-item:: 2. Disable the UDA cache

.. code-block:: python

import imas

URI = (
"imas://uda.iter.org:56565/uda?backend=hdf5"
"&path=/work/imas/shared/imasdb/ITER/3/121013/50&cache_mode=none"
)
with imas.DBEntry(URI, "r") as entry:
cp = entry.get("core_profiles")

.. md-tab-item:: 3. Explicitly provide the DD version
.. md-tab-item:: 2. Explicitly provide the DD version

.. code-block:: python

Expand Down
Loading
Loading