Repository navigation
IndexError while running OPS with Gromacs #918
Description
I am running one-way shooting OPS simulations with Gromacs using the 1.3.0.dev0 version. After a random amount of cyles (could be after 280 Monte Carlo cyles but also after 70) an error occurs and the simulation is stopped --> IndexError: Out of bounds on buffer access (axis 0). The ntseps in md.mdp file is 100000000 with nstxout 500. The initial trajectory used for OPS contains around 1500 frames. To get more information about what is happening I turned on the logging.config. However, currently we have no idea why the error is occurring or what to do to solve it.
The error and the information about the last run goes as follows :
Finished trajectory, length: 2
MC in ForwardShootMover using samples [<Sample @ 0x2abc121cb210>]
Trial was accepted
Working on Monte Carlo cycle number 89
Running for 55 minutes 41 seconds - 37.97 seconds per step
Estimated time remaining: 20 hours 10.02 minutes
RootMover (RandomChoiceMover) selecting ShootingChooser (index 0)
ShootingChooser (RandomChoiceMover) selecting OneWayShootingMover [SequentialEnsemble] (index 0)
OneWayShootingMover [SequentialEnsemble] (OneWayShootingMover) selecting ForwardShoot (index 0)
Running forward from frame 2 in [0:16]
Starting trajectory
gmx grompp -c ./mdwater.gro -f ./md.mdp -p ./topol.top -t ./initial_frame.trr
:-) GROMACS - gmx grompp, 2018.6 (-:
GROMACS is written by:
Emile Apol Rossen Apostolov Paul Bauer Herman J.C. Berendsen
Par Bjelkmar Aldert van Buuren Rudi van Drunen Anton Feenstra
Gerrit Groenhof Aleksei Iupinov Christoph Junghans Anca Hamuraru
Vincent Hindriksen Dimitrios Karkoulis Peter Kasson Jiri Kraus
Carsten Kutzner Per Larsson Justin A. Lemkul Viveca Lindahl
Magnus Lundborg Pieter Meulenhoff Erik Marklund Teemu Murtola
Szilard Pall Sander Pronk Roland Schulz Alexey Shvetsov
Michael Shirts Alfons Sijbers Peter Tieleman Teemu Virolainen
Christian Wennberg Maarten Wolf
and the project leaders:
Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
Copyright (c) 1991-2000, University of Groningen, The Netherlands.
Copyright (c) 2001-2017, The GROMACS development team at
Uppsala University, Stockholm University and
the Royal Institute of Technology, Sweden.
check out http://www.gromacs.org for more information.
GROMACS is free software; you can redistribute it and/or modify it
under the terms of the GNU Lesser General Public License
as published by the Free Software Foundation; either version 2.1
of the License, or (at your option) any later version.
GROMACS: gmx grompp, version 2018.6
Executable: /sw/arch/RedHatEnterpriseServer7/EB_production/2019/software/GROMACS/2018.6-intel-2018b-PLUMED-2.5.1/bin/gmx
Data prefix: /sw/arch/RedHatEnterpriseServer7/EB_production/2019/software/GROMACS/2018.6-intel-2018b-PLUMED-2.5.1
Working dir: /scratch/nodespecific/tcn113/vreede.8304105
Command line:
gmx grompp -c ./mdwater.gro -f ./md.mdp -p ./topol.top -t ./initial_frame.trr
Replacing old mdp entry 'unconstrained-start' by 'continuation'
Setting the LD random seed to 696021222
Generated 3828 of the 3828 non-bonded parameter combinations
Generating 1-4 interactions: fudge = 0.5
Generated 3828 of the 3828 1-4 parameter combinations
Excluding 3 bonded neighbours molecule type 'DNA_chain_E'
Excluding 3 bonded neighbours molecule type 'DNA_chain_F'
Excluding 3 bonded neighbours molecule type 'Protein_chain_B'
Excluding 3 bonded neighbours molecule type 'Protein_chain_D'
Excluding 2 bonded neighbours molecule type 'SOL'
Excluding 1 bonded neighbours molecule type 'NA'
Excluding 1 bonded neighbours molecule type 'CL'
Removing all charge groups because cutoff-scheme=Verlet
Number of degrees of freedom in T-Coupling group Protein is 5083.82
Number of degrees of freedom in T-Coupling group Water_and_ions is 79644.23
Number of degrees of freedom in T-Coupling group DNA is 1758.94
Reading Coordinates, Velocities and Box size from old trajectory
Will read whole trajectory
trr version: GMX_trn_file (single precision)
Last frame 0 time 0.000
Using frame at t = 0 ps
Starting time for run is 0 ps
Estimate for the relative computational load of the PME mesh part: 0.20
NOTE 1 [file ./md.mdp]:
This run will generate roughly 235301 Mb of data
There was 1 note
GROMACS reminds you: "It seems likely that significant software contributions to existing scientific software projects are not likely to be rewarded through the traditional reputation economy of science. Together these factors provide a reason to expect the over-production of independent scientific software packages, and the underproduction of collaborative projects in which later academics build on the work of earlier ones." (Howison & Herbsleb)
turning H bonds into constraints...
turning H bonds into constraints...
turning H bonds into constraints...
turning H bonds into constraints...
turning H bonds into constraints...
turning H bonds into constraints...
turning H bonds into constraints...
Analysing residue names:
There are: 21 DNA residues
There are: 117 Protein residues
There are: 13142 Water residues
There are: 265 Ion residues
Analysing residues not classified as Protein/DNA/RNA/Water and splitting into groups...
Analysing Protein...
Analysing residues not classified as Protein/DNA/RNA/Water and splitting into groups...
Determining Verlet buffer for a tolerance of 0.005 kJ/mol/ps at 300 K
Calculated rlist for 1x1 atom pair-list as 1.134 nm, buffer size 0.034 nm
Set rlist, assuming 4x4 atom pair-list, to 1.100 nm, buffer size 0.000 nm
Note that mdrun will redetermine rlist based on the actual pair-list setup
Calculating fourier grid dimensions for X Y Z
Using a fourier grid of 72x72x72, spacing 0.117 0.117 0.117
gmx mdrun -s topol.tpr -o ./md_trr/0000089.trr -e ./md_edr/0000089.edr -g ./md_log/0000089.log
Started engine: psutil.Popen(pid=31058, name='gmx', started='00:18:37')
:-) GROMACS - gmx mdrun, 2018.6 (-:
GROMACS is written by:
Emile Apol Rossen Apostolov Paul Bauer Herman J.C. Berendsen
Par Bjelkmar Aldert van Buuren Rudi van Drunen Anton Feenstra
Gerrit Groenhof Aleksei Iupinov Christoph Junghans Anca Hamuraru
Vincent Hindriksen Dimitrios Karkoulis Peter Kasson Jiri Kraus
Carsten Kutzner Per Larsson Justin A. Lemkul Viveca Lindahl
Magnus Lundborg Pieter Meulenhoff Erik Marklund Teemu Murtola
Szilard Pall Sander Pronk Roland Schulz Alexey Shvetsov
Michael Shirts Alfons Sijbers Peter Tieleman Teemu Virolainen
Christian Wennberg Maarten Wolf
and the project leaders:
Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
Copyright (c) 1991-2000, University of Groningen, The Netherlands.
Copyright (c) 2001-2017, The GROMACS development team at
Uppsala University, Stockholm University and
the Royal Institute of Technology, Sweden.
check out http://www.gromacs.org for more information.
GROMACS is free software; you can redistribute it and/or modify it
under the terms of the GNU Lesser General Public License
as published by the Free Software Foundation; either version 2.1
of the License, or (at your option) any later version.
GROMACS: gmx mdrun, version 2018.6
Executable: /sw/arch/RedHatEnterpriseServer7/EB_production/2019/software/GROMACS/2018.6-intel-2018b-PLUMED-2.5.1/bin/gmx
Data prefix: /sw/arch/RedHatEnterpriseServer7/EB_production/2019/software/GROMACS/2018.6-intel-2018b-PLUMED-2.5.1
Working dir: /scratch/nodespecific/tcn113/vreede.8304105
Command line:
gmx mdrun -s topol.tpr -o ./md_trr/0000089.trr -e ./md_edr/0000089.edr -g ./md_log/0000089.log
Reading file topol.tpr, VERSION 2018.6 (single precision)
Changing nstlist from 10 to 80, rlist from 1.1 to 1.225
Will use 20 particle-particle and 4 PME only ranks
This is a guess, check the performance at the end of the log file
Using 24 MPI threads
Using 1 OpenMP thread per tMPI thread
WARNING: This run will generate roughly 234965 Mb of data
starting mdrun 'Protein in water'
100000000 steps, 200000.0 ps.
Through frame: 0
step 160 Turning on dynamic load balancing, because the performance loss due to load imbalance is 11.4 %.
Traceback (most recent call last):
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/engines/external_snapshots/snapshot.py", line 50, in load_details
self.file_position
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/engines/gromacs/engine.py", line 197, in read_frame_data
filename, frame_num, len(f))
File "mdtraj/formats/xtc/trr.pyx", line 816, in mdtraj.formats.trr.TRRTrajectoryFile.__len__
File "mdtraj/formats/xtc/trr.pyx", line 751, in mdtraj.formats.trr.TRRTrajectoryFile.offsets.__get__
File "mdtraj/formats/xtc/trr.pyx", line 794, in mdtraj.formats.trr.TRRTrajectoryFile._calc_len_and_offsets
IndexError: Out of bounds on buffer access (axis 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "ops.py", line 2, in <module>
run_ops()
File "/scratch/nodespecific/tcn113/vreede.8304105/phase2.py", line 221, in run_ops
sampler.run(runs)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/pathsimulators/path_sampling.py", line 271, in run
movepath = self._mover.move(self.sample_set, step=self.step)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/pathmover.py", line 2580, in move
self.mover.move(sample_set),
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/pathmover.py", line 1582, in move
subchange = mover.move(sample_set)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/pathmover.py", line 1582, in move
subchange = mover.move(sample_set)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/pathmover.py", line 1582, in move
subchange = mover.move(sample_set)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/pathmover.py", line 613, in move
change = self.move_core(samples)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/pathmover.py", line 640, in move_core
trials, call_details = self(*samples)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/pathmover.py", line 786, in __call__
shooting_index)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/pathmover.py", line 900, in _run
trajectory, shooting_index
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/pathmover.py", line 860, in _make_forward_trajectory
running=[run_f])
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/engines/dynamics_engine.py", line 428, in generate
for trajectory in it:
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/engines/dynamics_engine.py", line 615, in iter_generate
continue_conditions=running)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/engines/dynamics_engine.py", line 382, in stop_conditions
can_continue = condition(trajectory, trusted)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/ensemble.py", line 2335, in can_append
trusted)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/ensemble.py", line 1859, in can_append
return self._generic_can_append(trajectory, trusted, strict=False)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/ensemble.py", line 1752, in _generic_can_append
trajectory, subtraj_first, ens_num, last_checked
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/ensemble.py", line 1639, in _find_subtraj_final
while ((ens.can_append(subtraj, trusted=True) or
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/ensemble.py", line 2190, in can_append
return self._trusted_call(trajectory, self._cache_can_append)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/ensemble.py", line 2180, in _trusted_call
cache.contents['previous'] = self._volume(frame)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/volume.py", line 204, in __call__
return not self.volume(snapshot)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/volume.py", line 135, in __call__
a = self.volume1(snapshot)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/volume.py", line 135, in __call__
a = self.volume1(snapshot)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/volume.py", line 402, in __call__
l = self.collectivevariable(snapshot).__float__()
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/netcdfplus/chaindict.py", line 229, in __getitem__
return self._post[items]
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/netcdfplus/chaindict.py", line 242, in __getitem__
return np.array(self._post[items])
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/netcdfplus/chaindict.py", line 260, in __getitem__
return self._post[[items]][0]
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/netcdfplus/chaindict.py", line 74, in __getitem__
rep = list(self._post[nones])
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/netcdfplus/chaindict.py", line 67, in __getitem__
results = self._get_list(items)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/netcdfplus/chaindict.py", line 359, in _get_list
results = self._eval(items)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/collectivevariable.py", line 570, in _eval
t = trajectory_to_mdtraj(trajectory, self.topology.mdtraj)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/engines/openmm/tools.py", line 367, in trajectory_to_mdtraj
return trajectory.to_mdtraj(md_topology)
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/engines/trajectory.py", line 544, in to_mdtraj
output = self.xyz
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/engines/trajectory.py", line 147, in __getattr__
out = [getattr(snap, item) for snap in self]
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/engines/trajectory.py", line 147, in <listcomp>
out = [getattr(snap, item) for snap in self]
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/engines/external_snapshots/features/coordinates.py", line 12, in xyz
snapshot.load_details()
File "/home/vreede/anaconda3/lib/python3.7/site-packages/openpathsampling/engines/external_snapshots/snapshot.py", line 54, in load_details
logger.debug("Exception reading from %s[%d]", self.file_name,
NameError: name 'logger' is not defined
Error in sys.excepthook:
Original exception was:
srun: error: tcn113: task 0: Exited with exit code 1
srun: Terminating job step 8304105.0
Activity
dwhswenson commentedon Jun 9, 2020
Is this not the same as the #907? I think this was solved by #908. It may be that you're on an older version of the 1.3 development cycle, and all you need to do is update. If you have an editable install, that's as easy as switching to the appropriate directory and typing
git pull upstream master.If that doesn't fix it, since you're on a development version, I'll need the full version to tell what you're working with. Use
print(paths.version.full_version). This will give the hash of the git commit, which we can then look up in the repository. Example: right now I get1.3.0.dev0+gf0ced85, which means that the last commit was f0ced85.jasperdenduijf commentedon Jun 9, 2020
Hey David, perhaps you can help. It might be true that the current version on github solves the problem, but the most recent version downloaded by anaconda not yet. Since we're not directly capable to pull from your repository, how would you recommend us to proceed?
sroet commentedon Jun 9, 2020
@jasperdenduijf
Assuming you have
pipas part of your conda environment (andgitif you don't you have a systemgitavailable) you can always install with:pip install git+https://github.com/openpathsampling/openpathsampling.gitBut this might not be possible, depending on what you mean by:
jasperdenduijf commentedon Jun 9, 2020
Exacly what I needed. Thank you very much!