Hi,
I have happily used drmaa-python for many years with our SGE cluster. Just recently a new cluster was installed, and this time it is configured to use MUNGE security.
If I create and submit a simple job, everything works fine, but if I run the job submission as part of a thread pool I get an error about MUNGE security.
For example:
import drmaa
from multiprocessing.pool import ThreadPool
import tempfile
import os
import stat
pool = ThreadPool(2)
session = drmaa.Session()
session.initialize()
def pTask(n):
smt = "ls . > test.out"
script_file = tempfile.NamedTemporaryFile(mode="w", dir=os.getcwd(), delete=False)
script_file.write(smt)
script_file.close()
print "Job is in file %s" % script_file.name
os.chmod(script_file.name, stat.S_IRWXG | stat.S_IRWXU)
jt = session.createJobTemplate()
print "jt created"
jt.jobEnvironment = {'BASH_ENV': '~/.bashrc'}
print "environment set"
jt.remoteCommand = os.path.join(os.getcwd(),script_file.name)
print "remote command set"
jobid = session.runJob(jt)
print "Job submitted with id: %s, waiting ..." % jobid
retval = session.wait(jobid, drmaa.Session.TIMEOUT_WAIT_FOREVER)
pool.map(pTask, (1,))produces the following output
Job is in file /home/userid/tmpbRa0IO
jt created
environment set
error: getting configuration: MUNGE authentication failed: Invalid credential format
remote command set
Traceback (most recent call last):
File "test_threads.py", line 31, in <module>
pool.map(pTask, (1,))
File "/home/mb1ims/.conda/envs/sharc/lib/python2.7/multiprocessing/pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
File "/home/mb1ims/.conda/envs/sharc/lib/python2.7/multiprocessing/pool.py", line 567, in get
raise self._value
drmaa.errors.DeniedByDrmException: code 17: MUNGE authentication failed: Invalid credential formatso the first sign of trouble is when jt.remoteCommand is set, but the script continues and gives an unhandled python error when session.runJob is executed.
Hi,
I have happily used drmaa-python for many years with our SGE cluster. Just recently a new cluster was installed, and this time it is configured to use MUNGE security.
If I create and submit a simple job, everything works fine, but if I run the job submission as part of a thread pool I get an error about MUNGE security.
For example:
produces the following output
so the first sign of trouble is when
jt.remoteCommandis set, but the script continues and gives an unhandled python error whensession.runJobis executed.