Skip to content

correct role handling and S3FS bypass #130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ErinWeisbart merged 22 commits into from
Jan 21, 2023
Merged

correct role handling and S3FS bypass #130

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
0417e7d
correct role handling
ErinWeisbart Jun 16, 2022
526411d
update generate_task_definition
ErinWeisbart Aug 2, 2022
38df4c9
import configparser in run.py
ErinWeisbart Aug 2, 2022
80a8c5b
enable separate source and destination buckets
ErinWeisbart Aug 2, 2022
5498710
oops shouldnt have deleted those
ErinWeisbart Aug 2, 2022
e657f11
fix multibucket handling
ErinWeisbart Aug 3, 2022
cefd9e0
move source destination buckets to config
ErinWeisbart Aug 3, 2022
865d515
switched in and not in
ErinWeisbart Aug 3, 2022
f7919b8
fix s3fs bybass
ErinWeisbart Aug 4, 2022
b9f8916
remove multibucket from job file
ErinWeisbart Aug 4, 2022
3d8cc82
remove changes to run_batch_general
ErinWeisbart Aug 4, 2022
fad0284
csv handling to keep structure
ErinWeisbart Aug 4, 2022
c840b44
pipelines path and run command fixes
ErinWeisbart Aug 5, 2022
8f57047
output, destination, same difference
ErinWeisbart Aug 5, 2022
4ff6463
add upload flags
ErinWeisbart Aug 5, 2022
22abe35
but what if it doesn't exist
ErinWeisbart Aug 5, 2022
e774d41
more things in task definition
ErinWeisbart Aug 5, 2022
7d8f4c8
stash changes
ErinWeisbart Sep 15, 2022
e907c6a
update to AWS CLI2
ErinWeisbart Sep 21, 2022
c66de9e
the right way to check for variable, finally
ErinWeisbart Oct 5, 2022
8cb1390
Merge branch 'master' into role_fix
ErinWeisbart Jan 6, 2023
2b95a36
Update exampleJob.json
ErinWeisbart Jan 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix s3fs bybass
  • Loading branch information
@ErinWeisbart
ErinWeisbart committed Aug 4, 2022
commit f7919b87390523492d317125122b243dc054992b
9 changes: 6 additions & 3 deletions worker/cp-worker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,10 +184,11 @@ def runCellProfiler(message):
if DOWNLOAD_FILES.lower() == 'true':
printandlog('Figuring which files to download', logger)
import pandas
s3 = boto3.resource('s3')
s3client = boto3.client('s3')
if not os.path.exists(localIn):
os.mkdir(localIn)
csv_in = pandas.read_csv(os.path.join(DATA_ROOT,message['data_file']))
s3client.download_file(SOURCE_BUCKET, message['data_file'], os.path.join(localIn,'load_data.csv'))
csv_in = pandas.read_csv(os.path.join(localIn,'load_data.csv'))
csv_in=csv_in.astype('str')
#Figure out what metadata fields we need in this experiment, as a dict
if type(message['Metadata'])==dict:
Expand All @@ -212,7 +213,9 @@ def runCellProfiler(message):
os.makedirs(os.path.split(new_file_name)[0])
printandlog('made directory '+os.path.split(new_file_name)[0],logger)
if not os.path.exists(new_file_name):
s3.meta.client.download_file(INPUT_BUCKET,prefix_on_bucket,new_file_name)
printandlog(prefix_on_bucket, logger)
printandlog(new_file_name, logger)
s3client.download_file(SOURCE_BUCKET,prefix_on_bucket,new_file_name)
downloaded_files.append(new_file_name)
printandlog('Downloaded '+str(len(downloaded_files))+' files',logger)
import random
Expand Down