Lustre extractor #19

Activity

1. blastmaster

   added

   enhancementNew feature or requestNew feature or request

   on Nov 7, 2018
2. blastmaster

   assigned

   blastmaster

   and

   k4lipso

   and unassigned

   blastmaster

   on Nov 7, 2018
3. blastmaster commented on Nov 22, 2018

   

   blastmaster

   on Nov 22, 2018

   Last edited by blastmaster

   OwnerAuthor

   More actions

   Hint to use the lustre API, include #include <lustre/lustreapi.h> and link against liblustreapi -llustreapi afterward. Lustre is a system installation on taurus, under /usr/lib64..., therefore loading additional modules is not necessary.
4. k4lipso commented on Nov 22, 2018

   

   k4lipso

   on Nov 22, 2018

   Last edited by k4lipso

   Collaborator

   More actions

   Overview of lfs commands. If they have a tick they are working and will be implemented.

   file required

   • lfs getstripe
   • lfs data_version
   • lfs hsm_state

   no file required

   • lfs quota - works partially "Some errors happened when getting quota info."
   • lfs check <osts|mds|server> - help i dont get it to work, what can i provide as "server"?
   • lfs getdirstripe - looks like it is allways returning the same
   • lfs df
   • lfs getname
   • lfs path2fid
   • lfs pool_list - no errors, but also no pools listed
   • lfs osts
   • lfs mdts
5. k4lipso commented on Mar 19, 2019

   

   k4lipso

   on Mar 19, 2019

   Collaborator

   More actions

   @blastmaster
   For gathering striping information we have to create a dummy file on a lustre mountpoint. Functions for dummy file creation and getting file striping information allready exists, but we still need to find a place to create that file.

   Right now, to find a place, we gather every lustre mountpoint using the lustreapi and then pass the mountpoint path to the following function:
   

   sysmap/lib/src/linux/lustre_extractor.cc

   Lines 298 to 322 in 3bf46ac

   	bool Lustre_Extractor::handle_getstripe(const std::string& mount_path, data& result)
   	{
   	//get username of current user
   	const std::string username = getlogin();
   	//create path from mountpoint of lustre filesystem and username
   	const std::string userpath = mount_path + '/' + username;
   	//check if path exists
   	if(boost::filesystem::exists(userpath)){
   	//create path to yet non existing dummy file using userpath
   	std::string dummy_file = userpath + "/lustre_dummy_test_file";
   	//try to create dummy file
   	if(create_dummy_file(dummy_file)){
   	//gather striping information of the dummy file
   	file_getstripe(dummy_file, result);
   	//try to remove dummy file
   	if( std::remove(dummy_file.c_str()) ){
   	utils::log::logging::error()
   	<< "[Lustre_Extractor] could not delete dummy file:";
   	}
   	return true;
   	}
   	}
   	return false;
   	}

   Basicly this function gets the username of the current user, appends it to the mountpoint path, checks if that path exists and if that is the case it creates the dummy file.
   The problem is that the dummy file creation only works if an folder named like the current user exists on that mountpoint. This approach could fail on other HPCs.
   Another approach would be to check every folder on the mountpoint for access rights which could be a very time intensive task.
   Do you have any recommendation on handling this problem?