Skip to content

Add file download validation #1024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aalhossary merged 8 commits into from
Jan 26, 2023
Merged

Add file download validation #1024

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b8e69a4
Two utility methods to validate the downloaded file
aalhossary Feb 20, 2022
aed24fa
JavaDock
aalhossary Feb 21, 2022
f59de95
more verbose
aalhossary Feb 22, 2022
a26adde
Unit testing
aalhossary Feb 22, 2022
06cde1b
Minor corrections
aalhossary Dec 23, 2022
630bb64
Addressing reviewer's comments
aalhossary Dec 24, 2022
3c7b04f
Adding placeholder for several common hashing algorithms
aalhossary Dec 24, 2022
07e65b2
Addressing reviewer's comments
aalhossary Jan 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
JavaDock
  • Loading branch information
@aalhossary
aalhossary committed Feb 21, 2022
commit aed24fad9664484780dce00fb3f073921e34ae36
38 changes: 38 additions & 0 deletions biojava-core/src/main/java/org/biojava/nbio/core/util/FileDownloadUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,17 @@ public static void downloadFile(URL url, File destination) throws IOException {

}

/**
* Creates validation files beside a file to be downloaded.<br>
* Whenever possible, for a <code>file.ext</code> file, it creates
* <code>file.ext.size</code> and <code>file.hash</code> for in the same
* folder where <code>file.ext</code> exists.
* If the file connection size could not be deduced from the URL, no size file is created.
* If <code>hashURL</code> is <code>null</code>, no hash file is created.
* @param url the remote file URL to download
* @param localDestination the local file to download into
* @param hashURL the URL of the hash file to download. Can be <code>null</code>.
*/
public static void createValidationFiles(URL url, File localDestination, URL hashURL){
try {
URLConnection resourceConnection = url.openConnection();
Expand All @@ -173,6 +184,19 @@ public static void createValidationFiles(URL url, File localDestination, URL has
logger.warn("could not open connection to resource file due to exception", e);
Comment thread
aalhossary marked this conversation as resolved.
Outdated
}
}
/**
* Creates validation files beside a file to be downloaded.<br>
* Whenever possible, for a <code>file.ext</code> file, it creates
* <code>file.ext.size</code> and <code>file.hash</code> for in the same
* folder where <code>file.ext</code> exists.
* If the file connection size could not be deduced from the resourceUrlConnection
* {@link URLConnection}, no size file is created.
* If <code>hashURL</code> is <code>null</code>, no hash file is created.
* @param resourceUrlConnection the remote file URLConnection to download
* @param localDestination the local file to download into
* @param hashURL the URL of the hash file to download. Can be <code>null</code>.
* @since 6.0.6
*/
public static void createValidationFiles(URLConnection resourceUrlConnection, File localDestination, URL hashURL){
long size = resourceUrlConnection.getContentLengthLong();
if(size != -1) {
Expand All @@ -197,6 +221,20 @@ public static void createValidationFiles(URLConnection resourceUrlConnection, Fi
}
}

/**
* Validate a local file based on pre-existing metadata files for size and hash.<br>
* If the passed in <code>localFile</code> parameter is a file named <code>file.ext</code>, the function searches in the same folder for:
* <ul>
* <li><code>file.ext.size</code>: if found, it compares the size stored in it to the length of <code>localFile</code> (in bytes).</li>
* <li><code>file.ext.hash</code>: if found, it compares the size stored in it to the hash code of <code>localFile</code>.</li>
* </ul>
* If any of these comparisons fail, the function returns <code>false</code>. otherwise it returns true.
* <p>
* This function does not implement hash code verification yet.
* @param localFile The file to validate
* @return <code>false</code> if any of the size or hash code metadata files exists but its contents does not match the expected value in the file, <code>true'</code> otherwise.
* @since 6.0.6
*/
public static boolean validateFile(File localFile) {
File sizeFile = new File(localFile.getParentFile(), localFile.getName()+".size");
if(sizeFile.exists()) {
Expand Down
4 changes: 2 additions & 2 deletions biojava-structure/src/main/java/org/biojava/nbio/structure/ecod/EcodInstallation.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,8 +395,8 @@ private boolean domainsAvailable() {
}

/**
* Downloads the domains file, overwriting any existing file
* @throws IOException
* Downloads the domains file +/- its validation metadata, overwriting any existing file
* @throws IOException in cases of file I/O, including failure to download a healthy (non-corrupted) file.
*/
private void downloadDomains() throws IOException {
domainsFileLock.writeLock().lock();
Expand Down
8 changes: 4 additions & 4 deletions biojava-structure/src/main/java/org/biojava/nbio/structure/io/LocalPDBDirectory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,7 +362,7 @@ public Structure getStructureById(PdbId pdbId) throws IOException {
* for direct parsing.
* @param pdbId
* @return
* @throws IOException
* @throws IOException in cases of file I/O, including failure to download a healthy (non-corrupted) file.
*/
protected InputStream getInputStream(PdbId pdbId) throws IOException{

Expand All @@ -388,7 +388,7 @@ protected InputStream getInputStream(PdbId pdbId) throws IOException{
*
* Used to pre-fetch large numbers of structures.
* @param pdbId
* @throws IOException
* @throws IOException in cases of file I/O, including failure to download a healthy (non-corrupted) file.
*/
public void prefetchStructure(String pdbId) throws IOException {

Expand Down Expand Up @@ -530,14 +530,14 @@ protected File downloadStructure(PdbId pdbId) throws IOException {
}

/**
* Download a file from the ftp server, replacing any existing files if needed
* Download a file from the ftp server +/- its validation metadata, replacing any existing files if needed
* @param pdbId PDB ID
* @param pathOnServer Path on the FTP server, e.g. data/structures/divided/pdb
* @param obsolete Whether or not file should be saved to the obsolete location locally
* @param existingFile if not null and checkServerFileDate is true, the last modified date of the
* server file and this file will be compared to decide whether to download or not
* @return
* @throws IOException
* @throws IOException in cases of file I/O, including failure to download a healthy (non-corrupted) file.
*/
private File downloadStructure(PdbId pdbId, String pathOnServer, boolean obsolete, File existingFile)
throws IOException{
Expand Down
6 changes: 6 additions & 0 deletions biojava-structure/src/main/java/org/biojava/nbio/structure/scop/ScopInstallation.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -739,6 +739,12 @@ protected void downloadComFile() throws FileNotFoundException, IOException{
throw new IOException("Unable to download SCOP .com file",exception);
}

/**
* Downloads the SCOP installation file +/- its validation metadata files.
* @param remoteURL The remote file to download
* @param localFile the local file to download to
* @throws IOException in cases of file I/O, including failure to download a healthy (non-corrupted) file.
*/
protected void downloadFileFromRemote(URL remoteURL, File localFile) throws IOException{
logger.info("Downloading " + remoteURL + " to: " + localFile);
FileDownloadUtils.createValidationFiles(remoteURL, localFile, null);
Expand Down