Since porting to 2.1.0, Dataflow is leaving Datasets/Tables behind in BigQuery #609

Activity

1. nguyent commented on Oct 10, 2017

   

   nguyent

   on Oct 10, 2017

   Last edited by nguyent

   More actions

   I haven't seen anything like this in 2.0.0; we run batch jobs on a daily basis and have restarted our streaming pipelines a few times now.

   Is this in streaming, batch, or both?
   What is the delta between job cancellation time and table creation/update time?
   Is there a reproducible case?
2. polleyg commented on Oct 13, 2017

   

   polleyg

   on Oct 13, 2017

   Author

   More actions

   Only seen it in batch so far, and cannot reproduce yet.
3. jamespercy commented on Dec 12, 2017

   

   jamespercy

   on Dec 12, 2017

   Last edited by jamespercy

   More actions

   Still happening in 2.2.0 templated batch jobs on our side. We're currently managing it with cleanup scripts but it's a PITA.
   I was wondering if it might be an idea to put an expiry on those datasets to auto cleanup?
   I guess determining that might be difficult depending on how long a batch can run for but a day seems safe. At least that would limit the number of temp files to 24 if one were running them every hour.
4. polleyg commented on Dec 12, 2017

   

   polleyg

   on Dec 12, 2017

   Author

   More actions

   I was just thinking about this today because it happened yet again. Agree, auto expire on the datasets makes sense.
5. jamespercy commented on Dec 12, 2017

   

   jamespercy

   on Dec 12, 2017

   More actions

   So I did a little investigation and it does look like that's actually implemented... not sure why it's still happening though.

   https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java

   LOG.info("Creating temporary dataset {} for query results", tableToExtract.getDatasetId());
   tableService.createDataset(
       tableToExtract.getProjectId(),
       tableToExtract.getDatasetId(),
       location,
       "Temporary tables for query results of job " + bqOptions.getJobName(),
       // Set a TTL of 1 day on the temporary tables, which ought to be enough in all cases:
       // the temporary tables are used only to immediately extract them into files.
       // They are normally cleaned up, but in case of job failure the cleanup step may not run,
       // and then they'll get deleted after the TTL.
       24 * 3600 * 1000L /* 1 day */);
   

   I think I'll try do a bit more debugging of my own... p.s. is this the correct forum to be discussing this?
6. lukecwik commented on Dec 12, 2017

   

   lukecwik

   on Dec 12, 2017

   Contributor

   More actions

   dev@beam.apache.org is a good place and also by opening a tracking issue on https://issues.apache.org/jira/projects/BEAM so people can follow the bug.
7. rumeshkrish commented on Jan 25, 2019

   

   rumeshkrish

   on Jan 25, 2019

   More actions

   I am also facing this issue. If job failed, I observed that table got delete after 1 day. But DataSet still remain exist. Can we have option to clean temp dataset and tables immediately if job failed. ?
   This method cleanupTempResource(options.as(BigQueryOptions.class)); is responsible for cleaning temp dataset and tables. This executed if job succeed in public List<BoundedSource<T>> split(long desiredBundleSizeBytes, PipelineOptions options) method call . If any error also we need to clean based on pipeline option.

   Can any one have better idea. ?
8. lobdellb commented on May 19, 2021

   

   lobdellb

   on May 19, 2021

   More actions

   I have this as well, python sdk version 2.27 running on Google dataflow. See attached.

   Would be nice if the tables would at least expire automatically. Or if the temp dataset name was configurable. Or something else.

   

   

   👍React with 👍2Reacted by Carl Thomé and Mykola Morozov
9. kennknowles commented on May 19, 2021

   

   kennknowles

   on May 19, 2021

   Contributor

   More actions

   Check out https://beam.apache.org/community/contact-us/ for ways to reach the Beam community with bug reports and questions.