Skip to content
This repository was archived by the owner on Nov 11, 2022. It is now read-only.
This repository was archived by the owner on Nov 11, 2022. It is now read-only.

TextIO - add "allow empty" flag #556

Open

Description

@rantibi

Since SDK 1.9:
Changed FileBasedSource to throw an exception when reading from a file pattern that has no matches. Pipelines will now fail at runtime rather than silently reading no data in this case.

In some of our pipelines we read from multiple buckets, and some of them could be empty in some cases (we still process data from the other buckets).
For this reason, while trying to upgrade to 1.9 we have to change our code so that if will check first if there is files in the bucket and return empty PCollection in case there is no files that match to the pattern.

It will be very useful if TextIO will have a flag allowEmpty() that will allow silently reading no data (like withoutValidation() in the init time).

Activity

  1. davorbonaci commented on Mar 6, 2017

    @davorbonaci
    Contributor

    Indeed -- this would be a nice improvement.

    Would you perhaps be interested in contributing that feature to the Apache Beam codebase in https://github.com/apache/beam?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions