Skip to content

Commit c75965f

Browse files
jorisvandenbosschejorisvandenbossche
committed
ARROW-14318: [Docs] Fix doc building of dataset docs multiple times
Closes apache#11479 from jorisvandenbossche/ARROW-14318 Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
1 parent 5f6c92a commit c75965f

3 files changed

Lines changed: 78 additions & 17 deletions

File tree

‎docs/source/python/dataset.rst‎

Lines changed: 34 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,17 @@
1515
.. specific language governing permissions and limitations
1616
.. under the License.
1717
18+
.. ipython:: python
19+
:suppress:
20+
21+
# set custom tmp working directory for files that create data
22+
import os
23+
import tempfile
24+
25+
orig_working_dir = os.getcwd()
26+
temp_working_dir = tempfile.mkdtemp(prefix="pyarrow-")
27+
os.chdir(temp_working_dir)
28+
1829
.. currentmodule:: pyarrow.dataset
1930

2031
.. _dataset:
@@ -66,7 +77,7 @@ of a directory with two parquet files:
6677
import pyarrow.parquet as pq
6778
import numpy as np
6879
69-
base = pathlib.Path(tempfile.gettempdir())
80+
base = pathlib.Path(tempfile.mkdtemp(prefix="pyarrow-"))
7081
(base / "parquet_dataset").mkdir(exist_ok=True)
7182
7283
# creating an Arrow Table
@@ -265,7 +276,7 @@ function can write such hive-like partitioned datasets.
265276
266277
table = pa.table({'a': range(10), 'b': np.random.randn(10), 'c': [1, 2] * 5,
267278
'part': ['a'] * 5 + ['b'] * 5})
268-
pq.write_to_dataset(table, str(base / "parquet_dataset_partitioned"),
279+
pq.write_to_dataset(table, "parquet_dataset_partitioned",
269280
partition_cols=['part'])
270281
271282
The above created a directory with two subdirectories ("part=a" and "part=b"),
@@ -277,7 +288,7 @@ should use a hive-like partitioning scheme with the ``partitioning`` keyword:
277288

278289
.. ipython:: python
279290
280-
dataset = ds.dataset(str(base / "parquet_dataset_partitioned"), format="parquet",
291+
dataset = ds.dataset("parquet_dataset_partitioned", format="parquet",
281292
partitioning="hive")
282293
dataset.files
283294
@@ -513,12 +524,8 @@ instead of a filename.
513524

514525
.. ipython:: python
515526
516-
base = pathlib.Path(tempfile.gettempdir())
517-
dataset_root = base / "sample_dataset"
518-
dataset_root.mkdir(exist_ok=True)
519-
520527
table = pa.table({"a": range(10), "b": np.random.randn(10), "c": [1, 2] * 5})
521-
ds.write_dataset(table, dataset_root, format="parquet")
528+
ds.write_dataset(table, "sample_dataset", format="parquet")
522529
523530
The above example will create a single file named part-0.parquet in our sample_dataset
524531
directory.
@@ -543,7 +550,7 @@ dataset to be partitioned. For example:
543550
part = ds.partitioning(
544551
pa.schema([("c", pa.int16())]), flavor="hive"
545552
)
546-
ds.write_dataset(table, dataset_root, format="parquet", partitioning=part)
553+
ds.write_dataset(table, "sample_dataset", format="parquet", partitioning=part)
547554
548555
This will create two files. Half our data will be in the dataset_root/c=1 directory and
549556
the other half will be in the dataset_root/c=2 directory.
@@ -565,16 +572,15 @@ into memory:
565572
new_part = ds.partitioning(
566573
pa.schema([("c", pa.int16())]), flavor=None
567574
)
568-
input_dataset = ds.dataset(dataset_root, partitioning=old_part)
569-
new_root = base / "repartitioned_dataset"
575+
input_dataset = ds.dataset("sample_dataset", partitioning=old_part)
570576
# A scanner can act as an iterator of record batches but you could also receive
571577
# data from the network (e.g. via flight), from your own scanning, or from any
572578
# other method that yields record batches. In addition, you can pass a dataset
573579
# into write_dataset directly but this method is useful if you want to customize
574580
# the scanner (e.g. to filter the input dataset or set a maximum batch size)
575581
scanner = input_dataset.scanner(use_async=True)
576582
577-
ds.write_dataset(scanner, new_root, format="parquet", partitioning=new_part)
583+
ds.write_dataset(scanner, "repartitioned_dataset", format="parquet", partitioning=new_part)
578584
579585
After the above example runs our data will be in dataset_root/1 and dataset_root/2
580586
directories. In this simple example we are not changing the structure of the data
@@ -600,7 +606,7 @@ to supply a visitor that will be called as each file is created:
600606
601607
.. ipython:: python
602608
603-
ds.write_dataset(table, base / "dataset_visited", format="parquet", partitioning=part,
609+
ds.write_dataset(table, "dataset_visited", format="parquet", partitioning=part,
604610
file_visitor=file_visitor)
605611
606612
This will allow you to collect the filenames that belong to the dataset and store them elsewhere
@@ -617,10 +623,21 @@ Parquet files:
617623

618624
.. ipython:: python
619625
620-
dataset_root = base / "sample_dataset2"
621-
dataset_root.mkdir(exist_ok=True)
622-
623626
parquet_format = ds.ParquetFileFormat()
624627
write_options = parquet_format.make_write_options(allow_truncated_timestamps=True)
625-
ds.write_dataset(table, dataset_root, format="parquet", partitioning=part,
628+
ds.write_dataset(table, "sample_dataset2", format="parquet", partitioning=part,
626629
file_options=write_options)
630+
631+
632+
.. ipython:: python
633+
:suppress:
634+
635+
# clean-up custom working directory
636+
import os
637+
import shutil
638+
639+
os.chdir(orig_working_dir)
640+
shutil.rmtree(temp_working_dir, ignore_errors=True)
641+
642+
# also clean-up custom base directory used in some examples
643+
shutil.rmtree(str(base), ignore_errors=True)

‎docs/source/python/getstarted.rst‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,17 @@
1515
.. specific language governing permissions and limitations
1616
.. under the License.
1717
18+
.. ipython:: python
19+
:suppress:
20+
21+
# set custom tmp working directory for files that create data
22+
import os
23+
import tempfile
24+
25+
orig_working_dir = os.getcwd()
26+
temp_working_dir = tempfile.mkdtemp(prefix="pyarrow-")
27+
os.chdir(temp_working_dir)
28+
1829
.. _getstarted:
1930

2031
Getting Started
@@ -143,3 +154,14 @@ Continuining from here
143154
For digging further into Arrow, you might want to read the
144155
:doc:`PyArrow Documentation <./index>` itself or the
145156
`Arrow Python Cookbook <https://arrow.apache.org/cookbook/py/>`_
157+
158+
159+
.. ipython:: python
160+
:suppress:
161+
162+
# clean-up custom working directory
163+
import os
164+
import shutil
165+
166+
os.chdir(orig_working_dir)
167+
shutil.rmtree(temp_working_dir, ignore_errors=True)

‎docs/source/python/ipc.rst‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,17 @@
1515
.. specific language governing permissions and limitations
1616
.. under the License.
1717
18+
.. ipython:: python
19+
:suppress:
20+
21+
# set custom tmp working directory for files that create data
22+
import os
23+
import tempfile
24+
25+
orig_working_dir = os.getcwd()
26+
temp_working_dir = tempfile.mkdtemp(prefix="pyarrow-")
27+
os.chdir(temp_working_dir)
28+
1829
.. currentmodule:: pyarrow
1930

2031
.. _ipc:
@@ -383,3 +394,14 @@ An object can be reconstructed from its component-based representation using
383394
384395
``deserialize_components`` is also available as a method on
385396
``SerializationContext`` objects.
397+
398+
399+
.. ipython:: python
400+
:suppress:
401+
402+
# clean-up custom working directory
403+
import os
404+
import shutil
405+
406+
os.chdir(orig_working_dir)
407+
shutil.rmtree(temp_working_dir, ignore_errors=True)

0 commit comments

Comments
 (0)