Skip to content

Commit 3da09e5

Browse files
amol-pitrou
amol-
authored and
pitrou
committed
ARROW-13404: [Doc][Python] Improve PyArrow documentation for new users
* Add link to the cookbooks * Improve a bit landing page for PyArrow for people that don't already know Arrow * Add a Getting Started section to introduce people to Arrow and PyArrow Closes apache#10999 from amol-/ARROW-13404 Authored-by: Alessandro Molina <amol@turbogears.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
1 parent ff5572c commit 3da09e5

3 files changed

Lines changed: 166 additions & 6 deletions

File tree

‎docs/source/index.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,16 @@ target environment.**
5555
Rust <https://docs.rs/crate/arrow/>
5656
status
5757

58+
.. _toc.cookbook:
59+
60+
.. toctree::
61+
:maxdepth: 1
62+
:caption: Cookbooks
63+
64+
C++ <https://arrow.apache.org/cookbook/cpp/>
65+
Python <https://arrow.apache.org/cookbook/py/>
66+
R <https://arrow.apache.org/cookbook/r/>
67+
5868
.. _toc.columnar:
5969

6070
.. toctree::

‎docs/source/python/getstarted.rst‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one
2+
.. or more contributor license agreements. See the NOTICE file
3+
.. distributed with this work for additional information
4+
.. regarding copyright ownership. The ASF licenses this file
5+
.. to you under the Apache License, Version 2.0 (the
6+
.. "License"); you may not use this file except in compliance
7+
.. with the License. You may obtain a copy of the License at
8+
9+
.. http://www.apache.org/licenses/LICENSE-2.0
10+
11+
.. Unless required by applicable law or agreed to in writing,
12+
.. software distributed under the License is distributed on an
13+
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
.. KIND, either express or implied. See the License for the
15+
.. specific language governing permissions and limitations
16+
.. under the License.
17+
18+
.. _getstarted:
19+
20+
Getting Started
21+
===============
22+
23+
Arrow manages data in arrays (:class:`pyarrow.Array`), which can be
24+
grouped in tables (:class:`pyarrow.Table`) to represent columns of data
25+
in tabular data.
26+
27+
Arrow also provides support for various formats to get those tabular
28+
data in and out of disk and networks. Most commonly used formats are
29+
Parquet (:ref:`parquet`) and the IPC format (:ref:`ipc`).
30+
31+
Creating Arrays and Tables
32+
--------------------------
33+
34+
Arrays in Arrow are collections of data of uniform type. That allows
35+
Arrow to use the best performing implementation to store the data and
36+
perform computations on it. So each array is meant to have data and
37+
a type
38+
39+
.. ipython:: python
40+
41+
import pyarrow as pa
42+
43+
days = pa.array([1, 12, 17, 23, 28], type=pa.int8())
44+
45+
Multiple arrays can be combined in tables to form the columns
46+
in tabular data when attached to a column name
47+
48+
.. ipython:: python
49+
50+
months = pa.array([1, 3, 5, 7, 1], type=pa.int8())
51+
years = pa.array([1990, 2000, 1995, 2000, 1995], type=pa.int16())
52+
53+
birthdays_table = pa.table([days, months, years],
54+
names=["days", "months", "years"])
55+
56+
birthdays_table
57+
58+
See :ref:`data` for more details.
59+
60+
Saving and Loading Tables
61+
-------------------------
62+
63+
Once you have tabular data, Arrow provides out of the box
64+
the features to save and restore that data for common formats
65+
like Parquet:
66+
67+
.. ipython:: python
68+
69+
import pyarrow.parquet as pq
70+
71+
pq.write_table(birthdays_table, 'birthdays.parquet')
72+
73+
Once you have your data on disk, loading it back is a single function call,
74+
and Arrow is heavily optimized for memory and speed so loading
75+
data will be as quick as possible
76+
77+
.. ipython:: python
78+
79+
reloaded_birthdays = pq.read_table('birthdays.parquet')
80+
81+
reloaded_birthdays
82+
83+
Saving and loading back data in arrow is usually done through
84+
:ref:`Parquet <parquet>`, :ref:`IPC format <ipc>` (:ref:`feather`),
85+
:ref:`CSV <csv>` or :ref:`Line-Delimited JSON <json>` formats.
86+
87+
Performing Computations
88+
-----------------------
89+
90+
Arrow ships with a bunch of compute functions that can be applied
91+
to its arrays and tables, so through the compute functions
92+
it's possible to apply transformations to the data
93+
94+
.. ipython:: python
95+
96+
import pyarrow.compute as pc
97+
98+
pc.value_counts(birthdays_table["years"])
99+
100+
See :ref:`compute` for a list of available compute functions and
101+
how to use them.
102+
103+
Working with large data
104+
-----------------------
105+
106+
Arrow also provides the :class:`pyarrow.dataset` API to work with
107+
large data, which will handle for you partitioning of your data in
108+
smaller chunks
109+
110+
.. ipython:: python
111+
112+
import pyarrow.dataset as ds
113+
114+
ds.write_dataset(birthdays_table, "savedir", format="parquet",
115+
partitioning=ds.partitioning(
116+
pa.schema([birthdays_table.schema.field("years")])
117+
))
118+
119+
Loading back the partitioned dataset will detect the chunks
120+
121+
.. ipython:: python
122+
123+
birthdays_dataset = ds.dataset("savedir", format="parquet", partitioning=["years"])
124+
125+
birthdays_dataset.files
126+
127+
and will lazily load chunks of data only when iterating over them
128+
129+
.. ipython:: python
130+
131+
import datetime
132+
133+
current_year = datetime.datetime.utcnow().year
134+
for table_chunk in birthdays_dataset.to_batches():
135+
print("AGES", pc.subtract(current_year, table_chunk["years"]))
136+
137+
For further details on how to work with big datasets, how to filter them,
138+
how to project them, etc., refer to :ref:`dataset` documentation.
139+
140+
Continuining from here
141+
----------------------
142+
143+
For digging further into Arrow, you might want to read the
144+
:doc:`PyArrow Documentation <./index>` itself or the
145+
`Arrow Python Cookbook <https://arrow.apache.org/cookbook/py/>`_

‎docs/source/python/index.rst‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,16 @@
1515
.. specific language governing permissions and limitations
1616
.. under the License.
1717
18-
Python bindings
19-
===============
18+
PyArrow - Apache Arrow Python bindings
19+
======================================
2020

21-
This is the documentation of the Python API of Apache Arrow. For more details
22-
on the Arrow format and other language bindings see the
23-
:doc:`parent documentation <../index>`.
21+
This is the documentation of the Python API of Apache Arrow.
22+
23+
Apache Arrow is a development platform for in-memory analytics.
24+
It contains a set of technologies that enable big data systems to store, process and move data fast.
25+
26+
See the :doc:`parent documentation <../index>` for additional details on
27+
the Arrow Project itself, on the Arrow format and the other language bindings.
2428

2529
The Arrow Python bindings (also named "PyArrow") have first-class integration
2630
with NumPy, pandas, and built-in Python objects. They are based on the C++
@@ -34,9 +38,10 @@ files into Arrow structures.
3438
:maxdepth: 2
3539

3640
install
37-
memory
41+
getstarted
3842
data
3943
compute
44+
memory
4045
ipc
4146
filesystems
4247
filesystems_deprecated

0 commit comments

Comments
 (0)