Skip to content

Commit 48ee38f

Browse files
romainfrancoisromainfrancois
committed
ARROW-5488: [R] Workaround when C++ lib not available
This is very much a work in progress. The idea is to replace the code generation that is usually done by `Rcpp::compileAttributes()` with something custom. This is driven by the `data-raw/codegen.R` file, which I'll polish. I'm opening this right now for feedback opportunities. All of the `.cpp` files are guarded by : ``` #if defined(ARROW_R_WITH_ARROW) ... #endif ``` And `ARROW_R_WITH_ARROW` is defined via the configure file, if the library was indeed used. For example, this function: ```cpp // [[arrow::export]] std::shared_ptr<arrow::Column> ipc___feather___TableReader__GetColumn( const std::unique_ptr<arrow::ipc::feather::TableReader>& reader, int i) { std::shared_ptr<arrow::Column> column; STOP_IF_NOT_OK(reader->GetColumn(i, &column)); return column; } ``` triggers generation of this in the `generated.cpp` file: ```cpp #if defined(ARROW_R_WITH_ARROW) std::string ipc___feather___TableReader__GetColumnName(const std::unique_ptr<arrow::ipc::feather::TableReader>& reader, int i); SEXP _arrow_ipc___feather___TableReader__GetColumnName(SEXP reader_sexp, SEXP i_sexp){ BEGIN_RCPP Rcpp::traits::input_parameter<const std::unique_ptr<arrow::ipc::feather::TableReader>&>::type reader(reader_sexp); Rcpp::traits::input_parameter<int>::type i(i_sexp);Rcpp::Shield<SEXP> rcpp_result_gen(Rcpp::wrap(ipc___feather___TableReader__GetColumnName( reader, i))); return rcpp_result_gen; END_RCPP } #else SEXP _arrow_ipc___feather___TableReader__GetColumnName(SEXP reader_sexp, SEXP i_sexp){ BEGIN_RCPP Rcpp::stop("arrow C++ library not available"); END_RCPP } #endif ``` So the generated R api `SEXP`y functions only call the real thing when the c++ library is available, otherwise they just throw an error. and this in the `generated.R` file: ```r ipc___feather___TableReader__GetColumnName <- function(reader, i) { .Call(`_arrow_ipc___feather___TableReader__GetColumnName` , reader, i ) } ``` This also needed some extra care in test functions so that the tests only run if Arrow is available. @wesm comment from https://issues.apache.org/jira/browse/ARROW-5488 might be more practical and closer to what @jjallaire mentioned about how the `RcppParallel` package does it with Intel tbb: https://github.com/RcppCore/RcppParallel/blob/master/R/hooks.R > One possibility is to bundle the Arrow header files with the CRAN package and build against them, but do not include libarrow and libparquet when linking. When the library is loaded, the libraries must be loaded in-process via dlopen before loading the Rcpp extensions. The C++ libraries can be installed then after the fact Author: Romain Francois <romain@rstudio.com> Author: Romain François <romain@rstudio.com> Closes apache#4471 from romainfrancois/ARROW-5488/workaround and squashes the following commits: d13dbfb <Romain Francois> update error message to mention arrow::install_arrow() 1b6a737 <Romain Francois> Merge branch 'ARROW-5488/workaround' of https://github.com/romainfrancois/arrow into ARROW-5488/workaround 418fc2d <Romain Francois> mention codegen in the README 9a1b3e6 <Romain Francois> glue() not needed here 496eaf5 <Romain Francois> when brew is available but apache-arrow is not installed, install it b8a6576 <Romain Francois> no need for stringr 238fa14 <Romain Francois> rm message about ARROW_R_DEV not being set 9788811 <Romain François> Merge branch 'master' into ARROW-5488/workaround 346978d <Romain Francois> update docker files. bb9eed2 <Romain Francois> not showing diff for generated files 1df1e39 <Romain Francois> phrase shim message more positively 3b04369 <Romain Francois> use `\dontrun{}` in examples 562670d <Romain Francois> RcppExport all the things 6c06581 <Romain Francois> Using arrow:::arrow_available() 75b0751 <Romain Francois> Move symbols.h back into arrow_types.h bd7f30d <Romain Francois> 🐀 e8ac3ea <Romain Francois> using arrowExports.(R,cpp). update configure script 84fa149 <Romain Francois> lint 54ee38d <Romain Francois> 🐀 d9e4f94 <Romain Francois> added a second R job to install and check the package on a system without libarrow 0479a75 <Romain Francois> code generation without rap package, using purrr instead 008eaf4 <Romain Francois> update generated code, fix merge conflicts f957504 <Romain Francois> update test_that shim 1170236 <Romain Francois> not necessary 1ed5596 <Romain Francois> Workaround so that the R package still checks without the C++ library. 78f69af <Romain Francois> move symbols declaration to their own file, might end up be generated automatically later somehow.
1 parent 9c753c4 commit 48ee38f

49 files changed

Lines changed: 4997 additions & 3746 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitattributes‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
r/R/RcppExports.R linguist-generated=true
2+
r/R/arrowExports.R linguist-generated=true
23
r/src/RcppExports.cpp linguist-generated=true
4+
r/src/arrowExports.cpp linguist-generated=true
35
r/man/*.Rd linguist-generated=true
46

‎.travis.yml‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -365,7 +365,22 @@ matrix:
365365
- pushd ${TRAVIS_BUILD_DIR}/r
366366
after_success:
367367
- Rscript ../ci/travis_upload_r_coverage.R
368-
368+
- name: R_no_libarrow
369+
language: r
370+
cache: packages
371+
latex: false
372+
dist: xenial
373+
before_install:
374+
# Have to copy-paste this here because of how R's build steps work
375+
- eval `python $TRAVIS_BUILD_DIR/ci/detect-changes.py`
376+
- if [ $ARROW_CI_R_AFFECTED != "1" ]; then exit; fi
377+
- |
378+
if [ $TRAVIS_OS_NAME == "linux" ]; then
379+
sudo bash -c "echo -e 'Acquire::Retries 10; Acquire::http::Timeout \"20\";' > /etc/apt/apt.conf.d/99-travis-retry"
380+
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
381+
sudo apt-get update -qq
382+
fi
383+
- pushd ${TRAVIS_BUILD_DIR}/r
369384

370385
after_failure:
371386
- |

‎ci/docker_build_r.sh‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,6 @@ export ARROW_HOME=$CONDA_PREFIX
2323
# Build arrow
2424
pushd /arrow/r
2525

26-
rm src/RcppExports*
27-
Rscript -e "Rcpp::compileAttributes()"
2826
R CMD build --keep-empty-dirs .
2927
R CMD INSTALL $(ls | grep arrow_*.tar.gz)
3028

‎cpp/build-support/lint_exclusions.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,4 @@
88
*thirdparty/ae/*
99
*vendored/*
1010
*RcppExports.cpp*
11+
*arrowExports.cpp*

‎dev/release/rat_exclude_files.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -193,8 +193,8 @@ csharp/test/Directory.Build.props
193193
*.svg
194194
*.devhelp2
195195
*.scss
196-
r/R/RcppExports.R
197-
r/src/RcppExports.cpp
196+
r/R/arrowExports.R
197+
r/src/arrowExports.cpp
198198
r/DESCRIPTION
199199
r/LICENSE.md
200200
r/NAMESPACE

‎r/DESCRIPTION‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,14 +52,14 @@ Collate:
5252
'Column.R'
5353
'Field.R'
5454
'List.R'
55-
'RcppExports.R'
5655
'RecordBatch.R'
5756
'RecordBatchReader.R'
5857
'RecordBatchWriter.R'
5958
'Schema.R'
6059
'Struct.R'
6160
'Table.R'
6261
'array.R'
62+
'arrowExports.R'
6363
'buffer.R'
6464
'io.R'
6565
'compression.R'

‎r/Dockerfile‎

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -48,14 +48,10 @@ RUN apt update && \
4848
# R CMD CHECK --as-cran needs pdflatex to build the package manual
4949
apt install -y \
5050
texlive-latex-base && \
51-
# Install vctrs from Github
52-
Rscript -e "install.packages('devtools', repos = 'https://cran.rstudio.com')" && \
53-
Rscript -e "devtools::install_github('romainfrancois/vctrs@bit64')" && \
54-
Rscript -e "devtools::install_github('r-lib/withr')" && \
55-
Rscript -e "devtools::install_github('RcppCore/Rcpp')" && \
56-
# R is not good at picking up dependencies and installing them automatically
51+
Rscript -e "install.packages('devtools', repos = 'http://cran.rstudio.com')" && \
52+
Rscript -e "devtools::install_github('romainfrancois/decor')" && \
5753
Rscript -e "install.packages(c( \
58-
'Rcpp', \
54+
'Rcpp', 'dplyr', 'stringr', 'glue', 'vctrs', \
5955
'purrr', \
6056
'assertthat', \
6157
'fs', \
@@ -67,6 +63,9 @@ RUN apt update && \
6763
'lubridate'), \
6864
repos = 'https://cran.rstudio.com')"
6965

66+
# So that arrowExports.* files are generated
67+
ENV ARROW_R_DEV=TRUE
68+
7069
# Tell R where it can find the source code for arrow
7170
ENV PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/build/cpp/src/arrow:/opt/conda/lib/pkgconfig
7271
ENV LD_LIBRARY_PATH=/opt/conda/lib/:/build/cpp/src/arrow:/arrow/r/src

‎r/NAMESPACE‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,6 @@ export(FeatherTableWriter)
9696
export(FileMode)
9797
export(FileOutputStream)
9898
export(FixedSizeBufferWriter)
99-
export(GetCpuThreadPoolCapacity)
10099
export(MessageReader)
101100
export(MessageType)
102101
export(MockOutputStream)
@@ -105,11 +104,11 @@ export(RecordBatchFileReader)
105104
export(RecordBatchFileWriter)
106105
export(RecordBatchStreamReader)
107106
export(RecordBatchStreamWriter)
108-
export(SetCpuThreadPoolCapacity)
109107
export(StatusCode)
110108
export(TimeUnit)
111109
export(Type)
112110
export(array)
111+
export(arrow_available)
113112
export(as_tibble)
114113
export(boolean)
115114
export(buffer)

‎r/R/Field.R‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,14 +64,17 @@
6464
#' @param metadata currently ignored
6565
#'
6666
#' @examples
67-
#' field("x", int32())
67+
#'
68+
#' \dontrun{
69+
#' field("x", int32())
70+
#' }
6871
#'
6972
#' @export
7073
field <- function(name, type, metadata) {
7174
assert_that(inherits(name, "character"), length(name) == 1L)
7275
assert_that(inherits(type, "arrow::DataType"))
7376
assert_that(missing(metadata), msg = "metadata= is currently ignored")
74-
shared_ptr(`arrow::Field`, Field__initialize(name, type))
77+
shared_ptr(`arrow::Field`, Field__initialize(name, type, TRUE))
7578
}
7679

7780
.fields <- function(.list){

0 commit comments

Comments
 (0)