[Rd] reference classes, LAZY_DUPLICATE_OK, and external pointers

[Ben Bolker]

We (the lme4 authors) are having a problem with doing a proper deep
copy of a reference class object in recent versions of R-devel with
the LAZY_DUPLICATE_OK flag in src/main/bind.c enabled.

Apologies in advance for any improper terminology.

TL;DR Is there an elegant way to force non-lazy/deep copying in our
case? Is anyone else using reference classes with a field that is an
external pointer?

This is how copying of reference classes works in a normal
situation:

library(data.table) ## for address() function
setRefClass("defaultRC",fields="theta")
d1 <- new("defaultRC")
d1$theta <- 1
address(d1$theta)  ##  "0xbbbbb70"
d2 <- d1$copy()
address(d2$theta)  ## same as above
d2$theta <- 2
address(d2$theta)  ## now modified, by magic
d1$theta  ## unmodified

The extra complication in our case is that many of the objects within
our reference class are actually accessed via an external pointer,
which is initialized when necessary -- details are copied below for
those who want them, or you can see the code at
https://github.com/lme4/lme4

The problem is that this sneaky way of copying the object's contents
doesn't trigger R's (new) rules for recognizing that a non-lazy copy
should be made.

library(lme4)
fm1 <- lmer(Reaction ~ Days + (Days|Subject), sleepstudy)
pp <- fm1 at pp
pp$theta ## [1] 0.96673279 0.01516906 0.23090960
address(pp$theta) ## something
pp$Ptr ## <pointer: ...>
xpp <- pp$copy() ## default is deep copy
xpp$Ptr  ## <pointer: (nil)>
address(xpp$theta)  ## same as above
xpp$setTheta(c(0,0,0)) ## referenced through Ptr field
xpp$Ptr  ## now set to non-nil
fm1 at pp$theta  ## changes to (0,0,0).  oops.

So apparently when the xpp$theta object is copied into the external
pointer, a reference/lazy copy is made.   (xpp$theta itself is
read-only, so I can't do the assignment that way)

I can hack around this in a very ugly way by doing a trivial
modification when assigning inside the copy method:

assign("theta",get("theta",envir=selfEnv)+0, envir=vEnv)

... but (a) this is very ugly and (b) it seems very unsafe --
as R gets smarter it should start to recognize trivial changes
like x+0 and x*1 and *not* copy in these cases ...

Method details:

## from R/AllClass.R, merPredD RC definition

ptr          = function() {
      'returns the external pointer, regenerating if necessary'
      if (length(theta)) {
            if (.Call(isNullExtPtr, Ptr)) initializePtr()
      }
      Ptr
},

## ditto

initializePtr = function() {
    Ptr <<- .Call(merPredDCreate, as(X, "matrix"), Lambdat,
                  LamtUt, Lind, RZX, Ut, Utr, V, VtV, Vtr,
                  Xwts, Zt, beta0, delb, delu, theta, u0)
 ...
}

merPredDCreate in turn just copies the relevant bits into a new C++
class object:

/* see src/external.cpp */

    SEXP merPredDCreate(SEXP Xs, SEXP Lambdat, SEXP LamtUt, SEXP Lind,
                        SEXP RZX, SEXP Ut, SEXP Utr, SEXP V, SEXP VtV,
                        SEXP Vtr, SEXP Xwts, SEXP Zt, SEXP beta0,
                        SEXP delb, SEXP delu, SEXP theta, SEXP u0) {
        BEGIN_RCPP;
        merPredD *ans = new merPredD(Xs, Lambdat, LamtUt, Lind, RZX,
Ut, Utr, V, VtV,
                                     Vtr, Xwts, Zt, beta0, delb, delu,
theta, u0);
        return wrap(XPtr<merPredD>(ans, true));
        END_RCPP;
    }