[R-sig-Geo] memory leaking of some rgeos functions?

Mon Oct 31 10:47:28 CET 2011

On Sat, 29 Oct 2011, Roger Bivand wrote:

> On Sat, 29 Oct 2011, Roman Luštrik wrote:
>
>> Dear list,
>> 
>> I've been looking into this and it would seem that some function in rgeos,
>> namely gBuffer and gDifference, are leaking memory. After the functions
>> finish, they don't release RAM. I've described this behavior in this
>> post<http://stackoverflow.com/questions/7927977/r-memory-leaking-using-rgeosgdifference-gbuffer>and
>> would appreciate if someone (Roger, Colin?) could take a look.
>
> Roman,

My earlier reply may have seemed bleak. I'll add some substance, and 
explain the problem. The GEOS documentation for the C API - comments in 
the header file geos_c.h, says:

"Currently you have to explicitly GEOSGeom_destroy() all GEOSGeom objects 
to avoid memory leaks, and to GEOSFree() all returned char * (unless 
const)."

The current rgeos source may be checked out from:

svn checkout svn://svn.r-forge.r-project.org/svnroot/rgeos

or browsed online. The offending example of gBuffer() calls the C function 
rgeos_buffer in src/rgeos_buffer.c, but this is only one of many affected 
functions:

https://r-forge.r-project.org/scm/viewvc.php/pkg/src/rgeos_buffer.c?view=markup&revision=267&root=rgeos

at the foot of which are calls to GEOSGeom_destroy_r() - the proper 
version of GEOSGeom_destroy() when an application handle is used, the 
non-handle interface is deprecated. The version in SVN has a loop cleaning 
the R allocated vector of GEOSGeometry* objects, which seems to perform 
OK, no crash. If we uncomment any of the direct destroy functions, we 
crash.

Both Colin and I have been loosing teeth and hair (speaking for myself 
anyway!) over this since he completed the GSoC rgeos project in September 
2010. We didn't release rgeos until March 2011 because we hoped that we 
could resolve the issue, but felt on balance that having rgeos with a 
memory leakage was better than not having it.

The challenge is to fresh eyes to try to see why the advice given in the 
documentation doesn't seem to work for us. For me under GDB on 64-bit 
RHEL5 Linux, GCC 4.1.2, it runs OK when the comments are in place, but 
removal of a comment sign, say in //GEOSGeom_destroy_r(GEOShandle, res);:

library(rgeos)
# GEOS runtime version: 3.3.1-CAPI-1.7.1
p1 = readWKT(paste("POLYGON((0 1,0.95 0.31,0.59",
   "-0.81,-0.59 -0.81,-0.95 0.31,0 1))"))
x <- gBuffer(p1,width=-0.2)

leads to:

*** glibc detected *** /home/rsb/topics/R/R2132/lib64/R/bin/exec/R: double 
free or corruption (fasttop): 0x0000000000df9a60 ***
/lib64/libc.so.6[0x3fbec7245f]
/lib64/libc.so.6(cfree+0x4b)[0x3fbec728bb]
/usr/local/lib/libgeos-3.3.1.so(_ZN4geos4geom8GeometryD0Ev+0x38)[0x2aaab516dcd8]
/usr/local/lib/libgeos_c.so.1(GEOSGeom_destroy_r+0x1f)[0x2aaab54758df]
/home/rsb/lib/r_libs/rgeos/libs/rgeos.so(rgeos_buffer+0x1b7)[0x2aaab4e85cf7]
...
(gdb) bt
#0  0x0000003fbec30265 in raise () from /lib64/libc.so.6
#1  0x0000003fbec31d10 in abort () from /lib64/libc.so.6
#2  0x0000003fbec6a99b in __libc_message () from /lib64/libc.so.6
#3  0x0000003fbec7245f in _int_free () from /lib64/libc.so.6
#4  0x0000003fbec728bb in free () from /lib64/libc.so.6
#5  0x00002aaab516dcd8 in ~auto_ptr (this=0x1183668,
     __in_chrg=<value optimized out>) at 
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/memory:259
#6  geos::geom::Geometry::~Geometry (this=0x1183668,
     __in_chrg=<value optimized out>) at Geometry.cpp:771
#7  0x00002aaab54758df in GEOSGeom_destroy_r (extHandle=0xc1d060, a=0x1980)
     at geos_ts_c.cpp:2113
#8  0x00002aaab4e85cf7 in rgeos_buffer (env=0x1837018,
     obj=<value optimized out>, byid=0x1, id=0x23ff9a8, width=0x2467648,
     quadsegs=0x2467618, capStyle=0x246ae98, joinStyle=0x246cfa8,
     mitreLimit=0x23ffb58) at rgeos_buffer.c:48

If anyone has insight into why this may be happening, and can show how we 
can follow the GEOS C API documentation, it would be extremely useful. 
Probably the answer is obvious, but fresh eyes are needed!

Roger

>
> Please understand that if you choose to post on stackoverflow, you fork the 
> information flow very seriously. Forcing us to go there to see what is 
> happening is not OK for me. Usually, blogs and stackoverflow are overflowing 
> with non-authoritative advice (which authors have not investigated adequately 
> before publishing), which is often simply wrong.
>
> Forking open source communities is a very serious choice, please don't. If 
> you need to document a query, put it somewhere where follow-up comes to the 
> list posting thread in which you link to the external document.
>
> You omitted to report your GEOS version. I am running GEOS runtime version: 
> 3.3.1-CAPI-1.7.1 on RHEL5 x64, and don't see abnormal behaviour, but also see 
> that memory allocated in your example is freed only when R exits.
>
> The R memory manager behaves differently between platforms, and is much less 
> forgiving on Windows, because the underlying technology differs from OSX and 
> Linux/Unix. I guess Windows users are running the CRAN rgeos/GEOS binary, but 
> I am not sure what its version is - GEOS memory leaks get addressed in new 
> releases, and are present, hard to fix, and generally not prioritised by the 
> GEOS team - there are more pressing problems.
>
> In reply to one SO comment, yes, we have run valgrind, but are not in a 
> position to resolve issues that may be in GEOS. Where we have been able to 
> see that the issue is in rgeos, it has been dealt with, but it should be 
> understood that GEOS is not simple, is C++, and often does things which seem 
> odd when using the recommended C API.
>
> In general, any memory allocated in rgeos uses the R memory allocation 
> system, so is potentially seen by the garbage collector, as this shows:
>
> $ grep alloc src/*.[ch]
>
> It may be that the objects are seen as still in use, or that we have not been 
> sufficiently dilligent in destroying C++ objects in GEOS. I would also be 
> hesitant to say that only gBuffer() and gDifference() are implicated here, 
> because the underlying mechanisms are similar for most functions - as you 
> would see if you had looked at the C source code.
>
> So rather than expressing surprise, please do read and audit the source 
> carefully, and once you have definitely found the cause, please offer a 
> patch. Examples just at the R level really don't show us anything, I'm 
> afraid. Perhaps look at the use of the GEOS C API in PostGIS for hints.
>
> Hope this clarifies,
>
> Roger
>
>> 
>> Cheers,
>> Roman
>> 
>> 
>> 
>> 
>
>

-- 
Roger Bivand
Department of Economics, NHH Norwegian School of Economics,
Helleveien 30, N-5045 Bergen, Norway.
voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no