[Rd] Parsing and deparsing of escaped unicode characters

[Jeroen Ooms]

In both R and JSON (and many other languages), unicode characters can
be escaped using a backslash followed by a lowercase "u" and a 4 digit
hex code. However when deparsing a character vector in R on Windows,
the non-latin characters get escaped as "<U+" followed by their 4
digit hex code and ">":

> x <- "I like \u5BFF\u53F8"
> cat(x)
I like 寿司
> src <- deparse(x)
> cat(src)
"I like <U+5BFF><U+53F8>"

Same thing happens on linux when we disable UTF8:

Sys.setlocale("LC_ALL", "C")
x <- "I like \u5BFF\u53F8"
nchar(x) #9, seems OK
cat(deparse(x))
"I like <U+5BFF><U+53F8>"

As a result, the code does not parse() back into the proper unicode
characters. I am currently using a regular expression to convert the
output of deparse into something that parse() (and json) supports:

utf8conv <- function(x) {
  gsub("<U\\+([0-9A-F]{4})>","\\\\u\\1",x)
}

> src <- utf8conv(src)
> y <- parse(text=src)[[1]]
> identical(x, y)
[1] TRUE

However this is suboptimal because it introduces a big performance
overhead for large text. Several things are unclear to me:

 - Why does deparse() use a different escape notation than parse? Is
there a way to make deparse output \uXXXX for unicode instead?
 - Why does deparse on windows escape this in the first place, and not
keep the actual character when the locale supports it?

 > sessionInfo()
R version 3.1.1 (2014-07-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base