xxmissingnoxx commented

(Migrated from github.com)

Thank you for sharing this project with the Emacs community.

I stumbled across an error "Error running timer 'org-indent-initialize-agent'" while importing an rmarkdown file into the org-mode format. A screenshot is attached below. Is this something you've encountered before? I'm currently using Doom emacs.

Thank you for sharing this project with the Emacs community. I stumbled across an error "Error running timer 'org-indent-initialize-agent'" while importing an rmarkdown file into the org-mode format. A screenshot is attached below. Is this something you've encountered before? I'm currently using Doom emacs. <img width="1792" alt="Screen Shot 2021-01-24 at 9 19 13 PM" src="https://user-images.githubusercontent.com/17442967/105654654-78cc0100-5e8c-11eb-9dfa-4eb2a1093a8d.png">

tecosaur commented

2021-01-25 02:44:22 +00:00

(Migrated from github.com)

Off the bat, I have no idea, and I can't say I hope to reproduce this. Is this occuring with a particular file?

tecosaur commented

2021-01-27 04:14:33 +00:00

(Migrated from github.com)

@xxmissingnoxx you're going to have to give me something to work with :P

xxmissingnoxx commented

2021-01-27 06:42:42 +00:00

(Migrated from github.com)

I apologize for going dark there. I've been trying to create a minimal example which triggers the behavior. I've seen this behavior on 3 out of 3 files which are actual analyses so far. I tend to use the same format for my r notebook analyses that may not be too surprising if it has something to do with the content of the file.

I created a small example:

Testing

library(data.table)
library(ggplot2)
library(rstanarm)

head(mtcars)

This yields:

Testing
=n{r import} library(data.table) library(ggplot2) library(rstanarm)=

=n{r head} head(mtcars)=

This does not trigger the error but similar behavior seems to appear in the more complicated files. The appearance of the letter "n" happens regularly. You should probably look at this post in the raw form as Github is rendering the rmd content. This is how it looks in the editor:

I used the command "org-pandoc-import-rmarkdown-as-org".

I apologize for going dark there. I've been trying to create a minimal example which triggers the behavior. I've seen this behavior on 3 out of 3 files which are actual analyses so far. I tend to use the same format for my r notebook analyses that may not be too surprising if it has something to do with the content of the file. I created a small example: # Testing ```{r import} library(data.table) library(ggplot2) library(rstanarm) ``` ```{r head} head(mtcars) ``` This yields: * Testing =n{r import} library(data.table) library(ggplot2) library(rstanarm)= =n{r head} head(mtcars)= This does not trigger the error but similar behavior seems to appear in the more complicated files. The appearance of the letter "n" happens regularly. You should probably look at this post in the raw form as Github is rendering the rmd content. This is how it looks in the editor: <img width="600" alt="Screen Shot 2021-01-27 at 1 45 34 AM" src="https://user-images.githubusercontent.com/17442967/105953633-7bbc2280-6041-11eb-92db-912cb3b160c4.png"> <img width="165" alt="Screen Shot 2021-01-27 at 1 45 52 AM" src="https://user-images.githubusercontent.com/17442967/105953649-824a9a00-6041-11eb-8dfc-1113d22887c4.png"> I used the command "org-pandoc-import-rmarkdown-as-org".

tecosaur commented

2021-01-27 06:54:09 +00:00

(Migrated from github.com)

Thanks 😃. By the way, what version of Org are you running?

Thanks :smiley:. By the way, what version of Org are you running?

xxmissingnoxx commented

2021-01-27 06:56:01 +00:00

(Migrated from github.com)

I realized I could try this without the named code blocks. I got different results. Assuming the code blocks have no name, should babel code blocks be the expected result or the example blocks generated below? I'm not sure if this is another clue or intended. It looks like Org 9.5 when I use the org-version command. I'm still getting acclimated to Emacs and Org stuff.

I realized I could try this without the named code blocks. I got different results. Assuming the code blocks have no name, should babel code blocks be the expected result or the example blocks generated below? I'm not sure if this is another clue or intended. It looks like Org 9.5 when I use the org-version command. I'm still getting acclimated to Emacs and Org stuff. <img width="176" alt="Screen Shot 2021-01-27 at 1 53 43 AM" src="https://user-images.githubusercontent.com/17442967/105954374-95119e80-6042-11eb-92b0-87daf52fe458.png"> <img width="155" alt="Screen Shot 2021-01-27 at 1 53 33 AM" src="https://user-images.githubusercontent.com/17442967/105954393-98a52580-6042-11eb-9f43-a24367800fb4.png">

xxmissingnoxx commented

2021-02-07 06:48:30 +00:00

(Migrated from github.com)

I've been tinkering with this off and on. It looks as though there are multiple factors in play.

The problem lies with your rmarkdown.lua filter. This problem occurs without any org or emacs in the mix. You can see this if you try to call pandoc on the command line with the filter. Rmarkdown is not compliant with any form of markdown supported by pandoc and thus is not appropriately handled by the AST and the lua filter. Rmarkdown is not a form of markdown that pandoc understands... but pandoc markdown is.

I think you can use an approximation of your existing code if you take a different approach.

First, use

library(rmarkdown)
render("test.rmd",md_document(variant="markdown"))

to convert the rmarkdown to the pandoc flavor of markdown which is equipped to handle both backticks and fenced code attributes and classes as mentioned here,
2. Now that you have pandoc markdown, you can take advantage of the elem.attr and elem.classes attributes and get rid of (some) your regular expressions. Update your filter accordingly.
2. Apply your filter when converting from pandoc markdown to org.

I have a basic version of this working. The current problem with this approach is that the current settings mean losing chunk option information (ex. eval=FALSE). I suspect there's a way to get rmarkdown to dump that in the pandoc markdown too but experimentation, or perhaps posting an issue in their repo may be the next step.

I've been tinkering with this off and on. It looks as though there are multiple factors in play. The problem lies with your rmarkdown.lua filter. This problem occurs without any org or emacs in the mix. You can see this if you try to call pandoc on the command line with the filter. Rmarkdown is not compliant with any form of markdown supported by pandoc and thus is not appropriately handled by the AST and the lua filter. Rmarkdown is not a form of markdown that pandoc understands... but pandoc markdown is. I think you can use an approximation of your existing code if you take a different approach. 1. First, use ```r library(rmarkdown) render("test.rmd",md_document(variant="markdown")) ``` to convert the rmarkdown to the pandoc flavor of markdown which is equipped to handle both backticks and fenced code attributes and classes as mentioned [here](https://rmarkdown.rstudio.com/authoring_pandoc_markdown.html#Verbatim_(code)_blocks ), 2. Now that you have pandoc markdown, you can take advantage of the elem.attr and elem.classes attributes and get rid of (some) your regular expressions. Update your filter accordingly. 2. Apply your filter when converting from pandoc markdown to org. I have a basic version of this working. The current problem with this approach is that the current settings mean losing chunk option information (ex. eval=FALSE). I suspect there's a way to get rmarkdown to dump that in the pandoc markdown too but experimentation, or perhaps posting an issue in their repo may be the next step.

tecosaur commented

2021-02-07 15:40:07 +00:00

(Migrated from github.com)

Thanks for following up @xxmissingnoxx. I'm using the following test document, I hope like yours.

However, I can't reproduce your issue.

I'm thinking perhaps it has something to do with the pre-processor? Do you have sed available?

Thanks for following up @xxmissingnoxx. I'm using the following test document, I hope like yours. ![image](https://user-images.githubusercontent.com/20903656/107151359-26ff9c80-699d-11eb-8841-86292828faf3.png) However, I can't reproduce your issue. ![image](https://user-images.githubusercontent.com/20903656/107151397-4eef0000-699d-11eb-9fe8-a620238a1d7e.png) I'm thinking perhaps it has something to do with the pre-processor? Do you have `sed` available?

xxmissingnoxx commented

2021-02-07 18:33:20 +00:00

(Migrated from github.com)

I have sed available. I'm on a Mac so apparently you can't get the version number, but it works well enough to yell at me about it:

sed: illegal option -- -
usage: sed script [-Ealn] [-i extension] [file ...]
       sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]

Pandoc version:

pandoc 2.11.3.2
Compiled with pandoc-types 1.22, texmath 0.12.1, skylighting 0.10.2,
citeproc 0.3.0.3, ipynb 0.1.0.1

Command line call:

pandoc --lua-filter=./org-pandoc-import/filters/rmarkdown.lua -f markdown -t org -o rmd.org test.md

Test file contents:

# Testing


Here is the first block:

```{r}
library(data.table)
library(ggplot2)
library(rstanarm) 
```

```{r .here}
library(data.table)
library(ggplot2)
library(rstanarm) 
```

```{r}
library(patchwork)
head(iris)
```

```{r  eval=FALSE}
library(ggplot2)
library(rstanarm)
library(arm)
library(faraway) 
```

org-pandoc-import-rmarkdown-as-org output:

* Testing
Here is the first block:

#+BEGIN_EXAMPLE
library(data.table)
library(ggplot2)
library(rstanarm) 
#+END_EXAMPLE
=n{r .here} library(data.table) library(ggplot2) library(rstanarm)=

#+BEGIN_EXAMPLE
library(patchwork)
head(iris)
#+END_EXAMPLE
=n{r  eval=FALSE} library(ggplot2) library(rstanarm) library(arm) library(faraway)=

Command line call output:

* Testing
  :PROPERTIES:
  :CUSTOM_ID: testing
  :END:
Here is the first block:

#+BEGIN_EXAMPLE
library(data.table)
library(ggplot2)
library(rstanarm) 
#+END_EXAMPLE

#+BEGIN_EXAMPLE
library(patchwork)
head(iris)
#+END_EXAMPLE

#+BEGIN_EXAMPLE
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          5.1         3.5          1.4         0.2  setosa
## 2          4.9         3.0          1.4         0.2  setosa
## 3          4.7         3.2          1.3         0.2  setosa
## 4          4.6         3.1          1.5         0.2  setosa
## 5          5.0         3.6          1.4         0.2  setosa
## 6          5.4         3.9          1.7         0.4  setosa
#+END_EXAMPLE
={r  eval=FALSE}  library(ggplot2) library(rstanarm) library(arm) library(faraway)=

#+BEGIN_EXAMPLE
library(ggplot2)
library(rstanarm)
library(arm)
library(faraway) 
#+END_EXAMPLE

#+BEGIN_EXAMPLE
library(lme4) 
#+END_EXAMPLE

I have sed available. I'm on a Mac so apparently you [can't](https://stackoverflow.com/questions/37639496/how-can-i-check-the-version-of-sed-in-os-x) get the version number, but it works well enough to yell at me about it: ``` sed: illegal option -- - usage: sed script [-Ealn] [-i extension] [file ...] sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...] ``` Pandoc version: ``` pandoc 2.11.3.2 Compiled with pandoc-types 1.22, texmath 0.12.1, skylighting 0.10.2, citeproc 0.3.0.3, ipynb 0.1.0.1 ``` Command line call: ```bash pandoc --lua-filter=./org-pandoc-import/filters/rmarkdown.lua -f markdown -t org -o rmd.org test.md ``` Test file contents: ```` # Testing Here is the first block: ```{r} library(data.table) library(ggplot2) library(rstanarm) ``` ```{r .here} library(data.table) library(ggplot2) library(rstanarm) ``` ```{r} library(patchwork) head(iris) ``` ```{r eval=FALSE} library(ggplot2) library(rstanarm) library(arm) library(faraway) ``` ```` org-pandoc-import-rmarkdown-as-org output: ```` * Testing Here is the first block: #+BEGIN_EXAMPLE library(data.table) library(ggplot2) library(rstanarm) #+END_EXAMPLE =n{r .here} library(data.table) library(ggplot2) library(rstanarm)= #+BEGIN_EXAMPLE library(patchwork) head(iris) #+END_EXAMPLE =n{r eval=FALSE} library(ggplot2) library(rstanarm) library(arm) library(faraway)= ```` Command line call output: ```` * Testing :PROPERTIES: :CUSTOM_ID: testing :END: Here is the first block: #+BEGIN_EXAMPLE library(data.table) library(ggplot2) library(rstanarm) #+END_EXAMPLE #+BEGIN_EXAMPLE library(patchwork) head(iris) #+END_EXAMPLE #+BEGIN_EXAMPLE ## Sepal.Length Sepal.Width Petal.Length Petal.Width Species ## 1 5.1 3.5 1.4 0.2 setosa ## 2 4.9 3.0 1.4 0.2 setosa ## 3 4.7 3.2 1.3 0.2 setosa ## 4 4.6 3.1 1.5 0.2 setosa ## 5 5.0 3.6 1.4 0.2 setosa ## 6 5.4 3.9 1.7 0.4 setosa #+END_EXAMPLE ={r eval=FALSE} library(ggplot2) library(rstanarm) library(arm) library(faraway)= #+BEGIN_EXAMPLE library(ggplot2) library(rstanarm) library(arm) library(faraway) #+END_EXAMPLE #+BEGIN_EXAMPLE library(lme4) #+END_EXAMPLE ````

tecosaur commented

2021-02-07 18:42:49 +00:00

(Migrated from github.com)

Thanks for the detailed example. I took your input file, ran the preprocessor on it, then ran pandoc with the filter.
Here's what I found:

Left = input, Middle = preprocessed, Right = final

Could you try calling (call-process "sed" nil (list :file "preprocessedfile.Rmd") nil "s/^```{/```\\n{/" "infile.Rmd") and seeing what your output is? It should match the middle column above.

Thanks for your patience.

Thanks for the detailed example. I took your input file, ran the preprocessor on it, then ran pandoc with the filter. Here's what I found: ![image](https://user-images.githubusercontent.com/20903656/107156030-d09f5780-69b6-11eb-9599-9724b8e5d6e5.png) Left = input, Middle = preprocessed, Right = final Could you try calling `(call-process "sed" nil (list :file "preprocessedfile.Rmd") nil "s/^```{/```\\n{/" "infile.Rmd")` and seeing what your output is? It should match the middle column above. Thanks for your patience.

xxmissingnoxx commented

2021-02-07 18:54:57 +00:00

(Migrated from github.com)

Thanks for the quick response. I get:

# Testing


Here is the first block:

```n{r}
library(data.table)
library(ggplot2)
library(rstanarm) 
```

```n{r .here}
library(data.table)
library(ggplot2)
library(rstanarm) 
```

```n{r}
library(patchwork)
head(iris)
```

```n{r  eval=FALSE}
library(ggplot2)
library(rstanarm)
library(arm)
library(faraway) 
```

Screenshots below if that helps.

Thanks for the quick response. I get: ```` # Testing Here is the first block: ```n{r} library(data.table) library(ggplot2) library(rstanarm) ``` ```n{r .here} library(data.table) library(ggplot2) library(rstanarm) ``` ```n{r} library(patchwork) head(iris) ``` ```n{r eval=FALSE} library(ggplot2) library(rstanarm) library(arm) library(faraway) ``` ```` Screenshots below if that helps. <img width="875" alt="Screen Shot 2021-02-07 at 1 53 50 PM" src="https://user-images.githubusercontent.com/17442967/107156374-f0e5fc80-694b-11eb-80cd-8067c8f61be6.png"> <img width="761" alt="Screen Shot 2021-02-07 at 1 52 42 PM" src="https://user-images.githubusercontent.com/17442967/107156353-cf851080-694b-11eb-815f-02ef354b3bae.png">

xxmissingnoxx commented

2021-02-07 19:08:33 +00:00

(Migrated from github.com)

Adding another slash before n seems to get your result. I'm not sure why you'd get different results though.
Screen Shot 2021-02-07 at 2 13 11 PM

Adding another slash before n seems to get your result. I'm not sure why you'd get different results though. <img width="767" alt="Screen Shot 2021-02-07 at 2 13 11 PM" src="https://user-images.githubusercontent.com/17442967/107156836-a5811d80-694e-11eb-8aee-9b88d83c208b.png">

tecosaur commented

2021-02-07 20:00:00 +00:00

(Migrated from github.com)

Interesting. I'm guessing this is a platform-specific difference. Let's see if I can't resolve this over the next few days 🙂

Interesting. I'm guessing this is a platform-specific difference. Let's see if I can't resolve this over the next few days :slightly_smiling_face:

xxmissingnoxx commented

2021-02-07 20:07:25 +00:00

(Migrated from github.com)

That's what I'd guess too.

I feel like you shouldn't need to jump through the regex hoops to do this and there should be a way to do this with the rmarkdown package itself. I asked a question that probably won't be answered on SO to see if there's a nicer way to dump rmarkdown attributes. If that works, then it should be easier to simplify your code, at the cost of the R dependency, and maybe build on it (ex. use chunk label as named code block in org or preserve attributes if you want to export back out to rmarkdown).

Aside: What are you using to conceal the begin_src and end_src blocks. My emacs-fu is weak.

That's what I'd [guess](https://stackoverflow.com/questions/46082397/insert-newline-n-using-sed) too. I feel like you shouldn't need to jump through the regex hoops to do this and there should be a way to do this with the rmarkdown package itself. I [asked](https://stackoverflow.com/questions/66092256/is-it-possible-to-convert-rmarkdown-to-pandoc-markdown-while-preserving-code-chu) a question that probably won't be answered on SO to see if there's a nicer way to dump rmarkdown attributes. If that works, then it should be easier to simplify your code, at the cost of the R dependency, and maybe build on it (ex. use chunk label as named code block in org or preserve attributes if you want to export back out to rmarkdown). Aside: What are you using to conceal the `begin_src` and `end_src` blocks. My emacs-fu is weak.

tecosaur commented

2021-02-07 20:13:01 +00:00

(Migrated from github.com)

I'll return to sed later this week.

On your aside, I use doom and it's a result of a (ligatures +extra) module 🙂

I'll return to `sed` later this week. On your aside, I use doom and it's a result of a `(ligatures +extra)` module :slightly_smiling_face:

org-indent-initialize-agent error when importing rmarkdown document #8

Testing