caught segfault *** 'memory not mapped` -An irrecoverable exception occurred. R is aborting now ...

[zackarno]

I have GHA running on ubuntu that returns this error:

*** caught segfault ***
address 0x1f1000160, cause 'memory not mapped'

Traceback:
 1: curl_fetch_memory(smtp_server, handle = h)
 2: curl::send_mail(mail_from = unname(from), mail_rcpt = unname(c(to,     cc, bcc)), message = email_qp, smtp_server = paste0(credentials$host,     ":", credentials$port), use_ssl = credentials$use_ssl %||%     TRUE, verbose = verbose, username = credentials$user, password = credentials$password,     login_options = login_options, ...)
 3: smtp_send(email = knitted_email, from = from_email,     to = df_email_receps$email[1], subject = ifelse(EMAIL_LIST !=         "full_list", paste0("TEST: ", email_txt$subj), email_txt$subj),     credentials = email_creds)
An irrecoverable exception occurred. R is aborting now ...
/home/runner/work/_temp/b0e91a74-b79a-4a57-8259-5[52](https://github.com/OCHA-DAP/ds-aa-lac-dry-corridor/actions/runs/13321702421/job/37207437114#step:10:53)0b923665c.sh: line 1:  8041 Segmentation fault      (core dumped) Rscript ./src/monitoring_2025/update_activation_status.R

Oddly it runs fine locally. Additionally I can get the GHA to run successfully when i toggle one of the date filters for the data which seems highly unrelated, but below you can see two runs and there different results. One was successful, whereas the other produced the above error. Locally I can set the RUN_DATE_USE env var either way and the email sends just fine.

run that worked

and the parameters set as:

run that failed

and the screenshot below showing the only change that was made to the env var (RUN_DATE_USE) set as workflow parameter (no change to code)

to make the issue even more confusing I can get the GHA runner to send the email successfully w/ the offending RUN_DATE_USE parameter (that causes failure only on runner) , if I turn off our in house ggtheme package.

issue seems like it could be relate to the below, but hard to say

#272
#270

My colleague is getting the error described in #19 when running locally as well - I've also gotten this, but it seems to have gone away locally, but perhaps something similar manifesting in the runner.

[meztez]

@zackarno Does the email have an attachment per chance? It seems the connect workdir is destroyed before curl had a chance to attach the email.

[meztez]

We've rebuild curl from source and forced the reinstall of libcurl4 and libcurl4-openssl-dev, seems to have fixed it.

[meztez]

And it's back to failing

[zackarno]

Hey thanks for the reply & update

• the email has no attachments.
• The runner is installing libcurl4-openssl-dev

Were you initially suggesting installing the latest version of {blastula}? I think the runner should have been picking that up anyways

[zackarno]

I think the runner should have been picking that up anyways

oh perhaps it wouldn't have picked it up as it was installing from CRAN

update - I installed the latest dev {blastula} directly from GH on my local and the emails have been sending without the abort R session bomb which was sometimes how the problem was manifesting. Will post back when I run it again through GHA with the latest dev blastula version installed there.

[Kjir]

I also have this issue, it happens in one of my containers, but NOT in another different container with a slightly different image.

The relevant part of the Dockerfile for the image that is NOT working:

FROM docker.io/rocker/r-ver:4.5.0

# libssl-dev must be installed first to work in older R versions. It might get
# downgraded to the correct version with the follow-up command.
RUN apt-get update \
	&& apt-get install -y libssl-dev \
	&& apt-get install -y \
	libpq-dev \
	libcurl4-openssl-dev \
	libxml2-dev \
	libssh-dev \
	libbz2-dev \
	liblzma-dev \
	zlib1g-dev \
	libpng-dev \
	libfontconfig1-dev \ 
	libcairo2-dev \
	libharfbuzz-dev \
	libfribidi-dev \
	libfreetype6-dev \
	libpng-dev \
	libtiff5-dev \
	libjpeg-dev \
	libsodium-dev \
	default-jdk \
	libgdal-dev \
	cmake \
	git \
	pandoc \
	&& rm -rf /var/lib/apt/lists/*

RUN mkdir /tasks

# RUN install2.r devtools
RUN install2.r renv --repos https://packagemanager.posit.co/cran/latest

# rdata/renv must be mounted to /renv in container for renv cache, path prefix for OS folder in renv cache
ENV RENV_PATHS_PREFIX_AUTO=TRUE
ENV RENV_PATHS_CACHE=/renv/cache

The image that works is instead like this:

FROM docker.io/rocker/verse:4.5.0

# Auto-accept ttf-mscorefonts-installer EULA license terms
RUN echo ttf-mscorefonts-installer msttcorefonts/accepted-mscorefonts-eula select true | sudo debconf-set-selections

RUN apt-get update && apt-get install -y \
	libpq-dev \
	libcurl4-openssl-dev \
	libxml2-dev \
	libssl-dev \
	libssh-dev \
	libbz2-dev \
	liblzma-dev \
	zlib1g-dev \
	libpng-dev \
	openssh-server \
	git \
	cmake \
	libfontconfig1-dev \ 
	libcairo2-dev \
	libharfbuzz-dev \
	libfribidi-dev \
	libfreetype6-dev \
	libtiff5-dev \
	libjpeg-dev \
	fonts-liberation \
	default-jdk \
	r-cran-rjava \
	libgeos-dev \
  	ttf-mscorefonts-installer \
	&& rm -rf /var/lib/apt/lists/*

RUN echo 'options(repos = c(RSPM = "https://packagemanager.posit.co/cran/latest", CRAN = "https://stat.ethz.ch/CRAN", KCRAN = "https://<redacted>"))' >>"${R_HOME}/etc/Rprofile.site"
RUN install2.r renv --repos https://packagemanager.posit.co/cran/latest
RUN tlmgr update --self && tlmgr update --all && tlmgr install psnfss xcolor framed booktabs geometry caption colortbl float fancyhdr babel-french babel-english babel-italian babel-german pgf pdfpages enumitem pdflscape epstopdf-pkg hyperref etoolbox

# rdata/renv must be mounted to /renv in container for renv cache, path prefix for OS folder in renv cache
ENV RENV_PATHS_PREFIX_AUTO=TRUE
ENV RENV_AUTOLOADER_ENABLED=FALSE
ENV RENV_PATHS_CACHE=/renv/cache

There is another difference: the image that doesn't work is run as root, the one that does is run with an unpriviledged user.

[Kjir]

Here is how the working e-mail is sent:

readable_time <- blastula::add_readable_time()
email <- blastula::compose_email(
  body = blastula::md(
    glue::glue(
      "
# Nowcasting Lab Report: {vintage_date}

Please find the daily report attached to this e-mail.
"
    )
  ),
  footer = blastula::md(glue::glue("Report generated on {readable_time}"))
)

creds <- blastula::creds_envvar(
  user = Sys.getenv("SMTP_USER"),
  host = Sys.getenv("SMTP_HOST"),
  port = Sys.getenv("SMTP_PORT")
)

recipients <- jsonlite::fromJSON(Sys.getenv("REPORT_RECIPIENTS"))
if (is.list(recipients)) {
  recipients <- unlist(recipients)
}
report_file <- file.path(output_path, paste0(report_name, ".pdf"))
email |>
  blastula::add_attachment(report_file) |>
  blastula::smtp_send(
    from = Sys.getenv("SMTP_FROM"),
    to = recipients,
    subject = glue::glue("Nowcasting Lab Report - {vintage_date}"),
    credentials = creds
  )

This is how the non-working e-mail is sent:

send_email_recap <- function(problematic_series) {
  .data <- rlang::.data
  body <- "
# Macrobond Problematic Series

The following projects have problematic series. The series were corrected in the following way:
  * Discontinued series have been deactivated and will not be synchronized anymore.
  * Non-feedable series have been marked as such. Synchronization should resume.
  * Other cases are currently not handled.

The project leads have to decide how to replace the discontinued series.
  "

  rows <- problematic_series |>
    dplyr::group_by(.data$project) |>
    dplyr::arrange(.data$name, .by_group = TRUE) |>
    dplyr::summarise(
      html = glue::glue(
        "<td>{name}</td>",
        "<td>{status}</td>",
        "<td>{replacements}</td>",
        "<td>{comment}</td>\n"
      ) |> glue::glue_collapse("</tr><tr>")
    )
  tables <- rows |> format_project_table()
  body <- paste(body, tables)
  message <- blastula::compose_email(
    body = blastula::md(body)
  )
  blastula::smtp_send(
    message,
    from = "<email-1>",
    to = "<email-2>",
    subject = "Macrobond Series fixed",
    credentials = blastula::creds_envvar(
      user = Sys.getenv("SMTP_USER"),
      host = Sys.getenv("SMTP_HOST"),
      port = Sys.getenv("SMTP_PORT")
    )
  )
}

format_project_table <- function(rows) {
  rows |>
    dplyr::mutate(
      html = glue::glue(
        "## Project {project}\n",
        "<table>",
        "<tr>",
        "<th>Key</th><th>Status</th><th>Replacements</th><th>Comment</th>\n",
        "</tr>",
        "<tr>{html}</tr>",
        "</table>\n\n"
      )
    ) |>
    dplyr::pull("html") |>
    glue::glue_collapse("\n")
}

[Kjir]

I tried using the "working" image for the other task, and it also fails with the same issue. It more likely has to do with the contents of the e-mail. The failing version doesn't have an attachment.

[Kjir]

Since the e-mail code has not changed in 2 years, I highly suspect the bug to be upstream in curl.

[tylerlittlefield]

I am also facing this on a bunch of my containers which send emails via blastula.

[ak-nge]

Same issue here - code works fine locally, consistently gives segfault in a container, whether with blastula v0.3.6 or v0.3.3 (that's the ones I tested explicitly). It was somewhat comforting to find this issue and realize I'm not alone. 😅
In the afflicted jobs, it does not make a difference whether the e-mail contains any HTML formatting, or if it is just a minimal scaffolding from blastula::compose_email() with nothing but ASCII characters in the respective fields. Once it is supposed to send the job always, and consistently, segfaults. So to me this also seems like the bug could be with curl, not {blastula}.

[DeepanshKhurana]

Going one version previous for libcurl did solve this for me. So, I can confirm curl has an issue. The following may help you out. Note that it takes one version previous so it is not super robust in case libcurl gets updated.

RUN set -eux; \
    apt-get update; \
    prev_ver="$(apt-cache madison libcurl4-openssl-dev | awk 'NR==2{print $3}')" ; \
    echo "Installing previous libcurl: ${prev_ver}"; \
    apt-get install -y --no-install-recommends \
      libcurl4-openssl-dev="${prev_ver}" \
      libcurl4="${prev_ver}" \
      curl="${prev_ver}" \
      libssl-dev \
      libsasl2-dev \
      libsasl2-modules \
      libxml2-dev \
      zlib1g-dev \
      ca-certificates \
      libpq-dev \
      pkg-config \
      cmake \
      libx11-dev \
      pandoc \
      git \
      nano \
    && rm -rf /var/lib/apt/lists/*

[tylerlittlefield]

In my case, I'm sending emails in a future:

• user submits a request
• we return 200 to signal event was triggered
• long running task executes in future, future::plan(future::multisession)
• we send email via blastula::smtp_send when long running task completes

I fixed by identifying the exact image used to build production, in this case it was:

FROM rstudio/plumber@sha256:18372b3e13ae369888c0bde7bbd5e7c3089224834cc3442ee3664f11e0ae8116

Once I switched to that, problem solved. It looks like this image gets its tags overwritten, so upstream environment changed since I last built.

Sharing my environment that's working. I hope to one day figure out a root cause. For now, I might avoid blastula::smtp_send in a future, I feel that's part of the issue in my situation.

Environment

cat /etc/os-release
#> PRETTY_NAME="Ubuntu 22.04.4 LTS"
#> NAME="Ubuntu"
#> VERSION_ID="22.04"
#> VERSION="22.04.4 LTS (Jammy Jellyfish)"
#> VERSION_CODENAME=jammy
#> ID=ubuntu
#> ID_LIKE=debian
#> HOME_URL="https://www.ubuntu.com/"
#> SUPPORT_URL="https://help.ubuntu.com/"
#> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
#> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
#> UBUNTU_CODENAME=jammy

R --version
#> R version 4.4.1 (2024-06-14) -- "Race for Your Life"
#> Copyright (C) 2024 The R Foundation for Statistical Computing
#> Platform: x86_64-pc-linux-gnu
#> 
#> R is free software and comes with ABSOLUTELY NO WARRANTY.
#> You are welcome to redistribute it under the terms of the
#> GNU General Public License versions 2 or 3.
#> For more information about these matters see
#> https://www.gnu.org/licenses/.

curl --version
#> curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.#> 3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.18
#> Release-Date: 2022-01-05
#> Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet #> tftp
#> Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP #> UnixSockets zstd

R -e 'lapply(c("blastula", "curl"), packageVersion)' --silent
#> > lapply(c("blastula", "curl"), packageVersion)
#> [[1]]
#> [1] ‘0.3.4’
#> 
#> [[2]]
#> [1] ‘5.2.2’