reactive-mail-server: a self-hosted mail server on Spring WebFlux, deployed as a staging mail trap on Jambotron

Abstract

This article describes reactive-mail-server, a small self-hosted mail server written on Spring Boot WebFlux, R2DBC and Reactor Netty, with an Angular web client, and how we deployed it on Jambotron as a mail trap for our staging environment. Since the switch, every email that staging sends (sign-up confirmations, password resets, notifications) lands in one web mailbox instead of reaching real people. The server receives mail over its own SMTP implementation with STARTTLS, routes it to local mailboxes or a catch-all mailbox, and can deliver outside through a retry queue with DKIM signing. The second half of the article covers what we learned while deploying it, from a truncated shared library inside a Docker layer to a Spring Mail setting that silently re-enables SMTP authentication.

Keywords: SMTP, mail trap, catch-all, STARTTLS, Spring Boot, WebFlux, Reactor Netty, R2DBC, Angular, Docker Compose, staging.

1. Introduction

Every web application sends email: confirmations, password resets, invoices, alerts. On a staging server this becomes a problem. Staging works with copies of real data, so it can easily send a "your order has shipped" message to a real customer. It also spends the quota of your email provider and makes it hard to check what was actually sent.

The usual fix is a mail trap: staging talks to an SMTP server that accepts everything and delivers nothing. Developers read the messages in a web UI.

We wanted a trap that also works as a small but real mail server: separate mailboxes, a web client with folders and attachments, an SMTP server that does not become an open relay, and TLS. We also wanted to test our own hosting platform, Jambotron, on something less trivial than a static site. reactive-mail-server is the result. In short, it is a project that:

  • deploys from one Docker Compose file, with no secrets in git;
  • accepts SMTP and stores mail in PostgreSQL;
  • keeps real mailboxes working while trapping everything else;
  • shows the result in a browser.

2. Existing solutions

If all you need is to look at the emails your app sends, use Mailpit. It is one container and has an excellent UI. We ran axllent/mailpit:v1.27 (version 1.27.11) while writing this article: it accepted a test message on port 1025 and showed it in the UI on port 8025 within a second.

| Solution | Type | Real mailboxes | Delivery outside | TLS for SMTP | Notes | |---|---|---|---|---|---| | Mailpit | Mail trap | No (one inbox) | Optional relay | Yes | The best choice for a pure trap | | MailHog | Mail trap | No | Optional relay | No | No releases since 2020 | | GreenMail | Test library / server | Yes | No | Yes | Great inside JUnit tests (we use it there) | | Postfix + Dovecot | Full MTA + IMAP | Yes | Yes | Yes | Production-grade, a lot of configuration | | Mailu, Mailcow | Full mail suites | Yes | Yes | Yes | Many containers, built for real domains | | reactive-mail-server | Small mail server | Yes | Yes (queue, DKIM) | Yes (STARTTLS, 465) | Trap + real mailboxes, one Java service |

reactive-mail-server sits between a trap and a full mail suite. Mail for an existing mailbox goes into that mailbox. Mail for any other address, including someone@gmail.com, goes into the catch-all mailbox. Nothing leaves the server unless you turn outbound delivery on.

3. Architecture

3.1 Overview

 SMTP client / other server ──► SmtpServer (Reactor Netty, :3025, STARTTLS) ──┐
                                                                              ├─► MailRouter ─┬─► LocalDelivery → mailbox (PostgreSQL)
 Web client (Angular) ──► REST /api ──► SubmissionService ────────────────────┘               └─► OutboundQueue → DeliveryWorker → recipient's SMTP

| Part | Technology | |---|---| | Backend | Java 21, Spring Boot 3.4 WebFlux, R2DBC (PostgreSQL), Flyway | | SMTP server | Reactor Netty TCP server, own protocol implementation | | Mail parsing and building | Jakarta Mail (Angus Mail) | | DKIM / SPF / DMARC | utils-mail-dkim (signing), Apache James jDKIM (verification), dnsjava | | Web client | Angular 22, Tailwind CSS 4, @jambotron/aria-grid for all tables | | Tests | JUnit 5, GreenMail, real JavaMail clients against the server |

The code is about 5,800 lines of Java and 2,400 lines of TypeScript. The backend has 38 tests.

3.2 One routing rule for SMTP and the web client

MailRouter decides where every recipient goes, and both entry points use it: the SMTP server and the "Send" button in the web client.

  1. An existing mailbox in one of our domains: store the message there.
  2. A catch-all mailbox is configured: store everything else there. Nothing is delivered outside.
  3. Our domain, but no such mailbox: 550 5.1.1 No such user here.
  4. Someone else's domain: the delivery queue, but only for a logged-in sender. Without authentication the answer is 550 5.7.1 Relaying denied, so the server is never an open relay.

A logged-in user may send only from their own address (553 otherwise).

4. The SMTP server

4.1 Lines, not packets

The first prototype treated each network packet as one command. That breaks as soon as a client pipelines several commands into one packet, or a large message arrives split across packets. The server now splits input into lines with Netty's LineBasedFrameDecoder. A test sends EHLO, MAIL FROM, RCPT TO and DATA in one packet, then splits a line of the message body across two packets. The server handles both cases.

4.2 Accept only what is stored

A message is written to the database (or the outbound queue) before the client gets 250 Message accepted. If storing fails, the client gets 451 4.3.0 Message not stored, try again later, and a well-behaved client retries. Database and DNS work runs on a worker thread, never on the Netty event loop.

4.3 STARTTLS without command injection

The server supports STARTTLS (RFC 3207) on the main port and implicit TLS (RFC 8314, usually port 465) on an optional second port. AUTH is offered only inside TLS, unless a flag allows it in a closed network.

Getting STARTTLS right takes two details:

  1. Order of writes. The TLS handler may be added only after 220 Ready to start TLS has left in plain text. Replies are written one at a time, and the TLS handler is attached to that particular reply. That way pipelined EHLO + STARTTLS cannot start TLS too early.
  2. Plain text that arrived together with STARTTLS. An attacker in the middle can append commands after STARTTLS in the same packet. A naive server runs them inside the encrypted session later (this class of bug is CVE-2011-0411). Every decoded line carries a flag that says whether it came through TLS. A plain-text line after STARTTLS closes the connection. The line decoder is also replaced with a fresh one, so leftover plain-text bytes cannot mix with decrypted data.

After STARTTLS the session starts over: the client must say EHLO again, and an earlier MAIL FROM is forgotten.

The certificate is either your own (PEM certificate and PKCS#8 key, for example from Let's Encrypt) or a self-signed one. The self-signed certificate is created once in the data volume and reused after redeploys. Live check on our staging stand:

$ openssl s_client -starttls smtp -connect 10.30.1.77:20001
subject=CN = mail.test
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256

4.4 Delivery outside

Outbound delivery is off by default. When it is on, mail goes either directly to the recipient's MX on port 25 (with STARTTLS when offered) or through a smarthost such as Resend or SES. The queue lives in a table. A temporary failure is retried after 1 min, 5 min, 15 min, 1 h, 3 h, 6 h and 12 h. A permanent failure, or the last retry, sends a bounce to the sender's inbox. Outgoing mail is DKIM-signed (rsa-sha256, relaxed/relaxed). Incoming mail is checked with DKIM, SPF and DMARC.

5. The web client

The web client is an Angular 22 single-page app built into the same JAR, in a "synthwave" style: glass panels, neon accents, an animated grid that turns off with prefers-reduced-motion. It has four colour palettes and a high-contrast mode.

  • Sign-up (if enabled) and sign-in. Service names such as postmaster and abuse are reserved.
  • Folders: Inbox, Drafts, Sent, Spam, Trash, plus your own folders.
  • Search; flag; read/unread; move between folders; empty Trash and Spam.
  • HTML messages render in a sandboxed iframe without scripts. External images load only on request, so the sender cannot tell that the message was opened.
  • Compose: several recipients, CC, contact suggestions, reply, reply all, forward. Drafts save themselves every 15 seconds, and attachments up to 10 MB go into the draft.
  • Contacts, blocked senders, settings (name, password, storage, appearance).

6. Deployment on Jambotron

6.1 One Compose file, no secrets in git

The repository has a separate docker-compose.jambotron.yml for deployment (shortened):

services:
  secrets-init:            # generates DB_PASSWORD and JWT_SECRET once, into a volume
    image: alpine:3.20
    restart: "no"
    volumes: [secrets:/secrets]

  db:
    image: postgres:17
    environment:
      POSTGRES_PASSWORD_FILE: /run/mail-secrets/DB_PASSWORD
    volumes: [pgdata:/var/lib/postgresql/data, secrets:/run/mail-secrets:ro]

  mail:
    build: { context: ., dockerfile: backend/Dockerfile }
    expose: ["8080", "3025"]          # web client + API; SMTP stays inside
    environment:
      MAIL_DOMAINS: ${MAIL_DOMAINS:-mail.test}
      MAIL_CATCH_ALL: ${MAIL_CATCH_ALL:-trap@mail.test}
      OUTBOUND_MODE: "off"            # nothing ever leaves
    volumes: [data:/app/data, secrets:/run/mail-secrets:ro]

On the first start the server creates admin@mail.test and trap@mail.test and writes their random passwords to a file with mode 600 in the data volume. Nothing has to be typed in, and nothing secret is in the repository.

6.2 Steps

  1. Create a project in Jambotron and upload the repository as a ZIP.
  2. In the deploy wizard, pick docker-compose.jambotron.yml and deploy. Jambotron builds the image on the VM and starts the stack.
  3. Give the mail service a domain, port 8080. The web client is then on HTTPS with a certificate issued automatically.
  4. For SMTP, open an internal TCP port: service mail, port 3025. Jambotron shows an address like 10.30.1.77:20001. It is reachable only inside Jambotron's private WireGuard network and not from the internet, so the trap cannot be abused from outside.

6.3 Pointing staging at the trap

Our staging is a Spring Boot application, so the switch is a few lines in the staging profile:

spring:
  mail:
    host: 10.30.1.77
    port: 20001
    username: ""
    password: ""          # important, see 7.4
    properties:
      mail.smtp.auth: false
      mail.smtp.starttls.enable: false

The real check was the "Forgot password" form on staging: the reset email appeared in trap@mail.test, and nothing reached Gmail.

7. Lessons from the deployment

Deploying the prototype found more problems than the code review did. Most of them would hit any project.

7.1 "Bus error" from a truncated library inside a Docker layer

On one VM, every program from Alpine 3.24.3 images that loads OpenSSL died with Bus error: PostgreSQL, socat, even --version. We first blamed the kernel and musl. Both guesses were wrong. A fresh clone of the same VM template with the same image IDs ran everything fine. On the broken VM, /usr/lib/libcrypto.so.3 in the shared base layer was exactly 4,194,304 bytes (4 MiB) instead of 4,993,808. Mapping past the end of a truncated file is exactly what causes SIGBUS. Removing the images and pulling them again fixed it. Lesson: if one image is broken on one host only, compare file sizes inside the layer before you suspect the kernel.

7.2 403 on login behind a reverse proxy

Sign-in worked with curl and failed in the browser with 403. Behind Jambotron's Caddy the browser sent Origin: https://…, while the application saw plain HTTP and treated the request as cross-origin. The fix was one line: server.forward-headers-strategy=framework. Our earlier test did not catch it because curl sends no Origin header. Test logins with -H "Origin: …".

7.3 The browser's Basic-auth popup

On a 401, Spring Security sent WWW-Authenticate: Basic, and browsers showed their native login dialog on top of the app. The API now answers 401 without that header, and the client redirects to its own login page.

7.4 An empty username does not turn SMTP AUTH off

After the switch, staging emails still failed with Authentication failed. We had set username: "" and mail.smtp.auth: false, but the old provider password was still in the environment. Jakarta Mail then fills in the system user name (user.name) and authenticates anyway. Both username and password must be empty. We confirmed it with the same JavaMailSender and settings against the trap: with the password set, Authentication failed; with it empty, the message arrived.

7.5 Improvements to Jambotron that came out of this project

  • A new ZIP refreshes the saved Compose file. Jambotron keeps the Compose file chosen in the wizard and writes it to the VM on deploy. Uploading a new ZIP did not update it, so a deploy could run an old Compose file. Now the saved file is refreshed from the new ZIP. If you edited it in the wizard, your version is kept and the UI warns you that the ZIP has a different one.
  • Internal TCP ports. SMTP is not HTTP and does not belong behind a public domain. Projects can now open a container port inside the private network only (section 6.2).

8. Testing

  • 38 backend tests. SMTP is tested with real clients: plain sockets for pipelining and split lines; JavaMail with starttls.required and with implicit TLS; injected plain text after STARTTLS; relay attempts without authentication; storage failure (451); oversized messages (552). Outbound delivery is tested against GreenMail; the retry queue and DKIM signatures have their own unit tests.
  • Live checks on staging: messages between two mailboxes; attachments; mail from the staging application into the trap; STARTTLS from the private network.

The unedited SMTP session from the staging application's container to the trap:

220 mail.test ESMTP ready
250-mail.test Hello staging
250-SIZE 26214400
250-8BITMIME
250-PIPELINING
250-AUTH PLAIN LOGIN
250 ENHANCEDSTATUSCODES
250 2.1.0 Sender OK
250 2.1.5 Recipient OK
354 End data with <CR><LF>.<CR><LF>
250 2.0.0 Message accepted
221 2.0.0 mail.test Bye

9. Limitations and next steps

  • IMAP is turned off. The prototype version blocked the event loop and needs a rewrite before Thunderbird or a phone can use the server. TLS for IMAP comes with that rewrite.
  • Brute-force protection for sign-in and sign-up (rate limits, captcha) is not there yet.
  • Real mail for a real domain needs its own small server: an IP address with a clean reputation, port 25 open, PTR record, Let's Encrypt certificate. It should not share an IP address with anything else.
  • Forwarding does not carry the original attachments yet. Inline cid: images show as attachments.

10. Conclusion

A mail trap does not have to be a mail server. For most teams Mailpit is the right answer. Building our own was still worth it. We now have a trap that keeps real mailboxes working, speaks STARTTLS correctly, and lives in the same stack as the rest of our services. Deploying it as an ordinary project turned up several real problems in a day, and two of them became features of Jambotron.

References

  1. RFC 5321 — Simple Mail Transfer Protocol. — https://www.rfc-editor.org/rfc/rfc5321
  2. RFC 3207 — SMTP Service Extension for Secure SMTP over TLS. — https://www.rfc-editor.org/rfc/rfc3207
  3. RFC 8314 — Cleartext Considered Obsolete: Use of TLS for Email Submission and Access. — https://www.rfc-editor.org/rfc/rfc8314
  4. RFC 6376 — DomainKeys Identified Mail (DKIM) Signatures. — https://www.rfc-editor.org/rfc/rfc6376
  5. CVE-2011-0411 — STARTTLS plaintext command injection. — https://nvd.nist.gov/vuln/detail/CVE-2011-0411
  6. Reactor Netty Reference Guide. — https://projectreactor.io/docs/netty/release/reference/
  7. Spring WebFlux. — https://docs.spring.io/spring-framework/reference/web/webflux.html
  8. Mailpit. — https://mailpit.axllent.org/
  9. GreenMail. — https://greenmail-mail-test.github.io/greenmail/

11. Source code and demo

Live demo

A test stand runs on Jambotron. Sign-up is open, and nothing is ever sent outside:

https://reactive-mail.staging.jambocheck.com/

Repository

https://gitea.jambotron.com/jambotron_gang/reactive-mail-server

git clone https://gitea.jambotron.com/jambotron_gang/reactive-mail-server.git
cd reactive-mail-server
docker compose -f docker-compose.jambotron.yml up -d --build

The Compose file publishes no ports (on Jambotron the web client gets a domain). To try it locally, add ports: ["8080:8080"] to the mail service and open http://localhost:8080. The passwords of the first mailboxes are in /app/data/initial-passwords.txt inside the mail container.

Based on the reactive-mail-server source code as of November 2026.