discourse_docker.git
14 months agoadd tmp directory to gitignore (#859)
Jeff Wong [Tue, 3 Sep 2024 04:12:34 +0000 (21:12 -0700)]
add tmp directory to gitignore (#859)

14 months agoFEATURE: add autocomplete (#857)
Jeff Wong [Tue, 3 Sep 2024 01:54:14 +0000 (18:54 -0700)]
FEATURE: add autocomplete (#857)

14 months agoFEATURE: add runtime features (#849)
Jeff Wong [Mon, 2 Sep 2024 22:09:38 +0000 (15:09 -0700)]
FEATURE: add runtime features (#849)

adds commands for: start, run, stop, cleanup, destroy, logs, enter, restart, rebuild -- carrying over existing run commands from launcher1.

Rebuild will also do its best to minimize downtime with the following steps:

* Detect if Discourse is running as a single container or external DB
* Detect if db:migrate is configured to run on container boot
* Build initial container (keeping existing one online)
* Exit running containers if it's a single container (otherwise keeps existing online)
* Run migrations
  * Defer migrations if db:migrate is configured to run on container boot
  * Run migrations with SKIP_POST_DEPLOYMENT_MIGRATIONS=1 if it's a 2 container setup
  * Otherwise, run all migrations
* Destroy the old container (finally stopping the current, if it's still up here)
* Start the new container
* Run post-deploy migrations
  * Run migrations with SKIP_POST_DEPLOYMENT_MIGRATIONS=0 if it's a 2 container setup

14 months agoDEV: install redis data folder if none exists (#804)
Jeff Wong [Mon, 2 Sep 2024 20:19:43 +0000 (13:19 -0700)]
DEV: install redis data folder if none exists (#804)

14 months agoMake compatible with pnpm (#854)
David Taylor [Mon, 2 Sep 2024 19:55:57 +0000 (20:55 +0100)]
Make compatible with pnpm (#854)

Switches behavior based on the presence of `yarn.lock`

See https://github.com/discourse/discourse/pull/28671

14 months agoSwitch to `discourse/ruby:3.3.4-bookworm-slim` (#853)
Alan Guo Xiang Tan [Mon, 2 Sep 2024 03:49:43 +0000 (11:49 +0800)]
Switch to `discourse/ruby:3.3.4-bookworm-slim` (#853)

14 months agoDEV: remove bundle cache (#852)
Jeff Wong [Sat, 31 Aug 2024 01:05:34 +0000 (18:05 -0700)]
DEV: remove bundle cache (#852)

On the current image this removes ~78MB from the layer

Uncompressed image sizes:
before: 3.3GB
after: 3.23GB

14 months agoRemove use of `discourse/base:aarch64` in launcher2 (#851)
Alan Guo Xiang Tan [Fri, 30 Aug 2024 02:03:17 +0000 (10:03 +0800)]
Remove use of `discourse/base:aarch64` in launcher2 (#851)

Since fc61b8a850e0a7f8c1ef0852a13d735121b4d862, we started shipping
manifests for the `discourse/base` images so we don't have to
specifically pull the tag for arm64.

14 months agoDrop `mac-os` tests for launcher_go (#850)
Alan Guo Xiang Tan [Fri, 30 Aug 2024 00:39:27 +0000 (08:39 +0800)]
Drop `mac-os` tests for launcher_go (#850)

The runners cost 10 times more and there are no macOS specific tests
which we are running.

14 months agoFEATURE: Add workflow for `launcher_go` (#848)
Jeff Wong [Thu, 29 Aug 2024 07:59:12 +0000 (00:59 -0700)]
FEATURE: Add workflow for `launcher_go` (#848)

14 months agoFEATURE: add migrate and bootstrap commands (#842)
Jeff Wong [Thu, 29 Aug 2024 03:01:28 +0000 (20:01 -0700)]
FEATURE: add migrate and bootstrap commands (#842)

Adds command for migrate only. Adds batch commands for bootstrap (build+migrate+configure)

14 months agoDEV: remove oxipng archive in layer (#846)
Jeff Wong [Thu, 29 Aug 2024 02:34:24 +0000 (19:34 -0700)]
DEV: remove oxipng archive in layer (#846)

* DEV: remove oxipng archive in layer

Saves ~52MB for the layer

* DEV: Remove nginx archives

Remove another 1.2MB from image

14 months agoDEV: Improve assertions of `cli_build_test.go` (#844)
Alan Guo Xiang Tan [Wed, 28 Aug 2024 17:02:49 +0000 (01:02 +0800)]
DEV: Improve assertions of `cli_build_test.go` (#844)

This commit improves the assertions by testing against the entire
command string and env so that we can be sure of the full command we are
running.

15 months agoIncrease timeout for scheduled build job (#845)
David Taylor [Wed, 28 Aug 2024 09:19:59 +0000 (10:19 +0100)]
Increase timeout for scheduled build job (#845)

Now that base & dev images are built in the same job, we need a little more time

15 months agoDEV: Minor readability improvements to launcher2 (#843)
Alan Guo Xiang Tan [Wed, 28 Aug 2024 06:20:17 +0000 (14:20 +0800)]
DEV: Minor readability improvements to launcher2 (#843)

15 months agoFEATURE: add configure command (#841)
Jeff Wong [Wed, 28 Aug 2024 05:13:39 +0000 (22:13 -0700)]
FEATURE: add configure command (#841)

Add 'configure' command - If run after the "build" command, this is equivalent to today's 'bootstrap' command.

Note that unlike build command, a docker run+commit pattern needs to be used here as this requires a running database + mounted volumes.

15 months ago[WIP] Launcher2 (#791)
Jeff Wong [Mon, 26 Aug 2024 19:36:41 +0000 (12:36 -0700)]
[WIP] Launcher2 (#791)

Partial merge of Launcher2's CLI

* FEATURE: merge launcher2 - build command

merge launcher2, only build commands

15 months agoFix timeout for arm64 builds not correctly set (#839)
Alan Guo Xiang Tan [Mon, 26 Aug 2024 07:05:20 +0000 (15:05 +0800)]
Fix timeout for arm64 builds not correctly set (#839)

15 months agoBase image in launcher (#838)
Alan Guo Xiang Tan [Mon, 26 Aug 2024 05:22:33 +0000 (13:22 +0800)]
Base image in launcher (#838)

This pulls in the Nginx 1.26.1 update, Ruby 3.3.4 update and Debian
bookworm update.

15 months agoDEV: Introduce arm64 dev image and multi-arch manifests (#829)
David Taylor [Mon, 26 Aug 2024 05:10:28 +0000 (06:10 +0100)]
DEV: Introduce arm64 dev image and multi-arch manifests (#829)

- Combines dev image build into the `base` job
- Calculates a single timestamp for all builds
- Pushes timestamped per-arch images to Dockerhub for base, base-slim and dev images
- If both arch build jobs are successful, multiarch manifests are generated & pushed

15 months agoUpdate GitHub actions runners to ubuntu-22.04 (#837)
David Taylor [Thu, 22 Aug 2024 10:09:50 +0000 (11:09 +0100)]
Update GitHub actions runners to ubuntu-22.04 (#837)

15 months agoBump Ruby to 3.3.4 (#836)
Alan Guo Xiang Tan [Thu, 22 Aug 2024 04:25:15 +0000 (12:25 +0800)]
Bump Ruby to 3.3.4 (#836)

Ruby 3.3.4 has fixes for some regressions in 3.3.3. I don't think it
affects us but still good to upgrade.

15 months agoSwitch to debian:bookworm-slim image (#831)
Alan Guo Xiang Tan [Thu, 22 Aug 2024 02:57:56 +0000 (10:57 +0800)]
Switch to debian:bookworm-slim image (#831)

Debian Bullseye is EOL.

15 months agoEnsure uid, gid for postgres, redis and discourse stays consistent (#835)
Alan Guo Xiang Tan [Thu, 22 Aug 2024 02:57:29 +0000 (10:57 +0800)]
Ensure uid, gid for postgres, redis and discourse stays consistent (#835)

When updating to `debian/base:bookworm` from `debian/base:bullseye`, we
noticed that the uid of the `postgres` and `redis` user changed leading
to permission issues when trying to access directories of mounted volumn
which was previously created with the old uid. The change is because the
`_apt` user is assigned a uid of `42` in Debian bookworm instead of `100`
in Debian bullseye. As a result, the `postgres` user created by the `postgres`
package is automatically assigned a uid of `100` in Debian bookworm instead of `101` in Debian
bullseye.

This commit updates the `slim.Dockefile` to manually add the `postgres`
user and group assigning it a gid of `104` and uid of `101`. The `redis`
user and group is assigned a uid of `103` and a gid of `106`. The
`discourse` user and group is assigned a uid of `1000` and gid of
`1000`.

15 months agoWarmup yarn and bundle caches in dev image (#832)
David Taylor [Tue, 20 Aug 2024 13:51:32 +0000 (14:51 +0100)]
Warmup yarn and bundle caches in dev image (#832)

Uses a multi-stage build with bind mount to avoid adding the discourse repo itself to the layers

15 months agodev: add user-installed gems to PATH (#833)
David Taylor [Tue, 20 Aug 2024 13:51:14 +0000 (14:51 +0100)]
dev: add user-installed gems to PATH (#833)

This is important for usability of things like discourse_theme, and editor tooling (e.g. ruby-lsp)

15 months agoUse flaky-test-retry system (#834)
David Taylor [Tue, 20 Aug 2024 12:44:25 +0000 (13:44 +0100)]
Use flaky-test-retry system (#834)

We use this on the main core repository, so it makes sense to use it here as well. It should reduce the overall flakiness of the discourse_docker build.

15 months agoDEV: Simplify discourse_dev postgres setup (#830)
David Taylor [Mon, 19 Aug 2024 15:44:23 +0000 (16:44 +0100)]
DEV: Simplify discourse_dev postgres setup (#830)

- Remove manual database creation, and instead promote discourse user to postgres SUPERUSER. This means that `db:drop` and `db:create` commands can be run in the dev image, just like in other local development environments. As well as simplifying things, it fixes turbo_rspec, which was previously impossible in the docker dev environment (because `discourse` didn't have permissions to create the parallel databases)

- Stop pre-migrating test database in dev image. It adds additional build time & image size, and doesn't actually help because core's `bin/docker/boot_dev` script overwrites the container's postgres directory with a volume mount

15 months agoUpgrade nginx to 1.26.1 since it is the new stable version (#827)
Alan Guo Xiang Tan [Mon, 19 Aug 2024 01:19:54 +0000 (09:19 +0800)]
Upgrade nginx to 1.26.1 since it is the new stable version (#827)

Since 1.26.x has been marked stable, 1.25.x is automatically EOL as
nginx only maintains one mainline version and one stable version.

15 months agoFix verification of ports (#818)
Pablo Ganuza Vidal [Sun, 18 Aug 2024 22:39:29 +0000 (00:39 +0200)]
Fix verification of ports (#818)

nc needs -p to publish verification code correctly, at least in Debian 12

15 months agoFIX: Broken config and variable handling in setup script (#693)
Derek J. Lambert [Sun, 18 Aug 2024 22:35:56 +0000 (17:35 -0500)]
FIX: Broken config and variable handling in setup script (#693)

* FIX: Variable isn't being set

* DEV: Goodbye tabs, they're probably feeling lonely anyway

* FIX: Variable is never used. No need for command substitution

* FIX: The likelihood of knowing the PID for the script before execution is exceptionally low

15 months agobash tab completion for launcher + discourse-setup (#676)
Jay Pfaffman [Sun, 18 Aug 2024 22:31:55 +0000 (15:31 -0700)]
bash tab completion for launcher + discourse-setup (#676)

add tab completion for launcher and discourse-setup.

For launcher, offers command (e.g., rebuild, start) and then offers yml files from containers directory. After that switches (e.g., --run-image) are offered. (Will not offer switches except in final position, sorry.)

discourse-setup offers switches (e.g., --two-container).

discourse-docter has no command line arguments.

15 months agoDEV: allow multiple configure runs (#828)
Jeff Wong [Mon, 12 Aug 2024 14:43:26 +0000 (07:43 -0700)]
DEV: allow multiple configure runs (#828)

when we have already run an initial setup, fall back to just checking for
socket, rather than outright failing if the init script has already been run.

This allows 'configure' steps to be re-run in standalone cases.

eg: `launcher2 configure app && launcher2 configure app`

current version: fails as it's missing the install_postgres file
with PR: checks for psql socket, and builds.

doing something like `launcher2 start app && launcher2 configure app` would also
print out a more correct error message, "postgres already running stop container"

15 months agoUpdate pgvector to 0.7 (#825)
Rafael dos Santos Silva [Fri, 9 Aug 2024 14:16:38 +0000 (11:16 -0300)]
Update pgvector to 0.7 (#825)

* Update pgvector to 0.7

* escape

* drop version

15 months agoRevert "Switch to Chrome for Testing and drop support for Chromium (#824)" (#826)
Alan Guo Xiang Tan [Fri, 9 Aug 2024 03:05:25 +0000 (11:05 +0800)]
Revert "Switch to Chrome for Testing and drop support for Chromium (#824)" (#826)

This reverts commit bdfcc8ad23d9c8a6754de083e9930caa8837d106.

Broke the test build

15 months agoSwitch to Chrome for Testing and drop support for Chromium (#824)
Alan Guo Xiang Tan [Fri, 9 Aug 2024 01:55:50 +0000 (09:55 +0800)]
Switch to Chrome for Testing and drop support for Chromium (#824)

We started installing Chromium because there is no linux ARM support
for Chrome yet. However, trying to run tests on Chromium seems to be
extra challenging. For example, upgrading to Debian 12 causes our
Javascript tests to fail on Chromium but not on Chrome.

Chrome for Testing was built specifically for web app testing so let's
follow Google's recommendation.

15 months agoBuild `discourse/base` and `discourse/discourse_test` image for bookworm (#823)
Alan Guo Xiang Tan [Thu, 8 Aug 2024 08:29:53 +0000 (16:29 +0800)]
Build `discourse/base` and `discourse/discourse_test` image for bookworm (#823)

We need to upgrade to bookworm because bullseye is EOL. This commit when merged into branch will push the following images to Docker hub:

1. `discourse/base:slim-bookworm`
2. `discourse/base:release-bookworm`
3. `discourse/discourse_test:slim-bookworm`
4. `discourse/discourse_test:slim-browsers-bookworm`
5. `discourse/discourse_test:release-bookworm`

15 months agoFIX: Remove `bundle config jobs` (#821)
Alan Guo Xiang Tan [Thu, 8 Aug 2024 03:23:54 +0000 (11:23 +0800)]
FIX: Remove `bundle config jobs` (#821)

This fixes a regression introduced in
bbefa1e5f387bcad0fac79fea00b39f15f6dee4e. Basically, we cannot configure
the default bundle jobs when building the image because the number of
cores used to build the image can be different from the number of cores
on the machine running the image.

15 months agoIncrease `yarn` network-timeout from 30s to 60s (#820)
David Taylor [Wed, 7 Aug 2024 13:13:50 +0000 (14:13 +0100)]
Increase `yarn` network-timeout from 30s to 60s (#820)

A number of people have reported hitting yarn timeouts on low-spec DO droplets, which causes the build to fail. This should provide a little more leeway

15 months agoFIX: Use `sharedscripts` in `/etc/logrotate.d/rails` (#819)
Alan Guo Xiang Tan [Wed, 31 Jul 2024 04:04:14 +0000 (12:04 +0800)]
FIX: Use `sharedscripts` in `/etc/logrotate.d/rails` (#819)

This commit adds `sharedscripts` which will ensure that our `postrotate`
script is only ran once even if multiple log files in the `/shared/log/rails/`
are rotated. If `sharedscripts` is not specified, we are sending `sv 1
unicorn` once per log file rotated and this has resulted in weird
behaviours like our Sidekiq process hanging indefinitely.

Note the following from the manpage for logrotate:

```
sharedscripts
Normally, prerotate and postrotate scripts are run for each log which is rotated and the absolute path to the log file is passed as first argument to the script. That means a single script may be run multiple times for log file entries which match multiple files (such as the /var/log/news/* example). If sharedscripts is specified, the scripts are only run once, no matter how many logs match the wildcarded pattern, and whole pattern is passed to them.
```

15 months agoDEV: Updated vanilla.template.yml (#817)
Juan David Martínez Cubillos [Tue, 30 Jul 2024 15:11:11 +0000 (17:11 +0200)]
DEV: Updated vanilla.template.yml (#817)

* DEV: Updated vanilla.template.yml

* updated vanilla.template.yml to make the migration process more straight forward

* removed branch pull

* implemented suggested changes

* added suggested chantes

* added before_code hook to set remote fork

* updated with suggested changes

16 months agoDEV: Remove useless lines from `web.template.yml` (#816)
Alan Guo Xiang Tan [Wed, 10 Jul 2024 23:28:47 +0000 (07:28 +0800)]
DEV: Remove useless lines from `web.template.yml` (#816)

The lines are not necessary because those config has already been set in
when we are building the image.

16 months agoBump timeout for arm64 scheduled builds to 60 mins. (#815)
Alan Guo Xiang Tan [Wed, 10 Jul 2024 02:46:30 +0000 (10:46 +0800)]
Bump timeout for arm64 scheduled builds to 60 mins. (#815)

We are hitting timeouts at 45mins.

16 months agoBump base image used by launcher to pull in Ruby 3.3.3 (#813)
Alan Guo Xiang Tan [Tue, 9 Jul 2024 02:06:19 +0000 (10:06 +0800)]
Bump base image used by launcher to pull in Ruby 3.3.3 (#813)

16 months agoDEV: update to use IM7 syntax magick in validation command (#814)
Kelv [Mon, 8 Jul 2024 09:34:01 +0000 (17:34 +0800)]
DEV: update to use IM7 syntax magick in validation command (#814)

17 months agoDEV: Add poppler-utils for PDF -> text handling (#812)
Rafael dos Santos Silva [Tue, 25 Jun 2024 14:41:23 +0000 (11:41 -0300)]
DEV: Add poppler-utils for PDF -> text handling (#812)

17 months agoDEV: Allow ImageMagick to handle PDFs. (#811)
Rafael dos Santos Silva [Mon, 24 Jun 2024 16:24:15 +0000 (13:24 -0300)]
DEV: Allow ImageMagick to handle PDFs. (#811)

17 months agoDEV: Bump Ruby to 3.3.3 (#807)
Alan Guo Xiang Tan [Fri, 14 Jun 2024 00:19:55 +0000 (08:19 +0800)]
DEV: Bump Ruby to 3.3.3 (#807)

Pulls in some bugfixes which may or may not be affecting us.

17 months agoFEATURE: Bump base image used by launcher to pull in Ruby 3.3.x take 2 (#805)
Alan Guo Xiang Tan [Mon, 3 Jun 2024 02:33:29 +0000 (10:33 +0800)]
FEATURE: Bump base image used by launcher to pull in Ruby 3.3.x take 2 (#805)

First attempt in 01ce8cf8f935bf8aeb3d96a7b124ba33a612c07d was reverted
because our new base image was not compatible with the stable branch of
discourse/discourse.

17 months agoBump Ruby to 3.3.2 (#806)
Alan Guo Xiang Tan [Thu, 30 May 2024 22:31:15 +0000 (06:31 +0800)]
Bump Ruby to 3.3.2 (#806)

Pulls in lots of bug fixes: https://github.com/ruby/ruby/releases/tag/v3_3_2

18 months agoRevert "FEATURE: Bump base image used by launcher to pull in Ruby 3.3.1 (#802)" ...
David Taylor [Mon, 20 May 2024 11:10:53 +0000 (12:10 +0100)]
Revert "FEATURE: Bump base image used by launcher to pull in Ruby 3.3.1 (#802)" (#803)

This reverts commit 01ce8cf8f935bf8aeb3d96a7b124ba33a612c07d.

We are investigating incompatibilities with Discourse stable.

18 months agoFEATURE: Bump base image used by launcher to pull in Ruby 3.3.1 (#802)
Alan Guo Xiang Tan [Mon, 20 May 2024 02:52:34 +0000 (10:52 +0800)]
FEATURE: Bump base image used by launcher to pull in Ruby 3.3.1 (#802)

18 months agoFEATURE: Bump Ruby to 3.3.1 (#801)
Alan Guo Xiang Tan [Mon, 20 May 2024 01:09:12 +0000 (09:09 +0800)]
FEATURE: Bump Ruby to 3.3.1 (#801)

Bumping Ruby to 3.3.1 to pull in latest performance and memory
improvements made to YJIT. On Discourse hosting services with Ruby 3.3.1
+ YJIT, we saw an
estimate 10-20% improvement in time spent executing Ruby code over Ruby
3.2.3 + YJIT.

18 months agoFEATURE: Update `discourse-setup` to prompt for MaxMind account ID (#796)
Alan Guo Xiang Tan [Thu, 9 May 2024 07:40:01 +0000 (15:40 +0800)]
FEATURE: Update `discourse-setup` to prompt for MaxMind account ID (#796)

In order to download the free MaxMind GeoLite2 databases, an account ID
and license key is required going forward. This commit updates
`discourse-setup` to start prompting the user to provide the MaxMind
Account ID first before asking for the MaxMind license key. If the user
does not provide the Account ID, the script will not prompt for the
license key as we assume the user has opted out.

We are aware that we don't have a reliable way to test for changes to
the `discourse-setup` script but it is what it is at this point in time.
We intend to invest resources in improving things in the future but now
is not the time.

18 months agoUpdate discourse-setup (#799)
Jay Pfaffman [Thu, 9 May 2024 07:19:54 +0000 (02:19 -0500)]
Update discourse-setup (#799)

DEV: `./launcher stop` needs to be skipped when in debug mode

18 months agoDEV: Build/release `discourse/base:release-ruby-3.3.1` for testing (#800)
Alan Guo Xiang Tan [Tue, 7 May 2024 05:29:45 +0000 (13:29 +0800)]
DEV: Build/release `discourse/base:release-ruby-3.3.1` for testing (#800)

This commit adds a `ruby_3_3` job to our Github workflow which releases
a `discourse/base:release-ruby-3.3.1` Docker image to allow us to test
Ruby 3.3.1 before eventually changing to that version as the default.

18 months agoFEATURE: Fix Discourse installation support in China (#793)
耗子 [Mon, 6 May 2024 05:47:35 +0000 (13:47 +0800)]
FEATURE: Fix Discourse installation support in China (#793)

This commit does 2 things:

1. Added a new yarn hook to replace the npm mirror before `yarn install`.
2. Modified `web.china.template.yml` to add more mirror sources.

Below is an explanation of these modifications:

- The GitHub proxy added in `web.china.template.yml` has existed in China for many years, and its repository https://github.com/hunshcn/gh-proxy has 6k+ stars, which can ensure its security and stability.
- The NPM mirror site added in `web.china.template.yml` is maintained by Alibaba Group, one of the largest Internet companies in China.
- Modified the Gem mirror in `web.china.template.yml` to the mirror provided by Tsinghua University, one of the top universities in China.
- The reason why sed is used to replace the `yarn.lock` file is because `yarn install --frozen-lockfile` is used for installation below. If the url is not replaced, the NPM mirror will not take effect.

After applying these modifications, I successfully installed Discourse on the Tencent Cloud China server. No more network problems.

18 months agoDEV: Bump default base image for launcher to `discourse/base:2.0.20240502-0021` ...
Alan Guo Xiang Tan [Thu, 2 May 2024 01:46:00 +0000 (09:46 +0800)]
DEV: Bump default base image for launcher to `discourse/base:2.0.20240502-0021` (#795)

This is necessary to pull in 303b646c3c48fc3179af954433d9fa797e70a3b9

18 months agoDEV: Bump Ruby to 3.2.4 (#794)
Alan Guo Xiang Tan [Tue, 30 Apr 2024 00:30:31 +0000 (08:30 +0800)]
DEV: Bump Ruby to 3.2.4 (#794)

This commit updates Ruby to 3.2.4 which includes security fixes for the
following CVEs:

* CVE-2024-27282: Arbitrary memory address read vulnerability with Regex search
* CVE-2024-27281: RCE vulnerability with .rdoc_options in RDoc
* CVE-2024-27280: Buffer overread vulnerability in StringIO

19 months agoDEV: update postgres 15 template for tags (#790)
Jeff Wong [Thu, 11 Apr 2024 18:52:00 +0000 (11:52 -0700)]
DEV: update postgres 15 template for tags (#790)

19 months agoAdd tags to pups templates (#751)
Jeff Wong [Wed, 10 Apr 2024 23:29:57 +0000 (16:29 -0700)]
Add tags to pups templates (#751)

* Add tags to pups templates

The purpose here is to allow greater flexibility in how and where
docker images are built and run. It achieves this by breaking up
build steps into distinct run steps which can be saved along the way.
Customizable base images may then be prebuilt with as many batteries
included as possible, with zero environment setup so those images
can then be configured at a later stage.

Add the ability to run partial pups configuration:
`build`: build base image with no db - ember build.
`precompile`: precompile stage that requires postgres and redis.
`migrate`: run migration tasks.
`db`: start bundled postgres/redis, if included.

Adds a create_db script in postgres template for creating db on the fly.
Called below in unicorn run:

updates unicorn run command with 3 env flags:
CREATE_DB_ON_BOOT: if 1, creates base db schema, allows for deferral of creation.
MIGRATE_ON_BOOT: if 1, runs db:migrate - allows for deferral of db migration.
PRECOMPILE_ON_BOOT: if 1, precompiles assets (without ember build).

PRECOMPILE_ON_BOOT initially defaults to 1 in base builds (no tags).
During the `precompile` build step, this updates the default to be 0.

All other new flags default to 0 (off). With these three flags, we're now able
to ship and start a container from a base image, and it'll be able to bootstrap
a blank database.

Updates hook to start redis before_db_migrate as before_code hook
is not guaranteed to fire before migrate tasks if pups is filtered by tags.

19 months agoRemoving the -p from the "nc" command. (#697)
SnR [Thu, 4 Apr 2024 02:23:47 +0000 (05:23 +0300)]
Removing the -p from the "nc" command. (#697)

Removing the -p from the "nc" command.
Reason:
# nc -w 4 -l -p 80
nc: cannot use -p and -l

Without -p it works just fine.

> -l' Used to specify that nc should listen for an incoming connection rather than initiate a connection to a remote host. It is an error to use this option in conjunction with the -p, -s, or -z options. Additionally, any timeouts specified with the -w option are ignored.

20 months agoDEV: Set RUBY_VERSION and DEBIAN_RELEASE env (#789)
Alan Guo Xiang Tan [Wed, 27 Mar 2024 22:40:38 +0000 (06:40 +0800)]
DEV: Set RUBY_VERSION and DEBIAN_RELEASE env (#789)

Why this change?

This envs allows the Debian release name and Ruby version to be easily
determined without having to know which commands to run.

20 months agoRevert "Upgrade Debian to Bookworm" (#788)
Alan Guo Xiang Tan [Tue, 26 Mar 2024 22:32:09 +0000 (06:32 +0800)]
Revert "Upgrade Debian to Bookworm" (#788)

This reverts commit 23e7b55d42a573bfc313ff965ed3c0a1ea7e0f45.

The CI build on `discourse/discourse` and our internal CI is broken
because of this.

20 months agoUpgrade Debian to Bookworm
Loïc Guitaut [Tue, 26 Mar 2024 09:13:38 +0000 (10:13 +0100)]
Upgrade Debian to Bookworm

20 months agoDEV: Add a default browser for testem in discourse_test
Loïc Guitaut [Mon, 25 Mar 2024 16:55:20 +0000 (17:55 +0100)]
DEV: Add a default browser for testem in discourse_test

Related to https://github.com/discourse/discourse/pull/26244

20 months agoAdd Chromium to our images
Loïc Guitaut [Thu, 21 Mar 2024 15:27:02 +0000 (16:27 +0100)]
Add Chromium to our images

Chrome isn’t available for aarch64 yet, but Chromium (which is basically
the same browser without the proprietary bits from Google) is shipped by
Debian. They also ship a Chrome driver compiled for aarch64.

This patch adds Chromium to our images without removing Chrome on
x86_64, allowing a smooth transition to using Chromium only.

20 months agoIncrease build time for arm64 scheduled build (#785)
Alan Guo Xiang Tan [Fri, 22 Mar 2024 12:41:24 +0000 (20:41 +0800)]
Increase build time for arm64 scheduled build (#785)

Scheduled build for arm64 running on 2cores is timing out at 30 minutes.
Let's give it some more time

20 months agoRevert "Use Chromium instead of Chrome (#782)" (#783)
Alan Guo Xiang Tan [Wed, 20 Mar 2024 23:46:57 +0000 (07:46 +0800)]
Revert "Use Chromium instead of Chrome (#782)" (#783)

This reverts commit e6ffa64d9d7622327b134c8397f59edf832e4299.

We need to fix the various Chrome assumptions in Discourse core.

20 months agoFix: Remove egrep for grep -E (#680)
Steffy Fort [Wed, 20 Mar 2024 21:54:55 +0000 (22:54 +0100)]
Fix: Remove egrep for grep -E (#680)

Co-authored-by: Steffy Fort <steffy.fort@cozycloud.cc>
20 months agoUse Chromium instead of Chrome (#782)
Loïc Guitaut [Wed, 20 Mar 2024 21:48:52 +0000 (22:48 +0100)]
Use Chromium instead of Chrome (#782)

Chrome isn’t available yet for aarch64, but Chromium (which is basically
the same browser without the proprietary bits from Google) is shipped by
Debian. They also ship a Chrome driver compiled for aarch64.

By using Chromium instead of Chrome, we unify how we do things
regardless of the architecture used in the generated image.

20 months agoRefactor Github action build workflow to build for both arm64/amd64 (#781)
Alan Guo Xiang Tan [Tue, 19 Mar 2024 22:26:36 +0000 (06:26 +0800)]
Refactor Github action build workflow to build for both arm64/amd64 (#781)

Why this change?

Now that we can efficiently build Docker images targeted at `linux/arm64`,
we will start to release images for `linux/arm64` in the same way we do
for `linux/amd64` images.

Images released for `linux/amd64` are tagged as follows:

1. discourse/base:2.0.\<datetime\>-slim
2. discourse/base:slim
3. discourse/base:2.0.\<datetime\>
4. discourse/base:release

For `linux/arm64`, the images are tagged as follows:

1. discourse/base:2.0.\<datetime\>-slim-arm64
2. discourse/base:slim-arm64
3. discourse/base:2.0.\<datetime\>-arm64
4. discourse/base:release-arm64
5. discourse/base:aarch64 (For backwards compatibility)

For `linux/arm64`, we unfortunately cannot install chrome because chrome
does not currently release binaries for the arch. Therefore, we install
chromium which chrome is based off and also install the chromedriver
binary for `linux/arm64` released by the electron project.

20 months agoUse Github hosted ARM runners to build arm64/aarch64 release image (#779)
Alan Guo Xiang Tan [Tue, 19 Mar 2024 02:29:56 +0000 (10:29 +0800)]
Use Github hosted ARM runners to build arm64/aarch64 release image (#779)

Why this change?

We have been given access to Github's private beta of ARM hosted
runners. Switching to ARM runners should drastically speed up the time
required for us to build our ARM image.

What does this change do?

1. Switch to use Github's ARM hosted runners.
2. Build release image for arm64 as well. We previously only built the
   slim image because building the release image through emulation is
   way too slow so we skipped the release image.
3. Update `bundle` in `release.Dockerfile` to install gems in parallel
   based on the number of cores instead of hardcoding it to 4 jobs.

20 months agoFEATURE: Update RUBY_ALLOCATOR to work on both x64 and arm64 (#777)
Rafael dos Santos Silva [Mon, 4 Mar 2024 15:42:49 +0000 (12:42 -0300)]
FEATURE: Update RUBY_ALLOCATOR to work on both x64 and arm64 (#777)

automatically

While x64 is still on jemalloc 3.6, arm64 is using latest jemalloc.

They have different names for the library file, so we will now use the
symlink to automatically load the one available.

20 months agoFIX: Config page size for Redis Jemalloc on ARM64 (#776)
Rafael dos Santos Silva [Fri, 1 Mar 2024 20:14:57 +0000 (17:14 -0300)]
FIX: Config page size for Redis Jemalloc on ARM64 (#776)

20 months agoFIX: Move arm to newer jemalloc (#775)
Rafael dos Santos Silva [Fri, 1 Mar 2024 17:34:46 +0000 (14:34 -0300)]
FIX: Move arm to newer jemalloc (#775)

20 months agoremove swapfile script
Andrew Schleifer [Thu, 29 Feb 2024 13:46:23 +0000 (13:46 +0000)]
remove swapfile script

The instructions have, for quite some time now, pointed users at the
`discourse-setup` script. That will prompt the user to create a swapfile
if necessary and configure relevant sysctls.

20 months agoFIX: Adapt jemalloc page size for pi5 bookworm PAGESIZE (#774)
Rafael dos Santos Silva [Thu, 29 Feb 2024 17:13:54 +0000 (14:13 -0300)]
FIX: Adapt jemalloc page size for pi5 bookworm PAGESIZE (#774)

Debian Bookwork on Pi 5 uses 64k pages, so we need to adapt the jemalloc.

This should be compatible with pi4 and older too.

20 months agouse DISCOURSE_BASE_URL in mail-receiver sample
Evgeni Golov [Fri, 22 Sep 2023 07:21:52 +0000 (09:21 +0200)]
use DISCOURSE_BASE_URL in mail-receiver sample

while `DISCOURSE_MAIL_ENDPOINT` is still accepted by the mail-receiver code, the documentation prefers `DISCOURSE_BASE_URL` and so should this example

see https://github.com/discourse/mail-receiver/blob/deae52039f4679b31e01302dadace3c0f018cec8/README.md

20 months agoFEATURE: Add template for PostgreSQL 15
Mwaniki Wairungu [Thu, 29 Feb 2024 09:36:58 +0000 (12:36 +0300)]
FEATURE: Add template for PostgreSQL 15

The new templates/postgres.15.template.yml file allows bootstrapping
new containers using PostgreSQL version 15, or upgrading an existing
container running on older PostgreSQL versions.

The default postgres template and base image shall be bumped in a
follow-up commit.

21 months agoPERF: Speed up building slim image by setting --jobs to number of cores (#770)
Alan Guo Xiang Tan [Tue, 6 Feb 2024 08:38:04 +0000 (16:38 +0800)]
PERF: Speed up building slim image by setting --jobs to number of cores (#770)

On a M3 Max macbook pro with 14 cores,

Before:

```
=> [25/44] RUN /tmp/install-imagemagick                                 150.6s
=> [27/44] RUN /tmp/install-jemalloc                                     54.9s
=> [31/44] RUN /tmp/install-redis                                        42.9s
```

After:

```
=> [25/44] RUN /tmp/install-imagemagick                                  44.4s
=> [27/44] RUN /tmp/install-jemalloc                                     13.7s
=> [31/44] RUN /tmp/install-redis                                        11.7s
```

21 months agoPERF: Use `-O2` gcc compilation flag for imagemagick (#768)
Alan Guo Xiang Tan [Mon, 5 Feb 2024 23:43:15 +0000 (07:43 +0800)]
PERF: Use `-O2` gcc compilation flag for imagemagick (#768)

Why this change?

We have noticed that our compiled imagemagick binary is slower than the
distributed binaries in the same environment and started debugging why.
One thing I noticed is that distributed binaries usually include the
`-O2` gcc compilation flag. When applying it locally, I saw significant
speed up.

Without -O2 flag:

```
root@1d7277f72a4f:/# time convert -limit memory 10GiB -limit disk 10GiB -size $(seq 8000 8500 | shuf | head -n1)x9000 xc:"rgb($(shuf -i 0-255 -n1),$(shuf -i 0-255 -n1),$(shuf -i 0-255 -n1))" random_image.png

real 0m3.376s
user 0m6.355s
sys 0m0.410s
root@1d7277f72a4f:/# time identify -format "%Q" random_image.png
92
real 0m1.018s
user 0m0.883s
sys 0m0.135s
```

With -O2 flag:

```
root@0779afa71102:/# time convert -limit memory 10GiB -limit disk 10GiB -size $(seq 8000 8500 | shuf | head -n1)x9000 xc:"rgb($(shuf -i 0-255 -n1),$(shuf -i 0-255 -n1),$(shuf -i 0-255 -n1))" random_image.png

real 0m1.118s
user 0m1.555s
sys 0m1.680s
root@0779afa71102:/# time identify -format "%Q" random_image.png
92
real 0m0.330s
user 0m0.197s
sys 0m0.133s
```

21 months agoFEATURE: Add early support for aarch64 in dev env
Loïc Guitaut [Thu, 11 Jan 2024 17:00:45 +0000 (18:00 +0100)]
FEATURE: Add early support for aarch64 in dev env

This patch adds some new steps to support the aarch64 architecture
on Linux.

An updated version of Rust is needed to compile the `selenium-manager`
binary as it’s not shipped with the `selenium-webdriver` gem yet.
In the same vein, Google doesn’t ship an aarch64 version of Chrome yet,
so it doesn’t make sense to install even Chromium in the image. We have
to rely on Firefox to run the system specs.

23 months agoBump default base image to discourse/base:2.0.20231218-0429
Chris Rendle-Short [Mon, 18 Dec 2023 23:30:51 +0000 (10:30 +1100)]
Bump default base image to discourse/base:2.0.20231218-0429

23 months agoSECURITY: Bump Nginx to 1.25.3 (#763)
Chris Rendle-Short [Mon, 18 Dec 2023 04:18:44 +0000 (15:18 +1100)]
SECURITY: Bump Nginx to 1.25.3 (#763)

The updated version contains mitigations against CVE-2023-44487 (HTTP/2 rapid reset attack).

Upstream changelog: https://nginx.org/en/CHANGES

23 months agoBump default base image to discourse/base:2.0.20231214-0023
Andrew Schleifer [Fri, 15 Dec 2023 05:14:16 +0000 (05:14 +0000)]
Bump default base image to discourse/base:2.0.20231214-0023

23 months agoDEV: add plugin_compatibility hook (#760)
Jeff Wong [Wed, 6 Dec 2023 21:08:48 +0000 (13:08 -0800)]
DEV: add plugin_compatibility hook (#760)

23 months agoFIX: Y/n prompt should treat "no value" as yes
Michael Brown [Thu, 30 Nov 2023 20:05:16 +0000 (15:05 -0500)]
FIX: Y/n prompt should treat "no value" as yes

2 years agoFIX: Install plugin gems in `discourse/discourse_test` image (#758)
Alan Guo Xiang Tan [Thu, 23 Nov 2023 10:12:37 +0000 (18:12 +0800)]
FIX: Install plugin gems in `discourse/discourse_test` image (#758)

Why this change?

In
https://github.com/discourse/discourse/commit/dec68d780c58ef0954865b786d04b653c20e26fa,
the `plugin:install_all_gems` Rake task was made a noop because the Rake
task itself was flawed and running a Rake task will actually activate
all plugins which installs the required gems in the process. However,
plugins are not automatically activated in the test environment which
this image operates in. As such, we need to set `LOAD_PLUGINS=1` to when
running the `plugin:install_all_gems` Rake task.

2 years agoDEV: Parameterize offline page repository (#754)
Jeff Wong [Thu, 9 Nov 2023 23:08:35 +0000 (15:08 -0800)]
DEV: Parameterize offline page repository (#754)

allow for a custom offline page repository

2 years agoadd env for migrate, precompile and create db on boot (#753)
Jeff Wong [Wed, 8 Nov 2023 02:10:32 +0000 (18:10 -0800)]
add env for migrate, precompile and create db on boot (#753)

2 years agoFEATURE: add offline page template (#752)
Jeff Wong [Thu, 2 Nov 2023 05:07:05 +0000 (22:07 -0700)]
FEATURE: add offline page template (#752)

* FEATURE: add offline page template

2 years agoBump base image for pups update (#749)
Jeff Wong [Mon, 23 Oct 2023 20:08:42 +0000 (13:08 -0700)]
Bump base image for pups update (#749)

2 years agoBump default base image to 2.0.20231023-1613 (#748)
Jeff Wong [Mon, 23 Oct 2023 18:35:30 +0000 (11:35 -0700)]
Bump default base image to 2.0.20231023-1613 (#748)

Updates latest version of pups, allowing --tags and --skip-tags arguments

2 years agoDEV: Fix broken ARM64 build (#745)
Alan Guo Xiang Tan [Tue, 17 Oct 2023 08:08:49 +0000 (16:08 +0800)]
DEV: Fix broken ARM64 build (#745)

Why this change?

The `docker tag discourse/base:aarch64-slim localhost:5000/discourse/base:aarch64-slim
` command broked in 0c93b2207d22b9de69b503460dc269f8b33935c2 because
there is no `aarch64-slim` image being built. Instead, the `ruby
auto_build.rb base_slime_64` command builds an image with the
`discourse/base:build_slim_arm64` tag.

What does this change do?

This change removes the tagging command because it is pointless for us
to tag it.

2 years agoDEV: Remove use of `--squash` flag and switch to buildx (#743)
Alan Guo Xiang Tan [Mon, 16 Oct 2023 02:41:05 +0000 (10:41 +0800)]
DEV: Remove use of `--squash` flag and switch to buildx (#743)

Why this change?

In CI, we are seeing the following warning message:

```
WARNING: experimental flag squash is removed with BuildKit. You should squash inside build using a multi-stage Dockerfile for efficiency.
```

Basically, the `--squash` flag has not been working for quite some time
and is redundant.

What does this change do?

* This change removes the use of the `--squash` flag.

* This change uses the `buildx` tool in `auto_build.rb` as we prepare to
  build images for multiple platforms.

2 years agofixup count of physical cores in discourse-setup (#739)
BigEd [Mon, 9 Oct 2023 13:47:25 +0000 (14:47 +0100)]
fixup count of physical cores in discourse-setup (#739)

See also PR #702 for branch master

2 years agoBump default base image to discourse/base:2.0.20231004-0028
Andrew Schleifer [Wed, 4 Oct 2023 03:52:52 +0000 (03:52 +0000)]
Bump default base image to discourse/base:2.0.20231004-0028

2 years agoDEV: Configure bundle jobs to 4 (#736)
Alan Guo Xiang Tan [Fri, 15 Sep 2023 01:49:13 +0000 (09:49 +0800)]
DEV: Configure bundle jobs to 4 (#736)

This ensures that we be default runs bundle install with at least 4
jobs.

2 years agoDEV: Install bundler version used by Discourse as a default. (#735)
Alan Guo Xiang Tan [Thu, 14 Sep 2023 00:38:47 +0000 (08:38 +0800)]
DEV: Install bundler version used by Discourse as a default. (#735)

Why this change?

Previously we were running `gem install bundler` to install bundler but
this would install the latest bundler version instead of the bundler
version being used by Discourse. As a result, we end up having to
install bundler over and over again when using the image most of the
time.