haze/README.md

Haze

Hazy with a chance of clouds.

Easy setup and management of Nextcloud test instances using docker

What

haze provides an easy way to set up Nextcloud test instances with a choice of php version, database server, optional s3 or ldap setup and more.

Setup

Requirements

• Docker

Installation

• Grab a binary from the Codeberg releases and place it in your $PATH

Config

Create a file ~/.config/haze/haze.toml with the following options:

sources_root = "/path/to/nextcloud/sources"

See the configuration section for more options.

Quick examples

• Start a Nextcloud instance with postgresql, and s3 primary storage:

  haze start pgsql s3
  

• Start a Nextcloud instance with sqlite, php 8.3 and an smb external storage:

  haze start 8.3 smb
  

• Run specific units test against an oracle database

  haze test oracle apps/dav/tests/unit/Connector/Sabre
  

Managing instances

Start an instance

haze start [--name <name>] [--detach] [database] [php-version] [services] [vX.Y.Z]

Where database is one of sqlite, mysql, mariadb, pgsql or oracle with an optional version (e.g. pgsql:12), defaults to sqlite. And php-version is one of 8.0, 8.1, 8.2, 8.3, 8.4 or 8.5, defaults to the maximum version support by the current Nextcloud version.

You can specify a version number (e.g. v32.0.2) to use the sources from a release instead of using the local sources.

Use --name <name> to give the instance a specific name instead of a randomly generated one. For example:

Use --detach to give the instance its own sources instead of sharing sources_root with all other instances.

Additionally, you can use the following options when starting an instance:

• s3: set up an S3 server and configure to Nextcloud to use it as primary storage.
  • s3s: enable TLS for the S3 setup.
  • s3mb: enable multi-bucket S3 setup.
  • s3m: enable multi-instance S3 setup.
• ldap: set up an LDAP server.
• office: set up a Nextcloud Office server.
• onlyoffice setup an onlyoffice document server.
• push set up client push.
• smb: set up a samba server for external storage use.
• dav: set up a WebDAV server for external storage use.
• sftp: set up a SFTP server for external storage use.
• sftp-key: set up a SFTP server for external storage use with public key authentication.
• kaspersky: set up a kaspersky scan engine server in http mode. ( Requires manually setting up the image)
• kaspersky-icap: setup a kaspersky scan engine server in ICAP mode.
• clamav: set up a local clam av scanner in executable mode.
• clamav-socket: set up a clam av scanner in socket mode.
• clamav-icap: set up a clam av scanner in ICAP mode.
• clamav-icap-tls: set up a clam av scanner in ICAP mode with TLS encryption.
• oc: start an ownCloud instance in the same network.
• imaginary: start an Imaginary service and configure it for preview generation.
• mail: start a smtp4dev server and configure it the mail server.
• webhook start a webhook tester
• redis: start a separate container for redis.
• redis-tls: connect to redis over TLS.
• <path to app.tar.gz>: by specifying the path to an app package this package will be extracted into the apps. directory of the new instance (overwriting any existing app code). This can be used to quickly test a packaged app.
• The name of any configured preset.

Run tests in a new instance

haze test [database] [php-version] [phpunit version] [path]

Where path is a file or folder to run PHPUnit in, relative to the sources root.

List running instances

haze

or

haze list

Remove all running instances

haze clean

Instances with their own sources

By default every instance shares the sources configured as sources_root, so all instances always run the same code. An instance started with --detach gets its own git worktree of sources_root instead, created in worktree_dir, which lets you run several instances on different branches at the same time.

haze start --name my-fix --detach

The worktree is checked out with a detached head. Every app in one of the app_directories that gets enabled during setup receives its own worktree as well, other apps keep running from the shared app directory.

The worktrees are removed together with the instance, by haze stop or haze clean.

Controlling running instances

The following commands run against the most recently started instance and allow optionally providing a match to select a specific instance by its name.

Open an instance

haze [match] open

Open the database of an instance

haze [match] db

Execute a command on an instance

haze [match] [service] [cmd]

If no cmd is specified it will launch bash

If a service name or db is provided, the command will be in the container of the service or database.

Create a new instance and run a command

haze [match] shell [cmd]

If no cmd is specified it will launch bash

Execute an occ command on an instance

haze [match] occ [cmd]

Connect to the database on an instance

haze [match] db

Show the logs of an instance

haze [match] logs

Stop an instance

haze [match] stop

Pin an instance

haze [match] pin

Pinned instances will not be removed by haze clean.

Unpin an instance

haze [match] unpin

Run a command with instance environment variables set

haze [match] env <cmd> [args]

Runs the provided command with NEXTCLOUD_URL, DATABASE_URL and REDIS_URL environment variables set for the matched instance.

This is intended to run a local push daemon against an instance.

Update the container images

haze update

Edit a file in an instance with the local $EDITOR

haze [match] edit <path>

Reload the php config of an instance

haze [match] reload

The php configuration can edit changed with haze edit /config/php.ini

Checkout a branch for all local apps

haze git checkout [branch]

Checks out the branch in all git repositories within the apps folder.

Defaults to the branch matching the current checked out server versions (e.g. master or stable33).

master and main can be used interchangeably.

Pull remote changes for all local apps

haze git pull

Performs a pull in all git repositories within the apps folder.

Federation

Multiple instances can reach each other by using their instance name as domain name to allow for testing federation between instances. Alternatively, you can set up the haze proxy and the proxied domains to get https support between instances.

Proxy

By default, instances can be accessed by their IP. In order to get more memorable URLs and allow supporting https, haze comes with a builtin reverse proxy to allow using a wildcard domain.

Requirements

• A domain name you can set wildcard DNS records for
• A reverse proxy like Nginx or Apache
• (optionally) a wildcard ssl certificate (can be acquiring using letsencrypt and dns verification)

DNS Setup

• Set a DNS record for *.haze.example.com and haze.example.com pointing to your development machine.
• Set the proxy configuration with your domain and desired listen endpoint.
• Set up a service to run haze proxy in the background as your own user. A systemd user service is recommended (see haze.service for an example).
• Configure your reverse proxy of choice to proxy *.haze.example.com and haze.example.com to the proxy's listen endpoint
• (optional) acquire a wildcard ssl certificate for your domain and set your reverse proxy to use it. This will be highly dependent on your DNS provider, this lists some DNS providers and supported ACME clients.

Local Setup

• Setup dnsmasq to resolve *.haze.test to your development machine.
• Generate a wildcard ssl certificate for *.haze.test using mkcert:

# Generate local wildcard certificate
mkcert -cert-file <path-to-your-certificats>haze.test.crt -key-file <path-to-your-certificats>haze.test.key '*.haze.test'

• Set up a service to run haze proxy in the background as your own user. A systemd user service is recommended (see haze.service for an example).
• Setup a reverse proxy to proxy *.haze.test and haze.test to the haze proxy's socket. Example for Nginx:

upstream haze-handler {
    server unix:/run/haze/haze.sock;
}

server {
    listen 80;
    listen 443 ssl;
    http2 on;
    server_name *.haze.test;

    ssl_certificate     <path-to-your-certificats>/haze.test.crt;
    ssl_certificate_key <path-to-your-certificats>/haze.test.key;

    location / {
            proxy_pass http://haze-handler;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Usage

When the proxy is configured, generated URLs for the instances will use a subdomain of the configured domain, e.g. the rolling-bees instance will be available at rolling-bees.haze.example.com. Additionally, haze.example.com will automatically point to the last created instance.

Additionally, the proxy allows access to the server containers trough either <instance id>-<service id>.haze.example.com for a specific instance, or <service-id>.haze.example.com for the last created instance. For example rolling-bees-mail.haze.example.com will give access to the smtp4dev web interface of the rolling-bees instance.

Haze scripts

Haze scripts combine a set of instance options and a script to run in the instance.

Haze scripts are intended to way to create automated ways of running more complex tests are setting up more complex instances.

A script contains of 3 paths

1. An optional shebang line setting haze as the interperter, e.g. #! /usr/bin/env haze
2. A shebang line setting the options for the instance creation as the interperter, e.g. #! haze shell pgsql s3
3. The rest that is ran as a script inside the created instace.

For example, the following script will create an instance with postgresql and s3 primary storage. Then creates a new file, gets the file id, read the object of the file from S3, validate that it contains the expected contents, and cleanup the instance.

#! /home/robin/Projects/haze/target/debug/haze script
#! haze shell pgsql s3

echo 'test' | occ file:put '-' /admin/files/test.txt
FILE_ID=$(occ info:file /admin/files/test.txt | grep fileid: | grep -oE '[0-9]+')
OBJECT_CONTENTS=$(occ file:object:get urn:oid:$FILE_ID -)
if [ "$OBJECT_CONTENTS" == 'test' ]; then
    echo "object contains expected contents"
else
    echo "object does not contains expected contents!"
fi

Script modes

Scripts can be either shell or start scripts.

shell scripts will automatically remove the created instance once the script is done, just like haze shell will. The intended use case for this is performing some tests in the created instance without leaving state behind.

start scripts on the other hand will keep the instance, like haze start will. The intended use case for this is creating more complex instances without having to configure a dedicated preset.

The script mode is determined based on the shebang line. The mode set in a script can be overriden by running the script with haze script [shell|start] path-to-script.sh.

Nushell scripts

By default, the script contents are executed as a bash script, you can instead execute the script as a nushell script by using .nu as the file extention.

Configuration

Configuration is loaded from ~/.config/haze/haze.toml and has the following options

sources_root = "/path/to/sources" # path of the nextcloud sources. required
app_directories = ["/path/to/sources/more_app"] # paths to additional app directories.
work_dir = "/path/to/temp/dir" # path to temporary directory. optional, defaults to "/tmp/haze"
worktree_dir = "/path/to/worktrees" # where to create the worktrees of detached instances. optional, defaults to "<work_dir>/worktrees"

[auto_setup] # optional
enabled = false # whether or not to automatically install nextcloud on `haze start`. enabled by default
username = "foo" # username for admin user during auto setup. optional, defaults to "admin"
password = "bar" # password for admin user during auto setup. optional, defaults to "admin"
enable_apps = ["files_external"] # apps to enable after setup, defaults to []
disable_apps = ["contacts"] # apps to disable after setup, defaults to []
post_setup = [# commands to execute after setup, defaults to []
    "occ group:add test",
]
config = { "foo" = "bar" } # configuration options to set before install

[[volume]] # optional
source = "/tmp/haze-shared"
target = "/shared"
create = true

[[volume]]
source = "/home/me/Downloads"
target = "/Downloads"
read_only = true

[proxy] # optional
address = "haze.example.com" # base domain
https = true # Is the proxy behind a https terminating proxy
listen = "/run/haze/haze.sock" # either a unix socket path
#listen = "127.0.0.1:8080"     # or a socket address

# presets allow for easy usage of commonly used setups
[[preset]]
name = "groupfolders" # name of the preset
apps = ["groupfolders"] # app to enable
commands = ["occ groupfolders:create gf", "occ groupfolders:group 1 admin read write share delete"] # commands to run post-setup