CLItrail Docs

Log streams

On Enterprise, a log stream copies your organisation’s records to your own Amazon S3 bucket, S3-compatible storage or webhook, in batches, in order and at least once, so you can keep them as long as you like.

#Plans and roles

Log streams are part of Enterprise, up to 10 per organisation. On Free and Standard, creating one answers 402. Admins and owners create, test, pause and remove streams. After a downgrade, streams over the new limit are paused, not deleted.

#Amazon S3 and S3-compatible storage

  1. #Create a bucket and a writer

    Create a bucket, and an IAM user or role whose only permission is to write under your prefix (below). Create an access key for it.

  2. #Add the stream

    In Log streams, choose Amazon S3 and enter the bucket, its region, an optional prefix, and the access key ID and secret access key. For Cloudflare R2, MinIO or other S3-compatible storage, also enter the endpoint, such as https://<account id>.r2.cloudflarestorage.com with region auto.

  3. #Send a test

    Send test writes one test record now, as its own object, without moving the stream forward.

Each batch is one gzip-compressed NDJSON object (Content-Type: application/x-ndjson, Content-Encoding: gzip), one record per line, at:

<prefix>/clitrail/<organisation id>/<yyyy>/<mm>/<dd>/<hh>/<batch>.ndjson.gz

The batch name is two 12-digit numbers joined by -; it grows from one batch to the next, so objects list in order, and stays the same when a batch is written again. A batch is written every 5 minutes, or sooner at 5,000 records or 16 MB. Requests are signed with AWS Signature Version 4. AWS buckets are addressed virtual-hosted style unless the bucket name has dots; custom endpoints use path style.

#IAM policy

Allow only s3:PutObject, only under your prefix. CLItrail never reads, lists or deletes objects:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "CLItrailLogStream",
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::YOUR_BUCKET/YOUR_PREFIX/clitrail/*"
  }]
}

Without a prefix, the resource is arn:aws:s3:::YOUR_BUCKET/clitrail/*. If the bucket encrypts with your own KMS key, also allow kms:GenerateDataKey on that key. On Cloudflare R2, use an API token with Object Read & Write on that bucket only.

#Webhook

Enter an https URL on a public address. CLItrail generates a whsec_ signing secret and shows it once; Rotate secret replaces it and shows the new one once. Every minute, or sooner at 1,000 records or 4 MB, it posts a batch:

{"records": [
  {"schema": "clitrail.telemetry.v1", "id": "tl_mufbcblk_3f9a1c7e5b2d4a60", "time": "2026-09-24T09:12:03.512Z", "kind": "inbound", …},
  {"schema": "clitrail.telemetry.v1", "id": "tl_mufbcblo_8e0b6d2f4a1c9e37", …}
]}

Requests carry the Standard Webhooks headers webhook-id, webhook-timestamp and webhook-signature, and User-Agent: CLItrail-LogStreams/1. Verify them exactly as for webhook destinations. webhook-id names the batch and stays the same on every retry, so deduplicate on it. Test deliveries have IDs starting with tltest_.

#Delivery

  • In order, at least once. Each stream keeps a cursor. A batch that fails keeps the cursor where it was and is sent again unchanged, with the same object key or webhook-id, so a retry overwrites or is deduplicated.
  • Backoff. Retries wait 1, 2, 5, 10, 30, then 60 minutes, or longer when your endpoint sends Retry-After (up to an hour). After three failures in a row, an alert record is written.
  • From creation. A new stream starts with the records written after it was created.
  • Bounded backlog. A stream that cannot deliver keeps at most 1,000,000 records or 7 days. Past that, the oldest are skipped, and only with an alert record saying how many.
  • Health. The stream shows its last success, last error, pending records and lag. Pause a stream to stop delivery; resuming continues from its cursor.
  • Tests. Up to 10 tests an hour per organisation, across all its streams.

Credentials are write-only and sealed in the separate secret store; error messages are scrubbed of them before they are shown.