* fix: k8s error handling
(cherry picked from commit f633a3efb42432a6d2492712aead865a950c8dca)
* include main in main cloud-runner pipeline
(cherry picked from commit a40fbe941bba1ba4593c83c754b37363a969bfe5)
* Initial support for adding a UNITY_LICENSING_SERVER parameter to build parameters
* Test to figure out what the working directory is of current bash script
* Outputting current directory and using $ACTION_FOLDER
* Add resources folder to mounted docker volumes. Used by activation script to copy over template file for unity licensing server
* use awk instead of sed due to http characters breaking syntax
* mkdir for unity config
* Add -p flag to mkdir so parents are also created if missing
* Initial work on returning floating license when using licensing server
* Checking licensing server first for now, since serial is always set
* Parse and save acquired floating license for use for returning after build
* Clean up duplicate commands in activate.sh
* Fixed running string as command, use it as input instead
* Fixed cloud runner tests failing when using a ssh remote.
* Clean up of test files and unnecessary logging
* Moved process of generating services-config.json file from platform specific activate scripts to typescript
* Fixed path
* Correct aws logs link
* Correct aws logs link
* better aws cli commands and better cleanup for aws
* better aws cli commands and better cleanup for aws
* improved garbage collection cli options
* Only allow ephemeral runners when using cloud runner integration tests flag to avoid unexpected hangup
* Only allow ephemeral runners when using cloud runner integration tests flag to avoid unexpected hangup
* fix issue #393
* Extract follow log stream service
* consolidate into one pipeline file
* consolidate into one pipeline file
* Update cloud-runner-aws-pipeline.yml
* Update cloud-runner-k8s-pipeline.yml
* yarn build
* yarn build
* correct branch ref
* correct branch ref passed to target repo
* Create k8s-tests.yml
* Delete k8s-tests.yml
* correct branch ref passed to target repo
* correct branch ref passed to target repo
* Always describe AWS tasks for now, because unstable error handling
* Remove unused tree commands
* Use lfs guid sum
* Simple override cache push
* Simple override cache push and pull override to allow pure cloud storage driven caching
* Removal of early branch (breaks lfs caching)
* Remove unused tree commands
* Update action.yml
* Update action.yml
* Support cache and input override commands as input + full support custom hooks
* Increase k8s timeout
* replace filename being appended for unknclear reason
* cache key should not contain whitespaces
* Always try and deploy rook for k8s
* Apply k8s files for rook
* Update action.yml
* Apply k8s files for rook
* Apply k8s files for rook
* cache test and action description for kuber storage class
* Correct test and implement dependency health check and start
* GCP-secret run, cache key
* lfs smudge set explicit and undo explicit
* Run using external secret provider to speed up input
* Update cloud-runner-aws-pipeline.yml
* Add nodejs as build step dependency
* Add nodejs as build step dependency
* Cloud Runner Tests must be specified to capture logs from cloud runner for tests
* Cloud Runner Tests must be specified to capture logs from cloud runner for tests
* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs
* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs
* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs
* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs
* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs
* better defaults for new inputs
* better defaults
* merge latest
* force build update
* use npm n to update node in unity builder
* use npm n to update node in unity builder
* use npm n to update node in unity builder
* correct new line
* quiet zipping
* quiet zipping
* default secrets for unity username and password
* default secrets for unity username and password
* ls active directory before lfs install
* Get cloud runner secrets from
* Get cloud runner secrets from
* Cleanup setup of default secrets
* Various fixes
* Cleanup setup of default secrets
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* AWS secrets manager support
* less caching logs
* default k8s storage class to pd-standard
* more readable build commands
* Capture aws exit code 1 reliably
* Always replace /head from branch
* k8s default storage class to standard-rwo
* cleanup
* further cleanup input
* further cleanup input
* further cleanup input
* further cleanup input
* further cleanup input
* folder sizes to inspect caching
* dir command for local cloud runner test
* k8s wait for pending because pvc will not create earlier
* prefer k8s standard storage
* handle empty string as cloud runner cluster input
* local-system is now used for cloud runner test implementation AND correctly unset test CLI input
* local-system is now used for cloud runner test implementation AND correctly unset test CLI input
* fix unterminated quote
* fix unterminated quote
* do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name
* remove head and heads from branch prefix
* fix reversed caching direction of cache-push
* fixes
* fixes
* fixes
* cachePull cli
* fixes
* fixes
* fixes
* fixes
* fixes
* order cache test to be first
* order cache test to be first
* fixes
* populate cache key instead of using branch
* cleanup cli
* garbage-collect-aws cli can iterate over aws resources and cli scans all ts files
* import cli methods
* import cli files explicitly
* import cli files explicitly
* import cli files explicitly
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* log parameters in cloud runner parameter test
* log parameters in cloud runner parameter test
* log parameters in cloud runner parameter test
* Cloud runner param test before caching because we have a fast local cache test now
* Using custom build path relative to repo root rather than project root
* aws-garbage-collect at end of pipeline
* aws-garbage-collect do not actually delete anything for now - just list
* remove some legacy du commands
* Update cloud-runner-aws-pipeline.yml
* log contents after cache pull and fix some scenarios with duplicate secrets
* log contents after cache pull and fix some scenarios with duplicate secrets
* log contents after cache pull and fix some scenarios with duplicate secrets
* PR comments
* Replace guid with uuid package
* use fileExists lambda instead of stat to check file exists in caching
* build failed results in core error message
* Delete sample.txt
* cloud-runner-system prefix changed to cloud-runner
* Update cloud-runner-aws-pipeline.yml
* remove du from caching, should run manually if interested in size, adds too much runtime to job to include by default
* github ephemeral pipeline support
* github ephemeral pipeline support
* Merge remote-tracking branch 'origin/main' into cloud-runner-develop
# Conflicts:
# dist/index.js.map
# src/model/cloud-runner/providers/aws/aws-task-runner.ts
# src/model/cloud-runner/providers/aws/index.ts
* garbage collection
* garbage collection
* self hosted runner pipeline
* self hosted runner pipeline
* self hosted runner pipeline
* self hosted runner pipeline
* self hosted runner pipeline
* self hosted runner pipeline
* self hosted runner pipeline
* self hosted runner pipeline
* self hosted runner pipeline
* self hosted runner pipeline
* ephemeral runner pipeline
* ephemeral runner pipeline
* ephemeral runner pipeline
* download runner each time
* download runner each time
* download runner each time
* garbage collect all older than 1d as part of cleanup
* download runner each time
* number container cpu and memory for aws
* per provider container defaults
* per provider container defaults
* per provider container defaults
* per provider container defaults
* Skip printing size unless cloudRunnerIntegrationTests is true
* transition zip usage in cache to uncompressed tar for speed
* transition zip usage in cache to uncompressed tar for speed
* transition zip usage in cache to uncompressed tar for speed
* transition zip usage in cache to uncompressed tar for speed
* per provider container defaults
* per provider container defaults
* per provider container defaults
* per provider container defaults
* per provider container defaults
* per provider container defaults
* per provider container defaults
* per provider container defaults
* baked in cloud formation template
* baked in cloud formation template
* baked in cloud formation template
* baked in cloud formation template
* baked in cloud formation template
* baked in cloud formation template
* baked in cloud formation template
* baked in cloud formation template
* better aws commands
* better aws commands
* parse number for cloud formation template
* remove container resource defaults from actions yaml
* remove container resource defaults from actions yaml
* skip all input readers when cloud runner is local
* prefer fs/promises
* actually set aws cloud runner step as failure if unity build fails
* default to 3gb of ram - webgl fails on 2
* Update cloud-runner-aws-pipeline.yml
* Update cloud-runner-k8s-pipeline.yml
* yarn build
* yarn build
* correct branch ref
* correct branch ref passed to target repo
* Create k8s-tests.yml
* Delete k8s-tests.yml
* correct branch ref passed to target repo
* correct branch ref passed to target repo
* Always describe AWS tasks for now, because unstable error handling
* Remove unused tree commands
* Use lfs guid sum
* Simple override cache push
* Simple override cache push and pull override to allow pure cloud storage driven caching
* Removal of early branch (breaks lfs caching)
* Remove unused tree commands
* Update action.yml
* Update action.yml
* Support cache and input override commands as input + full support custom hooks
* Increase k8s timeout
* replace filename being appended for unknclear reason
* cache key should not contain whitespaces
* Always try and deploy rook for k8s
* Apply k8s files for rook
* Update action.yml
* Apply k8s files for rook
* Apply k8s files for rook
* cache test and action description for kuber storage class
* Correct test and implement dependency health check and start
* GCP-secret run, cache key
* lfs smudge set explicit and undo explicit
* Run using external secret provider to speed up input
* Update cloud-runner-aws-pipeline.yml
* Add nodejs as build step dependency
* Add nodejs as build step dependency
* Cloud Runner Tests must be specified to capture logs from cloud runner for tests
* Cloud Runner Tests must be specified to capture logs from cloud runner for tests
* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs
* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs
* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs
* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs
* Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs
* better defaults for new inputs
* better defaults
* merge latest
* force build update
* use npm n to update node in unity builder
* use npm n to update node in unity builder
* use npm n to update node in unity builder
* correct new line
* quiet zipping
* quiet zipping
* default secrets for unity username and password
* default secrets for unity username and password
* ls active directory before lfs install
* Get cloud runner secrets from
* Get cloud runner secrets from
* Cleanup setup of default secrets
* Various fixes
* Cleanup setup of default secrets
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* Various fixes
* AWS secrets manager support
* less caching logs
* default k8s storage class to pd-standard
* more readable build commands
* Capture aws exit code 1 reliably
* Always replace /head from branch
* k8s default storage class to standard-rwo
* cleanup
* further cleanup input
* further cleanup input
* further cleanup input
* further cleanup input
* further cleanup input
* folder sizes to inspect caching
* dir command for local cloud runner test
* k8s wait for pending because pvc will not create earlier
* prefer k8s standard storage
* handle empty string as cloud runner cluster input
* local-system is now used for cloud runner test implementation AND correctly unset test CLI input
* local-system is now used for cloud runner test implementation AND correctly unset test CLI input
* fix unterminated quote
* fix unterminated quote
* do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name
* remove head and heads from branch prefix
* fix reversed caching direction of cache-push
* fixes
* fixes
* fixes
* cachePull cli
* fixes
* fixes
* fixes
* fixes
* fixes
* order cache test to be first
* order cache test to be first
* fixes
* populate cache key instead of using branch
* cleanup cli
* garbage-collect-aws cli can iterate over aws resources and cli scans all ts files
* import cli methods
* import cli files explicitly
* import cli files explicitly
* import cli files explicitly
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* import cli methods
* log parameters in cloud runner parameter test
* log parameters in cloud runner parameter test
* log parameters in cloud runner parameter test
* Cloud runner param test before caching because we have a fast local cache test now
* Using custom build path relative to repo root rather than project root
* aws-garbage-collect at end of pipeline
* aws-garbage-collect do not actually delete anything for now - just list
* remove some legacy du commands
* Update cloud-runner-aws-pipeline.yml
* log contents after cache pull and fix some scenarios with duplicate secrets
* log contents after cache pull and fix some scenarios with duplicate secrets
* log contents after cache pull and fix some scenarios with duplicate secrets
* PR comments
* Replace guid with uuid package
* use fileExists lambda instead of stat to check file exists in caching
* build failed results in core error message
* Delete sample.txt
* avoid building a custom image
* fix: remove unnecessary double-dash
* Rebuild with -- fix
* linting
* Remove unused variable
* support windows as well
* Fix -- command not found
* Fix unused import, remove docker build test
* no dockerfile anymore
Co-authored-by: Webber Takken <webber.nl@gmail.com>
* fix: misalignments in dev lifecycle
* fix: dist no longer added to staged
* fix: misalignments in dev lifecycle
* chore: multi-platform hooks and tests
* chore: multi-platform hooks and tests
* chore: add intention for colors
* chore: update lint-staged to fix color
* chore: update dist files
* feat: move to lefthook (remove husky and lint-staged)
* feat: move to lefthook (remove husky and lint-staged)
* fix: test aftereach
* fix: test aftereach
* fix: early restore call
* feat: jest fails if something gets logged to console
* chore: add todos of misplaced code
* chore: update dist files
* chore: move jest file
Running docker currently mounts the docker.sock file into the container.
This was introduced in 2ab738c083 but
there is no explanation provided.
The docker.sock file is only needed if we want to run docker inside the container
to create other images or start other containers.
I searched through the code and I did not find any such use.
In particular, on fedora this gives permission denied because docker.sock
is owned by root and the container runs under an unprivileged user.
One has to change the permissions of docker.sock
(which is actually a link to /run/podman/podman.sock) to be writeable by the user.
If we don't need to use docker inside the containers, then we can remove this file,
thus we can run this GitHub action as an unprivileged user out of the box.
In self hosted runners in fedora with SELinux enabled,
it sometimes gives a random error
```
Error: lsetxattr /var/run/docker.sock: operation not permitted
Error: The process '/usr/bin/docker' failed with exit code 126
```
This seems to happen with docker.sock which is a link to
/run/podman/podman.sock
looks like lsetxattr is broken for links.
* feat: compatibility with self-hosted runners with SELinux
When using a self-hosted runner with SELinux (fedora)
volumes need to be mounted with ":z" in order to have write access
these flags are documented [here](https://docs.docker.com/storage/bind-mounts/#configure-the-selinux-label)
* Ensure folders are created
* use if instead of short circuit
* ts convention either inline or use braces
* Fix linting
* fix linting errors
Co-authored-by: Webber Takken <webber.nl@gmail.com>
* Run yarn upgrade in order to resolve security vulnerability
* Remove unneeded git add from pre-commit hook
* Update packages
* Update non-eslint packages
* Implemented logic for windows based docker builds. Moved dockerfiles and scripts to platform specific folders.
* Add missing newline character
* Add build-tests for windows and a unity project configured to output il2cpp
* Add additional build targets (uwp and tvOS)
Adjustments to build scripts to not require win10 sdk when not needed (tvOS)
Platform-based prereq setup
Setup image tags for the new platforms with errors if building on the wrong base os
Rename test-project-il2cpp to test-project-windows to be used for all windows based project building (IL2CPP backend selected instead of mono)
Fix tests to be platform based
* Update dist/platforms/windows/steps/return_license.ps1
Co-authored-by: Webber Takken <webber.nl@gmail.com>
* Update src/model/docker.ts
Co-authored-by: Webber Takken <webber.nl@gmail.com>
* Update src/model/docker.ts
Co-authored-by: Webber Takken <webber.nl@gmail.com>
* Update src/model/docker.ts
Co-authored-by: Webber Takken <webber.nl@gmail.com>
* Fix outdated repository and homepage links in dockerfiles
* Fix comment style and rename validateWindowsPrereqs to validateWindowsPlatformRequirements
* Remove redundant comment
* Remove windows unity test project, add ProjectSettings for the il2cpp backend, and add logic to replace the projectsettings file with the il2cpp one on windows test builds.
* Fix action.test.ts to accept windows as a base platform
* Fix camelcase for wsaPlayer
* Switch from add to copy in windows dockerfile
* Change slash direction
* Switch ADD to COPY to conform with best practices, change ls to dir on windows dockerfile
* Improve error message for unset UNITY_EMAIL and UNITY_PASSWORD
* Further improve missing email and password error. Remove temppaths being mounted to docker image
* Add debug statement. TODO: Remove these
* Add more debug
* Explicitly pass in unity email to docker run
* Remove debug and fix environment variables for activation/deactivation scripts
* Prevent Unity serial from leaking to console
* Debug folder listings
* More debug print dirs
* fix debug print path
* fix reg export command
* Remove debug directory listings and try setSecret to mask serial
* Update src/model/action.ts
Co-authored-by: Webber Takken <webber.nl@gmail.com>
* Update src/model/docker.ts
Co-authored-by: Webber Takken <webber.nl@gmail.com>
* Update src/model/image-tag.ts
Co-authored-by: David Finol <davidmfinol@gmail.com>
* Update .github/workflows/build-tests.yml
Co-authored-by: David Finol <davidmfinol@gmail.com>
* Move platform validation and setup out of docker and into its own layer, remove branching on docker run command
* Fix test failure due to missing license
* Fix camelCase and duplicate variables
* Fix lint issues and make paths more understandable
* Fix typo in build-tests.yml
* Fix move command in build-tests.yml
* Different method to force move file
* Fix missing quote and backslash
* Pass unity email and password to builder action for windows build tests
* Push serial to windows test builds
* Make windows build tests only run on push to main
Co-authored-by: Webber Takken <webber.nl@gmail.com>
Co-authored-by: David Finol <davidmfinol@gmail.com>
* adding option to pass git credential
* trigger change
* trigger change
* build dist/index
* prettier
* adding set git credentials with more config
* correct docker.ts input
* change default of git credential
* try using git command line to set token
* remove git config cat
* adding api: to git config
* change to token
* change input name to reflect the type github private token
Co-authored-by: Alexander Brandstedt <alexander@infralium.com>
Co-authored-by: Alexander Brandstedt <mad01@users.noreply.github.com>
* using SSH_AUTH_SOCK (ssh agent forwarding) to pull upm private repos
* sshAgent as input parameter
* yarn run prettier --write "src/**/*.{js,ts}"
* yarn run lint --fix && yarn build
* fixed compilation after rebase
* removed RUN apt-get update && apt-get install -y openssh-client. This change needs to be done upstream. See game-ci/docker#117
* Chown files to a custom user after building
* Better describe `chownFilesTo` input
As suggested by @webbertakken: https://github.com/game-ci/unity-builder/pull/250#discussion_r624575666
Co-authored-by: Webber Takken <webber@takken.io>
* Simplify chown step in `build.sh`
Co-authored-by: Webber Takken <webber@takken.io>
Co-authored-by: Webber Takken <webber@takken.io>
To automatically upload symbols to unity, we need to define the `USYM_UPLOAD_AUTH_TOKEN` variable. Currently the build container ignores this variable, even if it's defined in the github action.
```
2021-03-26T02:35:35.5938747Z time="2021-03-26T02:35:35Z" level=fatal msg="Please provide an auth token with USYM_UPLOAD_AUTH_TOKEN environment variable"
```
Adds the ability to use a kubernetes container to run builds that are too large for the local machine running the unity-builder. Logs are streamed back during the build. Build results can then be downloaded separately.
