Skip to content

Instantly share code, notes, and snippets.

@lox

lox/README.md

Last active November 13, 2019 04:17
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save lox/98f9b16b2ec6a7092f746556de50e72f to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save lox/98f9b16b2ec6a7092f746556de50e72f to your computer and use it in GitHub Desktop.
Download ZIP
How Buildkite Environment is Created
Raw
README.md

I've done a bit of a deep dive into how we produce our env. The flow is slightly complicated as some environment comes from the shell that the agent runs in and some comes from buildkite.com and then some is generated each job.

For context, the code that generates the job environment is here: https://github.com/buildkite/agent/blob/2f3d6935996d45877071c01f7b19ae07db302d69/agent/job_runner.go#L307-L426

Paraphrased, the process is:

  • A build is triggered on buildkite.com, with user-provided environment (step or pipeline level) and bk specific env
  • Write out base job environment to an env file for future reference and set BUILDKITE_ENV_FILE
  • The agent job runner merges in it's env (overwriting anything set above, creating "protected" env)
  • The agent job runner takes the merged env and merges it over the top of system environment, which means it picks up anything set in the shell that buildkite-agent start is run from.
  • The agent runner forks off buildkite-agent bootstrap with the resulting env
  • Bootstrap sets some env of it's own
  • Bootstrap runs hooks, which export env. Some of these env can modify agent config
  • Bootstrap executes BUILDKITE_COMMAND

Job Environment set by Buildkite.com

These reflect what we got in our webhooks from github, etc, as well as pipeline configuration for the job.

  • BUILDKITE
  • BUILDKITE_ARTIFACT_PATHS (modifiable from hooks)
  • BUILDKITE_BRANCH
  • BUILDKITE_BUILD_CREATOR
  • BUILDKITE_BUILD_CREATOR_EMAIL
  • BUILDKITE_BUILD_CREATOR_TEAMS
  • BUILDKITE_BUILD_ID
  • BUILDKITE_BUILD_NUMBER
  • BUILDKITE_BUILD_URL
  • BUILDKITE_COMMAND
  • BUILDKITE_COMMIT
  • BUILDKITE_JOB_ID
  • BUILDKITE_LABEL
  • BUILDKITE_MESSAGE
  • BUILDKITE_ORGANIZATION_SLUG
  • BUILDKITE_PIPELINE_DEFAULT_BRANCH
  • BUILDKITE_PIPELINE_PROVIDER
  • BUILDKITE_PIPELINE_SLUG
  • BUILDKITE_PULL_REQUEST
  • BUILDKITE_PULL_REQUEST_BASE_BRANCH
  • BUILDKITE_PULL_REQUEST_REPO
  • BUILDKITE_REBUILT_FROM_BUILD_ID
  • BUILDKITE_REBUILT_FROM_BUILD_NUMBER
  • BUILDKITE_REFSPEC (modifiable from hooks)
  • BUILDKITE_REPO (modifiable from hooks)
  • BUILDKITE_RETRY_COUNT
  • BUILDKITE_SOURCE
  • BUILDKITE_TAG
  • BUILDKITE_TIMEOUT
  • BUILDKITE_TRIGGERED_FROM_BUILD_ID
  • BUILDKITE_TRIGGERED_FROM_BUILD_NUMBER
  • BUILDKITE_TRIGGERED_FROM_BUILD_PIPELINE_SLUG
  • BUILDKITE_UNBLOCKER
  • BUILDKITE_UNBLOCKER_UUID
  • BUILDKITE_UNBLOCKER_TEAMS
  • CI

Job Environment set by the Agent

These reflect the configuration of the agent, based on the configuration file and any runtime configuration set when starting the agent. These values will overwrite anything in the shell environment or set at a pipeline level, making them "protected".

  • BUILDKITE_AGENT_ENDPOINT (read from endpoint config)
  • BUILDKITE_AGENT_ACCESS_TOKEN
  • BUILDKITE_AGENT_DEBUG (read from debug config)
  • BUILDKITE_AGENT_PID
  • BUILDKITE_BIN_PATH
  • BUILDKITE_CONFIG_PATH
  • BUILDKITE_BUILD_PATH (read from build-path config)
  • BUILDKITE_HOOKS_PATH (read from hooks-path config)
  • BUILDKITE_PLUGINS_PATH (read from plugins-path config)
  • BUILDKITE_SSH_KEYSCAN (derived from no-ssh-keyscan config)
  • BUILDKITE_GIT_SUBMODULES (derived from no-git-submodules config)
  • BUILDKITE_COMMAND_EVAL (derived from no-command-eval config)
  • BUILDKITE_PLUGINS_ENABLED (derived from no-plugins config)
  • BUILDKITE_LOCAL_HOOKS_ENABLED (derived from no-local-hooks config)
  • BUILDKITE_GIT_CLONE_FLAGS (read from git-clone-flags config, modifiable from hooks)
  • BUILDKITE_GIT_CLEAN_FLAGS (read from git-clean-flags config, modifiable from hooks)
  • BUILDKITE_SHELL (read from shell config)
  • BUILDKITE_ENV_FILE
  • BUILDKITE_IGNORED_ENV
  • BUILDKITE_LAST_HOOK_EXIT_STATUS
  • BUILDKITE_COMMAND_EXIT_STATUS
  • BUILDKITE_BUILD_CHECKOUT_PATH (set in bootstrap, modifiable from hooks)

Bootstrap run-time configuration

These are ad-hoc "settings" that change behaviour in the bootstrap. These are generally read straight from the environment of the bootstrap, so can be set from pipeline env or hooks, with some exceptions.

  • BUILDKITE_PLUGIN_VALIDATION
  • BUILDKITE_CLEAN_CHECKOUT (can only be set in hooks)
  • BUILDKITE_NO_LOCAL_HOOKS
  • BUILDKITE_S3_DEFAULT_REGION
  • BUILDKITE_S3_ACL
  • BUILDKITE_S3_ACCESS_KEY_ID
  • BUILDKITE_S3_SECRET_ACCESS_KEY
  • BUILDKITE_S3_ACCESS_URL
  • BUILDKITE_ARTIFACT_UPLOAD_DESTINATION (can only be set in hooks)

Beyond that, there are also ENV that can be used to configure the agent commands. This ends up being slightly nuanced, as buildkite-agent start creates sub-processes that run bootstrap that then run shell scripts, which might in turn run things like buildkite-agent artifact upload.

Configuration for buildkite-agent start

Generated with buildkite-agent start --help | awk '{for(i=1;i<=NF;i++){if($i~/^\[\$BUILDKITE_/){gsub(/^\[\$|\]$/, "", $i); print $i}}}'

  • BUILDKITE_AGENT_CONFIG
  • BUILDKITE_AGENT_TOKEN
  • BUILDKITE_AGENT_NAME
  • BUILDKITE_AGENT_PRIORITY
  • BUILDKITE_AGENT_DISCONNECT_AFTER_JOB
  • BUILDKITE_AGENT_DISCONNECT_AFTER_JOB_TIMEOUT
  • BUILDKITE_CANCEL_GRACE_PERIOD
  • BUILDKITE_SHELL
  • BUILDKITE_AGENT_TAGS
  • BUILDKITE_AGENT_TAGS_FROM_HOST
  • BUILDKITE_AGENT_TAGS_FROM_EC2
  • BUILDKITE_AGENT_TAGS_FROM_EC2_TAGS
  • BUILDKITE_AGENT_TAGS_FROM_GCP
  • BUILDKITE_AGENT_WAIT_FOR_EC2_TAGS_TIMEOUT
  • BUILDKITE_GIT_CLONE_FLAGS
  • BUILDKITE_GIT_CLEAN_FLAGS
  • BUILDKITE_BOOTSTRAP_SCRIPT_PATH
  • BUILDKITE_BUILD_PATH
  • BUILDKITE_HOOKS_PATH
  • BUILDKITE_PLUGINS_PATH
  • BUILDKITE_TIMESTAMP_LINES
  • BUILDKITE_NO_PTY
  • BUILDKITE_NO_SSH_KEYSCAN
  • BUILDKITE_NO_COMMAND_EVAL
  • BUILDKITE_NO_PLUGINS
  • BUILDKITE_NO_PLUGIN_VALIDATION
  • BUILDKITE_NO_LOCAL_HOOKS
  • BUILDKITE_NO_GIT_SUBMODULES,
  • BUILDKITE_NO_HTTP2
  • BUILDKITE_METRICS_DATADOG
  • BUILDKITE_METRICS_DATADOG_HOST
  • BUILDKITE_AGENT_SPAWN
  • BUILDKITE_AGENT_EXPERIMENT
  • BUILDKITE_AGENT_ENDPOINT
  • BUILDKITE_AGENT_NO_COLOR
  • BUILDKITE_AGENT_DEBUG
  • BUILDKITE_AGENT_DEBUG_HTTP

Configuration for buildkite-agent bootstrap

Whilst buildkite-agent bootstrap is an internal detail, it's important to understand that it gets a different set of environment variables to configure it than are used to configure the start command. This is generally because the start command has opt-out things like --no-plugins, where as they get passed to the bootstrap as BUILDKITE_PLUGINS_ENABLED=1.

Unfortunately, if you use env to configure the agent, say by setting BUILDKITE_NO_PLUGINS=1, you will get both that env and BUILDKITE_PLUGINS_ENABLED=1 in your build.

Generated with buildkite-agent start --help | awk '{for(i=1;i<=NF;i++){if($i~/^\[\$BUILDKITE_/){gsub(/^\[\$|\]$/, "", $i); print $i}}}'

  • BUILDKITE_COMMAND
  • BUILDKITE_JOB_ID
  • BUILDKITE_REPO
  • BUILDKITE_COMMIT
  • BUILDKITE_BRANCH
  • BUILDKITE_TAG
  • BUILDKITE_REFSPEC
  • BUILDKITE_PLUGINS
  • BUILDKITE_PULL_REQUEST
  • BUILDKITE_AGENT_NAME
  • BUILDKITE_ORGANIZATION_SLUG
  • BUILDKITE_PIPELINE_SLUG
  • BUILDKITE_PIPELINE_PROVIDER
  • BUILDKITE_ARTIFACT_PATHS
  • BUILDKITE_ARTIFACT_UPLOAD_DESTINATION
  • BUILDKITE_CLEAN_CHECKOUT
  • BUILDKITE_GIT_CLONE_FLAGS
  • BUILDKITE_GIT_CLEAN_FLAGS
  • BUILDKITE_BIN_PATH
  • BUILDKITE_BUILD_PATH
  • BUILDKITE_HOOKS_PATH
  • BUILDKITE_PLUGINS_PATH
  • BUILDKITE_COMMAND_EVAL
  • BUILDKITE_PLUGINS_ENABLED
  • BUILDKITE_PLUGIN_VALIDATION
  • BUILDKITE_LOCAL_HOOKS_ENABLED
  • BUILDKITE_SSH_KEYSCAN
  • BUILDKITE_GIT_SUBMODULES
  • BUILDKITE_PTY
  • BUILDKITE_SHELL
  • BUILDKITE_BOOTSTRAP_PHASES
  • BUILDKITE_AGENT_DEBUG
@harrietgrace
Copy link

harrietgrace commented Jan 28, 2019

Could I please get definitions for: BUILDKITE_IGNORED_ENV, BUILDKITE_LAST_HOOK_EXIT_STATUS, and BUILDKITE_COMMAND_EXIT_STATUS?

Also, I just wanted to say again that this doc is so fantastic!

@lox
Copy link
Author

lox commented Jan 29, 2019

Sure!

BUILDKITE_IGNORED_ENV is a list of environment variables that have been set in your pipeline that are protected and will be overridden. They will show up as a warning in your build too. A good example might be when folks try and set BUILDKITE_GIT_CLEAN_FLAGS at a pipeline level. This is mostly an implementation detail, as we use to pass it to the bootstrap from the agent, I can't imagine users wanting to use this.

BUILDKITE_LAST_HOOK_EXIT_STATUS is the exit code for the last hook that ran, for instance if the pre-command hook fails, the pre-exit hook will still run and it will be able to see the exit code for the failed pre-command hook. In practice this is an internal detail, most folks would be interesting in BUILDKITE_COMMAND_EXIT_STATUS.

BUILDKITE_COMMAND_EXIT_STATUS is the exit code from the command that was run in the command hook. It's most useful for seeing if the command failed in the pre-exit hook!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment