Last active
September 25, 2020 17:58
-
-
Save nimaid/84af16cd21ffe98f692f7155e99ace83 to your computer and use it in GitHub Desktop.
A bash script template for Dockerfile `RUN` commands, with automatic APT/PIP package management.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| set -e # Causes script to exit on errors (and therefore causes a build failure) | |
| #~~~~~~~~~~~~~~~~~~~~~~ DOCKER SETUP SCRIPT BEST PRACTICES ~~~~~~~~~~~~~~~~~~~~~~~# | |
| # # | |
| # This bash script is designed to be used in a RUN command in a Dockerfile. It is # | |
| # structured so as to encourage good practices when creating a Docker image. # | |
| # # | |
| # When a Docker command finishes, a snapshot of the filesystem is taken. Then, # | |
| # Docker figures out what files were created/changed/deleted, and only saves the # | |
| # changes to the image. This a called a "layer". A Docker image is actually a # | |
| # series of these layers stacked on top of each other. When running the final # | |
| # image, Docker starts out the filesystem with just the first layer that was # | |
| # added (probably a FROM command). Then, it takes the differences of the next # | |
| # layer that was created and applies them, thereby updating what the filesystem # | |
| # looks like. Docker keeps doing this until all layers have been applied, and the # | |
| # resultant filesystem is then what is seen by the running image. # | |
| # # | |
| # Because of this, if files are created in one Docker command, and are then # | |
| # removed/changed in a later Docker command, the old files will be included in # | |
| # the final image size. Unless care is taken with managing packages and cleaning # | |
| # up, the final image can wind up being filled with "ghost" files. These are not # | |
| # visible to the final image, but are still hidden in the filesystem. This not # | |
| # only results in a bloated image with slower upload and download times, but it # | |
| # can also lead to security vulnerabilities. # | |
| # # | |
| # To avoid this, try to adhere to the following guidelines: # | |
| # # | |
| # * Every dependency installed in this script should be removed here as well. # | |
| # * Even if the dependency is needed later, re-install it in that step. # | |
| # * The only files/packages left behind after this script finishes should be # | |
| # those which are required in the final image (at runtime, after building). # | |
| # * Put temporarily needed packages in APT_TEMP_PACKAGES and PIP_TEMP_PACKAGES. # | |
| # * These are automatically handled correctly by the script template. # | |
| # * They will be removed when the script finishes. # | |
| # * Packages which are installed before this script runs will not be removed. # | |
| # * Put packages that need to be installed permanently in APT_PERM_PACKAGES and # | |
| # PIP_PERM_PACKAGES. # | |
| # * These are automatically handled correctly by the script template. # | |
| # * These will *not* be removed when the script finishes. # | |
| # * These can *not* be removed properly in later steps! # | |
| # * Packages here will override temporary packages. (Will not be removed.) # | |
| # * Only perform one self-contained "task" in this script. # | |
| # * A good example: # | |
| # * Download/Compile/Install source for application X. # | |
| # * Delete source/build folders for application X. # | |
| # * A **REALLY BAD** example: # | |
| # * Clean up files from application Y. # | |
| # * Should have been done previously. # | |
| # * Will just hide files from the final image. # | |
| # * Download/Compile/Install source for applications X + Z. # | |
| # * Should be separate steps. # | |
| # * (Script fails to delete source/build folders for applications X + Z.) # | |
| # * These files cannot be truly deleted later, only hidden. # | |
| # # | |
| #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~# | |
| # Package names which are automatically added and removed. | |
| # Unless it's used by the final image, put your packages here. | |
| APT_TEMP_PACKAGES=( | |
| ) | |
| PIP_TEMP_PACKAGES=( | |
| ) | |
| # Same as above, but upgrades will be forced even if the package already installed. | |
| # The old version will still be hidden in the filesystem (older layers) | |
| APT_TEMP_PACKAGES_UPGRADE=( | |
| ) | |
| PIP_TEMP_PACKAGES_UPGRADE=( | |
| ) | |
| # Same as the forced upgrade above, but installs recommends as well | |
| # This could install unneeded stuff, be careful! | |
| APT_TEMP_PACKAGES_FULL=( | |
| ) | |
| # Package names which are automatically added (and *not* removed). | |
| # ONLY put package names which are used in final image! | |
| # Uninstalling these packages later WILL NOT WORK CORRECTLY. | |
| APT_PERM_PACKAGES=( | |
| ) | |
| PIP_PERM_PACKAGES=( | |
| ) | |
| # Same as above, but upgrades will be forced even if the package already installed. | |
| # The old version will still be hidden in the filesystem (older layers) | |
| APT_PERM_PACKAGES_UPGRADE=( | |
| ) | |
| PIP_PERM_PACKAGES_UPGRADE=( | |
| ) | |
| # Same as the forced upgrade above, but installs recommends/missing as well | |
| # This could install unneeded stuff, be careful! | |
| APT_PERM_PACKAGES_FULL=( | |
| ) | |
| # Put your setup code in the function below. | |
| # Please install apt and pip packages using the variables above to avoid issues. | |
| # Only enter your setup code inside this function. Do not put code before or after. | |
| function setup_code() { | |
| } | |
| #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! DO NOT EDIT !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!# | |
| # Get current working directory and the directory of the script | |
| COMMAND_DIR=$PWD | |
| SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" | |
| # Change working directory to this script's directory | |
| cd $SCRIPT_DIR | |
| # Function to check if an apt package is installed | |
| #TODO: Also work for versioned | |
| function apt-package-installed() { | |
| echo $(dpkg-query -W -f='${Status}' $1 2>/dev/null | grep -c "ok installed" || echo > /dev/null) | |
| } | |
| # Function to remove items in one array from another array | |
| function remove_items_from_array() { | |
| local -n _item_array=$1 | |
| local -n _delete_array=$2 | |
| # Unset duplicates | |
| unset new_array | |
| for target in "${_delete_array[@]}"; do | |
| for i in "${!_item_array[@]}"; do | |
| if [[ ${_item_array[i]} = $target ]]; then | |
| unset '_item_array[i]' | |
| fi | |
| done | |
| done | |
| # Remove gaps | |
| for i in "${!_item_array[@]}"; do | |
| new_array+=( "${_item_array[i]}" ) | |
| done | |
| _item_array=("${new_array[@]}") | |
| unset new_array | |
| } | |
| # Ignore already installed apt packages (and do not uninstall) | |
| APT_TEMP_PACKAGES_CLEAN=( ) | |
| for PKG in ${APT_TEMP_PACKAGES[@]}; do | |
| PKG_INSTALLED=$(apt-package-installed $PKG) | |
| if [ $PKG_INSTALLED -eq 0 ]; then | |
| APT_TEMP_PACKAGES_CLEAN=( ${APT_TEMP_PACKAGES_CLEAN[@]} $PKG ) | |
| fi | |
| done | |
| APT_PERM_PACKAGES_CLEAN=( ) | |
| for PKG in ${APT_PERM_PACKAGES[@]}; do | |
| PKG_INSTALLED=$(apt-package-installed $PKG) | |
| if [ $PKG_INSTALLED -eq 0 ]; then | |
| APT_PERM_PACKAGES_CLEAN=( ${APT_PERM_PACKAGES_CLEAN[@]} $PKG ) | |
| fi | |
| done | |
| # Ignore apt temp packages that are also in perm packages | |
| remove_items_from_array APT_TEMP_PACKAGES_CLEAN APT_PERM_PACKAGES_CLEAN | |
| # Ignore apt temp upgrade packages that are in full | |
| remove_items_from_array APT_TEMP_PACKAGES_UPGRADE APT_TEMP_PACKAGES_FULL | |
| # Ignore apt temp regular packages that are in upgrade | |
| remove_items_from_array APT_TEMP_PACKAGES_CLEAN APT_TEMP_PACKAGES_UPGRADE | |
| # Ignore apt temp regular packages that are in full | |
| remove_items_from_array APT_TEMP_PACKAGES_CLEAN APT_TEMP_PACKAGES_FULL | |
| # Ignore apt perm upgrade packages that are in full | |
| remove_items_from_array APT_PERM_PACKAGES_UPGRADE APT_PERM_PACKAGES_FULL | |
| # Ignore apt perm regular packages that are in upgrade | |
| remove_items_from_array APT_PERM_PACKAGES_CLEAN APT_PERM_PACKAGES_UPGRADE | |
| # Ignore apt perm regular packages that are in full | |
| remove_items_from_array APT_PERM_PACKAGES_CLEAN APT_PERM_PACKAGES_FULL | |
| # Detect if pip is used, and if so, install it | |
| if [[ ! -z $(echo ${PIP_TEMP_PACKAGES[@]} ${PIP_PERM_PACKAGES[@]} ${PIP_TEMP_PACKAGES_UPGRADE[@]} ${PIP_PERM_PACKAGES_UPGRADE[@]}) ]]; then | |
| PIP_USED=1 | |
| for PKG in python3 python3-pip; do | |
| # See if pip requirement not installed... | |
| PKG_INSTALLED=$(apt-package-installed $PKG) | |
| if [ $PKG_INSTALLED -eq 0 ]; then | |
| # If it's not already in temp or perm... | |
| if [[ -z $(echo ${APT_TEMP_PACKAGES_CLEAN[@]} ${APT_PERM_PACKAGES_CLEAN[@]} | grep -w $PKG || echo > /dev/null) ]]; then | |
| APT_TEMP_PACKAGES_CLEAN=( ${APT_TEMP_PACKAGES_CLEAN[@]} $PKG ) | |
| fi | |
| fi | |
| done | |
| else | |
| PIP_USED=0 | |
| fi | |
| # Consolidate apt packages | |
| APT_PACKAGES_CLEAN=( ${APT_TEMP_PACKAGES_CLEAN[@]} ${APT_PERM_PACKAGES_CLEAN[@]} ) | |
| APT_PACKAGES_UPGRADE=( ${APT_TEMP_PACKAGES_UPGRADE[@]} ${APT_PERM_PACKAGES_UPGRADE[@]} ) | |
| APT_PACKAGES_FULL=( ${APT_TEMP_PACKAGES_FULL[@]} ${APT_PERM_PACKAGES_FULL[@]} ) | |
| APT_USED=0 | |
| # Detect if apt is used for upgrade packages, and if so, install packages | |
| if [[ ! -z ${APT_PACKAGES_FULL[@]} ]]; then | |
| APT_USED=1 | |
| DEBIAN_FRONTEND=noninteractive apt-get update | |
| DEBIAN_FRONTEND=noninteractive apt-get install --yes --upgrade --fix-missing --quiet ${APT_PACKAGES_FULL[@]} | |
| fi | |
| # Detect if apt is used for upgrade packages, and if so, install packages | |
| if [[ ! -z ${APT_PACKAGES_UPGRADE[@]} ]]; then | |
| if [ $APT_USED -eq 0 ]; then | |
| DEBIAN_FRONTEND=noninteractive apt-get update | |
| fi | |
| APT_USED=1 | |
| DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends --yes --upgrade --quiet ${APT_PACKAGES_UPGRADE[@]} | |
| fi | |
| # Detect if apt is used for regular packages, and if so, install packages | |
| if [[ ! -z ${APT_PACKAGES_CLEAN[@]} ]]; then | |
| if [ $APT_USED -eq 0 ]; then | |
| DEBIAN_FRONTEND=noninteractive apt-get update | |
| fi | |
| APT_USED=1 | |
| DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends --yes --quiet ${APT_PACKAGES_CLEAN[@]} | |
| fi | |
| # Install pip packages, if used | |
| if [ $PIP_USED -eq 1 ]; then | |
| # Get list of installed pip packages | |
| PIP_PKGS_INSTALLED=$(python3 -m pip list --format=columns) | |
| # Function to check if a pip package is installed | |
| function pip-package-installed() { | |
| echo $(echo ${PIP_PKGS_INSTALLED[@]} | grep -wc $1 || echo > /dev/null) | |
| } | |
| # Ignore already installed pip packages (and do not uninstall) | |
| PIP_TEMP_PACKAGES_CLEAN=( ) | |
| for PKG in ${PIP_TEMP_PACKAGES[@]}; do | |
| PKG_INSTALLED=$(pip-package-installed $PKG) | |
| if [ $PKG_INSTALLED -eq 0 ]; then | |
| PIP_TEMP_PACKAGES_CLEAN=( ${PIP_TEMP_PACKAGES_CLEAN[@]} $PKG ) | |
| fi | |
| done | |
| PIP_PERM_PACKAGES_CLEAN=( ) | |
| for PKG in ${PIP_PERM_PACKAGES[@]}; do | |
| PKG_INSTALLED=$(pip-package-installed $PKG) | |
| if [ $PKG_INSTALLED -eq 0 ]; then | |
| PIP_PERM_PACKAGES_CLEAN=( ${PIP_PERM_PACKAGES_CLEAN[@]} $PKG ) | |
| fi | |
| done | |
| # Ignore pip temp regular packages that are also in perm packages | |
| remove_items_from_array PIP_PERM_PACKAGES_CLEAN PIP_PERM_PACKAGES_CLEAN | |
| # Ignore pip temp regular packages that are in upgrade | |
| remove_items_from_array PIP_TEMP_PACKAGES_CLEAN PIP_TEMP_PACKAGES_UPGRADE | |
| # Ignore pip perm regular packages that are in upgrade | |
| remove_items_from_array PIP_PERM_PACKAGES_CLEAN PIP_PERM_PACKAGES_UPGRADE | |
| # Consolidate pip packages | |
| PIP_PACKAGES_CLEAN=( ${PIP_TEMP_PACKAGES_CLEAN[@]} ${PIP_PERM_PACKAGES_CLEAN[@]} ) | |
| PIP_PACKAGES_UPGRADE=( ${PIP_TEMP_PACKAGES_UPGRADE[@]} ${PIP_PERM_PACKAGES_UPGRADE[@]} ) | |
| # Install all pip packages | |
| if [[ ! -z ${PIP_PACKAGES_UPGRADE[@]} ]]; then | |
| python3 -m pip --no-cache-dir install --upgrade --retries 10 --timeout 60 ${PIP_PACKAGES_UPGRADE[@]} | |
| fi | |
| if [[ ! -z ${PIP_PACKAGES_CLEAN[@]} ]]; then | |
| python3 -m pip --no-cache-dir install --retries 10 --timeout 60 ${PIP_PACKAGES_CLEAN[@]} | |
| fi | |
| fi | |
| # Run user setup code | |
| setup_code | |
| # Uninstall temporary pip packages | |
| PIP_TEMP_PACKAGES=( ${PIP_TEMP_PACKAGES_CLEAN[@]} ${PIP_TEMP_PACKAGES_UPGRADE[@]} ) | |
| if [ $PIP_USED -eq 1 ]; then | |
| if [[ ! -z ${PIP_TEMP_PACKAGES[@]} ]]; then | |
| python3 -m pip uninstall -y ${PIP_TEMP_PACKAGES[@]} | |
| fi | |
| fi | |
| # Uninstall temporary apt packages | |
| APT_TEMP_PACKAGES=( ${APT_TEMP_PACKAGES_CLEAN[@]} ${APT_TEMP_PACKAGES_UPGRADE[@]} ${APT_TEMP_PACKAGES_FULL[@]} ) | |
| if [ $APT_USED -eq 1 ]; then | |
| if [[ ! -z ${APT_TEMP_PACKAGES[@]} ]]; then | |
| apt-get remove -y --quiet ${APT_TEMP_PACKAGES[@]} | |
| fi | |
| fi | |
| # Clean up apt | |
| apt-get clean | |
| apt-get autoremove -y | |
| rm -rf /var/lib/apt/lists/* | |
| # Clean up tmp | |
| rm -rf /tmp/* | |
| # Update symbolic links | |
| ldconfig | |
| # Return to the original working directory where the script was called | |
| cd $COMMAND_DIR | |
| #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! DO NOT EDIT !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!# | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment