From 6313695d3d1237f756d1c94438070f33f80962cb Mon Sep 17 00:00:00 2001 From: bdmc Date: Mon, 6 Sep 2021 15:51:37 -0400 Subject: [PATCH 1/6] Created How-To page for DevTest environment. Created new page describing how to create a VirtualBox VM for Developers to use to properly re-create the CAcert production and test environments. --- source/devtest.rst | 225 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 source/devtest.rst diff --git a/source/devtest.rst b/source/devtest.rst new file mode 100644 index 0000000..982c929 --- /dev/null +++ b/source/devtest.rst @@ -0,0 +1,225 @@ +======================================================= +Build yourself a CAcert Test Machine for Development +======================================================= + +As developers, at some point we all need a machine, whether for raw development, +Unit Testing, Integration or System Testing, so that we can be assured that our +code is ready to submit to the Software Reviewers and Approvers on its way to +deployment in Production. + +That machine should be structured and configured to be as close a match +as possible to CAcert's production servers. + +I have written this document and prepared scripts and other tools to help other +Developers easily create such a machine for their own purposes. + +At present the machine that we are trying to match is a very old Debian machine +with PHP 5 installed. Since we can no longer install a version of Debian that +is as old as is running in Production, I found that Debian 8 would still install, +at least at present, and PHP 5.7 can still be installed on that version of Debian. + + +Assumptions and Expectations +---------------------------- + +This document will describe the use of VirtualBox to create a working copy +of CAcert's Test Server for use in development of bug fixes and other code +intended for deployment to CAcert's servers. + +It will describe all of the pieces that were used to create a working Debian 8 +system, complete with various helper tools that allow this system to be used in +a stand-alone manner, effectively air-gapped from the Internet and E-Mail. + +Since the CAcert code makes use of E-Mail for validating users and other +internal purposes, this code must trap any outgoing e-mail so that it can +be examined and then returned to the system to fulfil those validation +purposes. + +For future code development, much of this instruction can be repeated, with +principally, replacement of the Debian ISO and, therefore, upgrade of the +database server within. + + +Software Development and Code Submission in CAcert +-------------------------------------------------- + +For those who are unfamiliar, there is a good document in the CAcert code tree, +describing the process of managing the code involved in a CAcert bug fix or feature +submission. The current version can be found at `Contributing`_. Please note that +you should only make the minimum code changes necessary +for each Pull Request ( PR ), to assist +the people involved in QA and Software Assesment. + +Pre-requisites +-------------- + +* Internet connection + - Not for testing, but for obtaining the software that we are putting + into our virtual machine. +* VirtualBox [#f1]_ + - Remember that the OS of the host system does not have to match the OS of the virtual machine. +* ISO of Debian 8 or older, if possible. [#f2]_ [#f4]_ + - I was not able to install Debian 7, because of its repository requirement. [#f3]_ +* Membership in the CAcert Development group and Mailing List: cacert-devel@lists.cacert.org. +* Access to our Bug Tracker `Mantis`_. + +Everything else that I installed, I will provide in the instructions. + + +Create a running Debian +----------------------- + +VirtualBox +++++++++++ + +For those of us familiar with VirtualBox, this is relatively straight-forward. +I will try to ease the path for the others. If you have any questions, feel free +to contact me and other CAcert developers on the CAcert Developers mailing list +cacert-devel@lists.cacert.org. + +VirtualBox can be found as an installable package for most modern systems, +whether Linux, as I run, MacOS ( OS X ) or Windows, as well as Solaris! [#f1]_ + +For our purposes, most modern machines, including laptops, can run VirtualBox +and our Development VM. + + +Debian Versions ++++++++++++++++ + +Since CAcert's current production systems are running on ten-year-old versions of +Debian, for any bug fixing or enhancements to the current system, we need to try +and reproduce that environment as much as possible. + +For that reason, I am specifying Debian 8, Jessie, as our standard Development +Environment, since it is the oldest still supported version of Debian. If absolutely +necessary, we could experiment by downloading older CD versions of Debian, but for +now, let us use Jessie. + +Creating our Development and Test Platform +++++++++++++++++++++++++++++++++++++++++++ + +If you have not yet installed VirtualBox, do so now. + +Then create a new VM, using the default values offered by VirtualBox. +When asked about memory, allocate at least 1GB, but no more than half +of the memory installed in the host machine. Disks will normally +be created as compressed devices, so that you will not use as much as +you select for the disk size except under exceptional circumstances. + +Mount the Debian ISO, and start the VM. + + +First Installation Steps +++++++++++++++++++++++++ + +* Aptitude + - I prefer the command line version of aptitude to apt-get, so install it on every machine that I build. + - Once it has been installed, do: + + aptitude update + + aptitude safe-upgrade + - And then install the following tools with "aptitude install vim git rsync" +* Vim +* Git +* Rsync +* Others? +* Create a working directory + - mkdir work +* Create a source directory + - mkdir cacert + + +Installing PHP 5.7 +++++++++++++++++++ + +* Add Sources.list file + - Since PHP 5 is not normally provided by Debian 8, we need to make use of a repository provided by Ondrej Sury. + He has been creating and supporting various versions of PHP ( and a few other related packages ) for Debian + and Ubuntu for quite a few years. These currently range from PHP 5.6 through the whole PHP 7.x range. + - Perform the following steps: + + cd work + + sudo apt install ca-certificates apt-transport-https + + wget -q https://packages.sury.org/php/apt.gpg -O- | sudo apt-key add - + + sudo echo "deb https://packages.sury.org/php/ jessie main" | tee /etc/apt/sources.list.d/php.list +* Installing PHP + - aptitude update + - aptitude install php5.6 php5.6-cli php5.6-common php5.6-curl php5.6-mbstring php5.6-mysqlnd php5.6-xml + + +Installing Mail and MailHog ++++++++++++++++++++++++++++ + +* Configuring Exim4 + - cd /etc/exim4 + - vim update-exim4.conf.conf + - find line with *dc_smarthost* + - insert *localhost::1025* between quotes + - Save and Exit + - Restart Exim by *service exim4 restart* + + This will compile the configuration file +* Downloading MailHog and Installing +* Finishing Configuration + + +Apache +++++++ + +* Editing Virtual Host + +Installing Required Environment Variables +_________________________________________ + +* Deciding on values +* Editing Apache Virtual Host + +Editing PHP.ini +_______________ + + +Installing Test Manager ++++++++++++++++++++++++ + + +Installing Source Code +++++++++++++++++++++++ + + +Any More Steps +++++++++++++++ + +Starting Development +-------------------- + +Starting MailHog +++++++++++++++++ + +Etc +--- + + + +.. rubric:: Footnotes +.. [#f1] VirtualBox can be obtained either from a Linux distribution repository + or directly from `VirtualBox`_. +.. [#f2] You can download Debian ISOs from `Debian`_, but finding older ones, such + as the one that we want, can be a bit tricky. I use the `NetInst version`_ + because it is much smaller, and therefore quicker to download, but might be a bit longer to install than a full + `DVD copy`_. Note that while the NetInst version is + less than 300 MB, there are three DVD images, totalling about 13 GB! On the other hand, the NetInst version depends on a working + Debian Mirror, while the CD or DVD images might be able to be installed without that requirement. We might want to experiment. +.. [#f3] I did try downloading the DVD image for Debian 6, Squeeze, and created a Virtual Box VM using that. It was 4.2 GB, so took a + while to download. I only downloaded DVD-1, instead of the whole set, and, in fact the DVD named debian-update-6.0.10-amd64-DVD-1.iso. + Further information about creating the Debian 6.10 VM can be found in `DevTest 6`_. +.. [#f4] Note that the source for Debian CD and DVD images older that the current version is `Debian Archive`_. The NetInst + version can be found in the iso-cd directory. + + +.. _Mantis: https://bugs.cacert.org +.. _VirtualBox: https://www.virtualbox.org/wiki/Downloads +.. _Debian: https://debian.org +.. _NetInst version: https://cdimage.debian.org/cdimage/archive/8.11.1/amd64/iso-cd/debian-8.11.1-amd64-netinst.iso +.. _DVD copy: https://cdimage.debian.org/cdimage/archive/8.11.1/amd64/iso-dvd/ +.. _Contributing: https://github.com/jandd/cacert-devel/blob/contribution-guide/CONTRIBUTING.md +.. _Debian Archive: http://cdimage.debian.org/cdimage/archive/ +.. _DevTest 6: other-doc + From 78baf71ef23c897af06e6f151f30fd47361df372 Mon Sep 17 00:00:00 2001 From: bdmc Date: Wed, 8 Sep 2021 12:34:36 -0400 Subject: [PATCH 2/6] How to Build a Developer machine for CAcert First commit of new chapter on building a Developer-oriented VM for CAcert Developers and Testers. --- source/devtest.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/source/devtest.rst b/source/devtest.rst index 982c929..eec95a7 100644 --- a/source/devtest.rst +++ b/source/devtest.rst @@ -207,8 +207,9 @@ Etc `DVD copy`_. Note that while the NetInst version is less than 300 MB, there are three DVD images, totalling about 13 GB! On the other hand, the NetInst version depends on a working Debian Mirror, while the CD or DVD images might be able to be installed without that requirement. We might want to experiment. -.. [#f3] I did try downloading the DVD image for Debian 6, Squeeze, and created a Virtual Box VM using that. It was 4.2 GB, so took a - while to download. I only downloaded DVD-1, instead of the whole set, and, in fact the DVD named debian-update-6.0.10-amd64-DVD-1.iso. +.. [#f3] I did try downloading the DVD image for Debian 6, Squeeze, and created a Virtual Box VM using that. Each disk image was 4.4 GB, so took a + while to download. I needed to download both DVD-1 and DVD-2, because of missing software, the DVDs + named debian-6.0.10-amd64-DVD-1.iso and debian-6.0.10-amd64-DVD-2.iso. Further information about creating the Debian 6.10 VM can be found in `DevTest 6`_. .. [#f4] Note that the source for Debian CD and DVD images older that the current version is `Debian Archive`_. The NetInst version can be found in the iso-cd directory. From 35f86a1fffde39427b1fc8760282cea36f782fdb Mon Sep 17 00:00:00 2001 From: bdmc Date: Wed, 8 Sep 2021 12:50:58 -0400 Subject: [PATCH 3/6] Updated for Debian 7. Updated to show target as Debian 7, rather than Debian 8, since version 7 includes PHP 5 and MySQL 5, being compatible with CAcert's current production environment. --- source/devtest.rst | 48 ++++++++++++++++++++++++---------------------- 1 file changed, 25 insertions(+), 23 deletions(-) diff --git a/source/devtest.rst b/source/devtest.rst index eec95a7..3d2dc12 100644 --- a/source/devtest.rst +++ b/source/devtest.rst @@ -37,7 +37,7 @@ purposes. For future code development, much of this instruction can be repeated, with principally, replacement of the Debian ISO and, therefore, upgrade of the -database server within. +database server and PHP version within. Software Development and Code Submission in CAcert @@ -58,8 +58,8 @@ Pre-requisites into our virtual machine. * VirtualBox [#f1]_ - Remember that the OS of the host system does not have to match the OS of the virtual machine. -* ISO of Debian 8 or older, if possible. [#f2]_ [#f4]_ - - I was not able to install Debian 7, because of its repository requirement. [#f3]_ +* ISO of Debian 7 or older, if possible. [#f2]_ [#f4]_ + - I was not able to install Debian 7 from a NetInst ISO, because of its repository requirement. [#f3]_ * Membership in the CAcert Development group and Mailing List: cacert-devel@lists.cacert.org. * Access to our Bug Tracker `Mantis`_. @@ -123,27 +123,30 @@ First Installation Steps * Git * Rsync * Others? -* Create a working directory - - mkdir work +* Create a useful directory + - mkdir tools * Create a source directory - mkdir cacert -Installing PHP 5.7 +Installing PHP 5 ++++++++++++++++++ -* Add Sources.list file - - Since PHP 5 is not normally provided by Debian 8, we need to make use of a repository provided by Ondrej Sury. - He has been creating and supporting various versions of PHP ( and a few other related packages ) for Debian - and Ubuntu for quite a few years. These currently range from PHP 5.6 through the whole PHP 7.x range. - - Perform the following steps: - + cd work - + sudo apt install ca-certificates apt-transport-https - + wget -q https://packages.sury.org/php/apt.gpg -O- | sudo apt-key add - - + sudo echo "deb https://packages.sury.org/php/ jessie main" | tee /etc/apt/sources.list.d/php.list +PHP 5 is a normal part of Debian 7, so does not require any special effort to install. + * Installing PHP - aptitude update - - aptitude install php5.6 php5.6-cli php5.6-common php5.6-curl php5.6-mbstring php5.6-mysqlnd php5.6-xml + - aptitude install php5 php5-mysql + + +Installing MySQL 5 +++++++++++++++++++ + +MySQL 5 is a normal part of Debian 7, so does not require any special effort to install. + +* Installing MySQL + - aptitude update + - aptitude install mysql-5.5-server mysql-client Installing Mail and MailHog @@ -202,15 +205,15 @@ Etc .. [#f1] VirtualBox can be obtained either from a Linux distribution repository or directly from `VirtualBox`_. .. [#f2] You can download Debian ISOs from `Debian`_, but finding older ones, such - as the one that we want, can be a bit tricky. I use the `NetInst version`_ + as the one that we want, can be a bit tricky. I would normally use the `NetInst version`_ because it is much smaller, and therefore quicker to download, but might be a bit longer to install than a full `DVD copy`_. Note that while the NetInst version is - less than 300 MB, there are three DVD images, totalling about 13 GB! On the other hand, the NetInst version depends on a working - Debian Mirror, while the CD or DVD images might be able to be installed without that requirement. We might want to experiment. + less than 300 MB, there are three DVD images, totalling about 13 GB! However, only the first DVD is required for most + purposes. Since the NetInst version requires a working Debian Mirror, while the DVD does not, for anything older than + Debian 8, only the DVD ISO will work successfully. .. [#f3] I did try downloading the DVD image for Debian 6, Squeeze, and created a Virtual Box VM using that. Each disk image was 4.4 GB, so took a - while to download. I needed to download both DVD-1 and DVD-2, because of missing software, the DVDs - named debian-6.0.10-amd64-DVD-1.iso and debian-6.0.10-amd64-DVD-2.iso. - Further information about creating the Debian 6.10 VM can be found in `DevTest 6`_. + while to download. I only needed to download DVD-1 for the install, named debian-6.0.10-amd64-DVD-1.iso. + However, after I created the Debian 6 version, I discovered that the *git* in that version would not work with GitHUB. .. [#f4] Note that the source for Debian CD and DVD images older that the current version is `Debian Archive`_. The NetInst version can be found in the iso-cd directory. @@ -222,5 +225,4 @@ Etc .. _DVD copy: https://cdimage.debian.org/cdimage/archive/8.11.1/amd64/iso-dvd/ .. _Contributing: https://github.com/jandd/cacert-devel/blob/contribution-guide/CONTRIBUTING.md .. _Debian Archive: http://cdimage.debian.org/cdimage/archive/ -.. _DevTest 6: other-doc From 2d8a725e598cef8a203699ea38f70d10939a4b8d Mon Sep 17 00:00:00 2001 From: bdmc Date: Wed, 8 Sep 2021 13:21:33 -0400 Subject: [PATCH 4/6] Added explanation. --- source/devtest.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/source/devtest.rst b/source/devtest.rst index 3d2dc12..2c31dbf 100644 --- a/source/devtest.rst +++ b/source/devtest.rst @@ -60,6 +60,7 @@ Pre-requisites - Remember that the OS of the host system does not have to match the OS of the virtual machine. * ISO of Debian 7 or older, if possible. [#f2]_ [#f4]_ - I was not able to install Debian 7 from a NetInst ISO, because of its repository requirement. [#f3]_ + However, I was able to do so from the Debian 7 DVD ISO, and that is what I describe here. * Membership in the CAcert Development group and Mailing List: cacert-devel@lists.cacert.org. * Access to our Bug Tracker `Mantis`_. From 97e1308ce06f5ceb4a3e290644bfd824c2025f15 Mon Sep 17 00:00:00 2001 From: bdmc Date: Wed, 8 Sep 2021 13:25:55 -0400 Subject: [PATCH 5/6] Corrected issue with comment formatting. --- source/devtest.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/devtest.rst b/source/devtest.rst index 2c31dbf..d0d6c73 100644 --- a/source/devtest.rst +++ b/source/devtest.rst @@ -59,8 +59,8 @@ Pre-requisites * VirtualBox [#f1]_ - Remember that the OS of the host system does not have to match the OS of the virtual machine. * ISO of Debian 7 or older, if possible. [#f2]_ [#f4]_ - - I was not able to install Debian 7 from a NetInst ISO, because of its repository requirement. [#f3]_ - However, I was able to do so from the Debian 7 DVD ISO, and that is what I describe here. + - I was not able to install Debian 7 from a NetInst ISO, because of its repository requirement. [#f3]_ + However, I was able to do so from the Debian 7 DVD ISO, and that is what I describe here. * Membership in the CAcert Development group and Mailing List: cacert-devel@lists.cacert.org. * Access to our Bug Tracker `Mantis`_. From 1411239a3cd1649a30f4ff6314e65aaf6b232d88 Mon Sep 17 00:00:00 2001 From: bdmc Date: Wed, 8 Sep 2021 17:34:15 -0400 Subject: [PATCH 6/6] Added editing of Apache site file. --- source/devtest.rst | 65 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 52 insertions(+), 13 deletions(-) diff --git a/source/devtest.rst b/source/devtest.rst index d0d6c73..2f87417 100644 --- a/source/devtest.rst +++ b/source/devtest.rst @@ -92,10 +92,18 @@ Since CAcert's current production systems are running on ten-year-old versions o Debian, for any bug fixing or enhancements to the current system, we need to try and reproduce that environment as much as possible. -For that reason, I am specifying Debian 8, Jessie, as our standard Development -Environment, since it is the oldest still supported version of Debian. If absolutely -necessary, we could experiment by downloading older CD versions of Debian, but for -now, let us use Jessie. +For that reason, I am specifying +Debian 7, Stretch, as our standard Development Environment, even though +Debian 8, Jessie, is the oldest still supported version of Debian. + +During experimentation, I found that Debian 6 could not be used after +installation, since a critical part of our environment, *git* from that +version of Debian, will not work with GitHUB or other parts of our +ecosystem. + +I also found that PHP 5 is still standard in Debian 7, where it is +not in Debian 8, but has been replaced with PHP 7. + Creating our Development and Test Platform ++++++++++++++++++++++++++++++++++++++++++ @@ -116,7 +124,7 @@ First Installation Steps * Aptitude - I prefer the command line version of aptitude to apt-get, so install it on every machine that I build. - - Once it has been installed, do: + - It comes pre-installed in Debian 7, so do: + aptitude update + aptitude safe-upgrade - And then install the following tools with "aptitude install vim git rsync" @@ -147,7 +155,7 @@ MySQL 5 is a normal part of Debian 7, so does not require any special effort to * Installing MySQL - aptitude update - - aptitude install mysql-5.5-server mysql-client + - aptitude install mysql-server-5.5 mysql-client Installing Mail and MailHog @@ -169,34 +177,65 @@ Apache ++++++ * Editing Virtual Host + - cd + - cd tools + - wget setenv.conf + + vim setenv.conf + + *Some of the values in this file need to be set for your environment, others can be left* + - cd /etc/apache2/sites-available + - sudo vim default + - *duplicate the /var/www/html Directory block* + - *change /var/www/html in one of them to your cacert directory followed by: cacert-devel/www* + - Set the Document Root to that same directory, like: /home//cacert/cacert-devel/www + - Read the file *setenv.conf* from your tools directory into a blank space in *default* between + the ** branches +related to their work. + +* cd +* cd cacert +* git clone https://github.com/CAcertOrg/cacert-devel.git +* git checkout + Any More Steps ++++++++++++++ -Starting Development --------------------- +At this point, you now should have a working VM ready to start work. + + +Starting a Development Session +------------------------------ Starting MailHog ++++++++++++++++ +The first step in any development session is to start MailHog running, +since Exim4 and Apache will start when the VM start. + + Etc ---