Optimize .env and .zopen-config processing

.gitattributes

@@ -1,4 +1,5 @@
 *	text	working-tree-encoding=ISO8859-1
 *.pem   text working-tree-encoding=UTF-8
-*.png   binary  working-tree-encoding=UTF-8
+*.png   binary
 *.gif binary
+*.pax.Z binary

.version

@@ -0,0 +1 @@
+0.3.0-dt-stable

README.md

@@ -1,90 +1,98 @@
-# meta
+# meta-dt
+A reworking of the main meta project to add additional package management facilities, similar to utilities like apt, dpkg, yum, yast2, emerge .... just written in pure shell script (and a tiny bit of REXX) to remove any pre-reqs (like python/perl/bash etc).
+This fork is designed for everyday usage of the zos Open Tools ports within the USS environment or for those who wish to download the tools; users who are interested in building and/or porting packages from scratch should use the main release of meta at this time.
 
-Meta repository to tie together the various underlying z/OS Open Source tools repositories here.
+## Installation
+- Download the pax to a suitable location (for example /tmp).
+- Expand the pax using the command ```pax -rvf <filename>.pax```.  This will expand the pax to the current directory, listing the various included files as it does so.
+- From the ```meta-<ver>/bin``` directory run the following command, answering the questions appropriately:
+```
+>zopen init
+```
 
-View our documentation at https://zosopentools.github.io/meta/.
 
-## Background
+## Pre-config
+It is advised to have the following set on the system to ensure correct operation:
+```
+export _BPXK_AUTOCVT=ON
+export _CEE_RUNOPTS="$_CEE_RUNOPTS FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
+export _TAG_REDIR_ERR=txt
+export _TAG_REDIR_IN=txt
+export _TAG_REDIR_OUT=txt
+```
 
-There are some key 'foundational' Open Source technologies needed to port software. The goal of this set of repositories is to provide minimal 'port' repositories
-that can be used to get a foundational software package building on z/OS.
-We are starting with what we view as some 'foundational' technologies. One is the stack of technology to be able to build [zsh](https://sourceforge.net/projects/zsh/postdownload). Another is to
-be able to build [protocol buffers](https://github.com/protocolbuffers/protobuf/releases).
+## Sample usage
+```
+>./zopen init
+>. $HOME/.zopen-config
+>./zopen list --installed
+>zopen install which
+>zopen list --installed
+>which which
+>zopen upgrade
+```
 
-But - there is a _transitive closure_ problem to address with porting a software package, namely understanding what packages are pre-requisites
-for the software package you want to port. As an example, for zsh, we see the following:
+## Important usage notes
+- On first run, ```zopen init``` will copy this forked version of meta into the "package" area of zopen (ie. where packages are expanded and accessed from).  It will also be pinned to this release to prevent any possible updates from the "real" meta package. Removing the .pinned file from the meta-dt directory will allow for the main meta port to be installed however this will cause incompatabilities if run.
 
-zsh requires autoconf to configure the build scripts, GNU make to run Makefiles, ncurses
-zsh is easier to develop (but doesn't require) curl and git natively on z/OS, and GNU techinfo for documentation.
+- Remote respositores utilise the suffix ```port``` - where required, packages should be specified **withOUT** the suffix. eg using ```zopen install which``` rather than ```zopen install whichport```
 
-autoconf requires m4, automake, and GNU make
-m4 requires a c99 compiler with c11 features, so make sure you have the latest C/C++ compilers installed on your system.
+## System install
+- Selecting ```'/' ``` as the root filesystem will allow the tools to be available system wide for all users who configure their usage. The install needs to be done by a sysadmin [or someone with sufficient rights using the sudo port for example] as the installer will write files to the /usr tree and configuration information to /etc.  The installing sysadmin will have a `.zopen-config` generated in their ```$HOME``` directory, specific to their install properties - their GitHub Access Token for example.  There will also be a "skeleton" configuration file written as ```/etc/skel/.zopen-config``` - this can be used as a template for individual users.  The skeleton template does not contain the installer's GH token (if configured during install) but inserts a placeholder should the user have their own GH token.  Copying the ```/etc/skel/.zopen-config``` to a user's $HOME directory will allow them to source it easily on login using ```. $HOME/.zopen-config```.
+- Removing zopen and the z/OS Open Tools once installed involves: uninstalling all installed packages; removing any copies of ```.zopen-config```; removing the configured zopen root directory (by default ```/usr/local/zopen``` but is set during installation); and then running a command to find any final orphaned symlinks on the system, such as: ```/bin/find $ZOPEN_ROOTFS -type l -exec test ! -e {} \; -print```  where $ZOPEN_ROOTFS is '/' [replace `-print` with `rm -rf` to actually remove symlinks, the example command should only list what was found - care should be taken when removing any files with sysadmin authority to prevent removing critial files!]
 
-GNU make requires GNU m4, automake, autoconf, Perl, and a C compiler that is gcc compatible.
 
-## Order to Build _from scratch_
+## Problem resolution
+If the meta package does get updated to a non-dt version, then running ```./zopen alt meta -s``` from the meta-dt/bin directory within the z/OS Open Tools root filesystem should allow selection of the dt-forked meta; running any other non-forked zopen commands might break the installation or cause unexpected behaviour! 
+zopen commands take a ```--verbose``` parameter to produce additional messages that can be used to aid problem diagnosis.
 
-If you want to build the tools from scratch and not use the binary pax files available, you will want
-to tackle this in a particular order.
-First, you need to have some tools installed on your system:
+## Basic command introduction
+Most commands have extended help available using the ```--help``` parameter.  The following usage guidance should be sufficient to get a system running
 
-### System Prerequisites
+>zopen init
 
-Use `zopen-setup` to download the bootstrap environment.  This will provide you with the
-minimum required set of tools to get started.  Otherwise you can download the tools individually at the below links.
+Used to initialise a z/OS Open Tools environment. By default, this will create a ```zopen``` directory in your ```$HOME``` directory as the root filesystem (rootfs).  The rootfs holds the various packages, configuration and environment for z/OS Open Tools packages - removing this directory will revert the system without a trace.  A z/OS Open Tools main configuration file is generated in ```$HOME/.zopen-config``` - to enable the z/OS Open Tools, this will either need to be sourced after logon to the system or the following line can be added to ```$HOME/.profile``` (or .bash_profile or...) to automatically source the z/OS Open Tools configuration file
+>[ -e "$HOME/.zopen-config" ] && . $HOME/.zopen-config
+It is possible to reinitialize a system using the ```re-init``` option - doing so will remove the previous configuration though the rootfs can overlap the old filesystem to reuse installed and/or cached packages for example.  Initialisation on a system that has previously had a z/OS Open Tools configuration should allow some parameters to be copied across, such as Github tokens.
 
-| Project | License | Download link |
-|---------|---------|------------------------|
-| [gnu make 4.1](https://www.gnu.org/software/make/) | [GPL V3](https://www.gnu.org/licenses/gpl-3.0.html) | [z/OS Open Tools release](https://github.com/ZOSOpenTools/makeport/releases/tag/boot) |
-| [IBM XL C/C++ V2.4.1](https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pages/xlCC++V241ForZOsV24) | IBM [^ibm] | [ibm.com web download](https://www.ibm.com/marketing/iwm/iwm/web/dispatcher.do?source=swg-zosxlcc) |
-| [git](https://git.kernel.org/pub/scm/git/git.git/) | [LGPL V2.1](https://git.kernel.org/pub/scm/git/git.git/tree/LGPL-2.1) | [z/OS Open Tools release](https://github.com/ZOSOpenTools/gitport/releases/tag/boot)  |
-| [curl](https://github.com/curl/curl) | [curl-license](https://github.com/curl/curl/blob/master/COPYING) | [z/OS Open Tools release](https://github.com/ZOSOpenTools/curlport/releases/tag/boot) |
-| [gunzip](https://www.gnu.org/software/gzip/) | [GPL V3](https://www.gnu.org/licenses/gpl-3.0.html) | [z/OS Open Tools release](https://github.com/ZOSOpenTools/unzipport/releases/tag/boot) |
+>zopen install <package>...
 
-[^ibm]: a no-charge add-on feature for clients that have enabled the XL C/C++ compiler (an optionally priced feature) on z/OS
+Used to install a z/OS Open Tools package.  By default, the latest version is installed although options are available to install specific versions, tags or to pick from a selection [see ```--help``` for more]
 
-### Recommended software
+>zopen upgrade
 
-| Project | License | Download link |
-|---------|---------|------------------------|
-| [bash](https://www.gnu.org/software/bash/) | [GPL V3](https://www.gnu.org/licenses/gpl-3.0.html) | [z/OS Open Tools release](https://github.com/ZOSOpenTools/bashport/releases/) |
+Used to upgrade z/OS Open Tools packages.  Without parameters, all installed packages will be upgraded; individual packages can be specified on the command line to upgrade only those packages.  Packages can be "pinned" to prevent upgrading in case a later release is found to be broken or incompatible - creating a ```.pinned``` file in the package directory prevents the upgrade; removing the file allows upgrades to occur.
 
-Both IBM and Rocket provide supported versions of the software above for a fee.
+>zopen clean [--cache] [--unused] [--dangling] [--all]
 
-Taking the defaults will mean there are less variables for you to configure. We recommend you structure your sandbox as follows:
+Used to remove resources on the system.  zopen will retain old versions of packages to allow for switching versions or quick re-installs; downloaded pax files are held in a cache while the specific version files are maintained in the directory configured during initialisation.  Using the ```clean``` option removes those unused resources.  In addition, zopen utilises symlinks for maintaining the appropriate file structure - should a problem occur during package installation, removal or version change, danling symlinks might be present; the ``--dangling`` option will (slowly) analyse the rootfs and prune any dangling symlinks.
 
-- Have the root of your development file system be `$HOME/zopen` (you will want to have several gigabytes of storage for use - we recommend at least 15GB)
-- Have sub-directories called _boot_, _prod_, _dev_.
-  - _boot_: sub-directory for each tool required to bootstrap (make, git, curl, gunzip, m4)
-  - _prod_: sub-directory for tools to be installed once built. These tools will be used by downstream software, e.g. make build process will use the Perl prod build
-  - _dev_: sub-directory for tools you are building
+>zopen alt [package] [-s]
 
-### Build Order
+Used to list versions of a package if there are multiple versions present; using the ```-s``` parameter allows for the active version to be changed
 
-The tools have dependencies on other tools, and there are also typically 2 ways the tools are packaged:
+>zopen remove <package>...
 
-- one that is pre-configured and therefore doesn't need autoconf/automake and associated tools
-- one that is not pre-reconfigured and therefore does require autoconf/automake and associated tools
+Removes a package from the system, leaving the version on the system for re-use later; the ```---purge`` directory will also remove the versioned directory, requiring a potential download and/or an unpax to occur to re-install that version
 
-To build from scratch, start with the tarballs of the following tools:
 
-| Project | License | Pre-requisites |
-|---------|---------|------------------------|
-| [m4](https://www.gnu.org/software/m4/m4.html) | [GPL V3 ](https://www.gnu.org/licenses/gpl-3.0.html) | m4, curl in boot and xlclang installed on the system |
-| [perl](https://dev.perl.org/) | [GPL V3 ](https://www.gnu.org/licenses/gpl-3.0.html) | additionally requires make, git in boot and m4 in prod |
-| [make](https://www.gnu.org/software/make/) | [GPL V3 ](https://www.gnu.org/licenses/gpl-3.0.html) | perl in prod for running test cases |
-| [zlib](https://zlib.net/) | [zlib license](https://zlib.net/zlib_license.html) | make in prod |
-| [autoconf](https://www.gnu.org/software/autoconf/) | [GPL V3](https://www.gnu.org/licenses/gpl-3.0.html) | libz in prod |
-| [automake](https://www.gnu.org/software/automake/) | [GPL V3](https://www.gnu.org/licenses/gpl-3.0.html) | autoconf in prod |
+>zopen list [--installed]
 
-Once you either have these tools built, or have downloaded a pre-built pax file for the build, you may want to build other tools.
-Each tool has a _buildenv_ file and one of the entries will describe the tools it requires to build, depending
-on where the source is from (currently TARBALL or GIT clone). So for example m4 requires:
+With no parameters, will list the available packages against the currently installed versions; more usefully, using the ```--installed``` parameter lists the actually locally-installed packages rather than all potential packages
 
-`ZOPEN_TARBALL_DEPS="curl gzip make m4"`
+>zopen search <package>
 
-and:
+Searches the remote repositiory for the specified package, returning appropriate meta-data about the package if found
+
+>zopen query <option>
+
+Queries the local z/OS Open Tools sytem. See ```---help``` for more details
+
+
+
+
+### Useful resources
+- View the z/OS Open Tools project home: https://github.com/ZOSOpenTools
+- View the main Meta documentation at https://zosopentools.github.io/meta/
 
-`ZOPEN_GIT_DEPS="git make m4 help2man perl makeinfo xz autoconf automake gettext"`
 
-If you want to build from the `git` clone, you can see you will need to have more software pre-installed.