CMS Pipelines
 | |
| Paradigm | Dataflow programming |
|---|---|
| Developer | John P. Hartmann (IBM) |
| First appeared | 1986 |
| Stable release | 1.1.12/000C / 2015-07-30 |
| Website | http://vm.marist.edu/~pipeline |
| Influenced by | |
| Pipeline (Unix) | |
CMS Pipelines implements the pipeline concept under the VM/CMS operating system. The programs in a pipeline operate on a sequential stream of records. A program writes records that are read by the next program in the pipeline. Any program can be combined with any other because reading and writing is done through a device independent interface.
Overview
CMS Pipelines provides a CMS command, PIPE. The argument string to the PIPE command is the pipeline specification. PIPE selects programs to run and chains them together in a pipeline to pump data through.
Because CMS programs and utilities don't provide a device independent stdin and stdout interface, CMS Pipelines has a built-in library of programs that can be called in a pipeline specification. These built-in programs interface to the operating system, and perform many utility functions.
CMS Pipelines users issue pipeline commands from the terminal or in EXEC procedures. Users can write programs in REXX that can be used in addition to the built-in programs.
Example
A simple example that reads a disk file, separates records containing the string "Hello" from those that do not, and writes both sets of records to different disk files can be written as:
(end ;) < input.txt | A: locate /Hello/ | > found.txt ; A: | > notfound.txt
where the < stage reads the input disk file, the two > stages write the output disk files, and the locate stage separates the input stream into two output streams. locate's primary output (records containing Hello) is passed to the first > stage, and its secondary output (records not containing Hello) is passed through the A: connector to the second > stage. The ; divides the specification into 2 pipelines. The collection of pipelines is called a pipeline set.
Features
The concept of a simple pipeline is extended in these ways:
- A program can define a subroutine pipeline to perform a function on all or part of its input data.
- A network of intersecting pipelines can be defined. Programs can be in several pipelines concurrently, which gives the program access to multiple data streams. The specification of CMS Pipelines makes it possible to construct pipelines such that the behaviour of a multi-stream pipeline and the flow of records is predictable.
- A program can dynamically redefine the pipeline topology. It can replace itself with another pipeline, it can insert a pipeline segment before or after itself, or both. A program can use data in the pipeline to build pipeline specifications.
CMS Pipelines offers several features to improve the robustness of pipelines:
- A syntax error in the overall pipeline structure or in any one program causes the entire pipeline to be suppressed.
- Startup of the programs in the pipeline and allocation of resources is coordinated by the CMS Pipelines dispatcher. Individual programs can participate in that coordination to ensure irreversable actions are postponed to a point where all programs in the pipelines have had a chance to verify the arguments and are ready to process data. When the pipeline is terminated, the dispatcher ensures resources are released again.
- Errors while data flow in the pipeline can be detected by all participating programs. For example, a disk file might not be replaced in such circumstances.
History
John Hartmann, of IBM Denmark, started development of CMS Pipelines in 1980.[1] The product was marketed by IBM as a separate product during the 80's and integrated in VM/ESA late 1991. With each release of VM, the CMS Pipelines code was upgraded as well until it was functionally frozen at the 1.1.10 level in VM/ESA 2.3 in 1997. Since then, the latest level of CMS Pipelines has been available for download from the CMS Pipelines homepage for users who wish to explore new function.
An implementation of CMS Pipelines for TSO was released in 1995 as BatchPipeWorks in the BatchPipes/MVS product. The up-to-date TSO implementation has been available as a Service Offering from IBM Denmark until 2010.
Both versions are maintained from a single source code base and commonly referred to as CMS/TSO Pipelines. The specification is available in the Author's Edition.[2]
See also
References
- ↑ VM and the VM Community, Melinda Varian
- ↑ CMS/TSO Pipelines Author's Edition Author's Edition