Spatial ETL
Spatial ETL, also known as Geospatial Transformation and Load (GTL), provides the data processing functionality of traditional Extract, Transform, Load (ETL) software, but with a primary focus on the ability to manage spatial data (which may also be called GIS, geographic, or map data).
A spatial ETL system may translate data directly from one format to another, or via an intermediate format; the latter being more common when transformation of the data is to be carried out.
Transform
The transformation phase of a spatial ETL process allows a variety of functions; some of these are similar to standard ETL, but some are unique to spatial data.
Spatial data commonly consists of a geographic element and related attribute data; therefore spatial ETL transformations are often described as being either geometric transformations - transformation of the geographic element - or attribute transformations - transformations of the related attribute data.
Common geospatial transformations
- Reprojection: the ability to convert spatial data between one coordinate system and another.
- Spatial transformations: the ability to model spatial interactions and calculate spatial predicates
- Topological transformations: the ability to create topological relationships between disparate datasets
- Resymbolisation: the ability to changes the cartographic characteristics of a feature, such as colour or line-style
- Geocoding: the ability to convert attributes of tabular data into spatial data
Additional features
Desirable features of a spatial ETL application are:
- Data comparison: Ability to carry out change detection and perform incremental updates
- Conflict management: Ability to manage conflicts between multiple users of the same data
- Data dissemination: Ability to publish data via the internet or deliver by email regardless of source format
- Semantic processing: Ability to understand the rules of different data formats to minimize user input whilst preserving meaning
Spatial ETL uses
Spatial ETL has a number of distinct uses:
- Data cleansing: The removal of errors within a dataset
- Data merging: The bringing together of multiple datasets into a common framework - conflation is a good example of this
- Data verification: The comparison of multiple datasets for verification and quality assurance purposes
- Data conversion: Conversion between different data formats.
Spatial ETL - origins and history
Although ETL tools for processing non-spatial data have existed for some time, ETL tools that can manage the unique characteristics of spatial data only emerged in the early 1990s.
Spatial ETL tools emerged in the GIS industry to enable interoperability (or the exchange of information) between the industry’s diverse array of mapping applications and associated proprietary formats. However, spatial ETL tools are also becoming increasingly important in the realm of Management Information Systems as a tool to help organizations integrate spatial data with their existing non-spatial databases, and also to leverage their spatial data assets to develop more competitive business strategies.
Traditionally, GIS applications have had the ability to read or import a limited number of spatial data formats, but with few specialist ETL transformation tools; the concept being to import data then carry out step-by-step transformation or analysis within the GIS application itself. Conversely, spatial ETL does not require the user to import or view the data, and generally carries out its tasks in a single predefined process.
With the push to achieve greater interoperability within the GIS industry, many existing GIS applications are now incorporating spatial ETL tools within their products; the ArcGIS Data Interoperability Extension being a good example of this.
Tools
Programmers can set up ETL processes using almost any programming language, but building such processes from scratch can become complex. Increasingly, companies are buying ETL tools to help in the creation of ETL processes.[1]
By using an established ETL framework, one may increase one's chances of ending up with better connectivity and scalability. A good ETL tool must be able to communicate with the many different relational databases and read the various file formats used throughout an organization. ETL tools have started to migrate into Enterprise Application Integration, or even Enterprise Service Bus, systems that now cover much more than just the extraction, transformation, and loading of data. Many ETL vendors now have data profiling, data quality, and metadata capabilities. A common use case for ETL tools include converting CSV files to formats readable by relational databases. A typical translation of millions of records is facilitated by ETL tools that enable users to input csv-like data feeds/files and import it into a database with as little code as possible.
ETL tools are typically used by a broad range of professionals - from students in computer science looking to quickly import large data sets to database architects in charge of company account management, ETL tools have become a convenient tool that can be relied on to get maximum performance. ETL tools in most cases contain a GUI that helps users conveniently transform data, using a visual data mapper, as opposed to writing large programs to parse files and modify data types.[2]
While ETL Tools have traditionally been for developers and I.T. staff, the new trend is to provide these capabilities to business users so they can themselves create connections and data integrations when needed, rather than going to the I.T. staff.[3] Gartner refers to these non-technical users as Citizen Integrators.[4]
Open Source Tools
Commercial Tools
Commercially available Spatial ETL tools include:
See also
References
- ↑ ETL poll produces unexpected results
- ↑ "ETL Tools Vendor Comparison". Retrieved 31 January 2016.
- ↑ "The Inexorable Rise of Self Service Data Integration". Retrieved 31 January 2016.
- ↑ "Embrace the Citizen Integrator".
- "Data Transformation/Spatial Extract, Transform, and Load". Retrieved 5 July 2012.
- "Unleash the Power of Your Spatial Data with Spatial ETL Technology". Retrieved 5 July 2012.