In some areas, reaction time is critical. One can think of an industrial company's production lines, which operate on a just-in-time basis: the slightest failure, the smallest delay, can have consequences throughout the entire value chain.
This is why it is important to be able to detect the slightest indication early on, so that we can react as quickly as possible.
This detection will be possible if we know how to integrate the data coming from the sensors of industrial equipment.
The observation will be the same for the processing of an order or incidents relating to an order. The faster we are notified, the more we can react at the right time..
Capture information and datas from the Web
We now know that not all data is found only within an organization. Many data analysis or management processes now incorporate external data (such as weather, stock market prices or exchange rates, etc.).
Much of this data is available on the Internet either publicly or through private APIs.
It has become essential for any company to be able to collect external information in order to know the state of its ecosystem and to optimize operations.
Connected objects (IoTs) are the talk of the town.
Indeed, every individual today has at least one permanently connected object at hand (telephone, watch, computer, personal scale, refrigerator, vacuum cleaner, etc.).
Each of these objects carries information about itself (its own health) or about the activity, or even the health, of the person who uses it. This information, extracted and used at the right time, can be of great service not only to organizations but also to individuals, sometimes even saving their lives.
The implementation of mechanisms for detecting data coming from connected objects can therefore be a key element in the success of certain company projects.
One of the first types of event is data presence on system devices.
The presence of files in a directory, on an FTP or SSH server, or even the presence of an email with an attachment.
This is still the most frequently used event within organizations.
Stambia offers mechanisms for intelligent detection of these events, as well as automated management mechanisms, such as the creation or automatic providing of tables to track and control these events.
More and more present within organizations, message-based architectures, offering "publish & subscribe" mechanisms, are also sources or targets of Stambia processing.
From the traditional JMS or AMQP connector, to the use of Open Source buses such as Kafka, the reception or publication of messages is done graphically and allows the design of Asynchronous or streaming type flows..
Physical data (Connected objects / MQTT / Sensors, etc.)
Data from connected sensors or objects are also important elements in the digitization process of some organizations.
Stambia makes it possible to connect to these sources (or even targets) using the standard protocols of these architectures, such as MQTT for connected objects and OPC for industrial sensors.
The advantage of the Stambia solution is that it remains identical in its development mode whatever the type of project, source, target, or the frequency of processing execution.
Most of the important data still comes mainly from traditional applications, very often based on relational databases.
Thus, the detection of changes in these databases, through Change Data Capture mechanisms, remains one of the relevant elements to feed systems in near-real time.
Stambia automatically offers change detection functionalities in the source databases. These change detections with Stambia can be done in several ways:
By trigger on tables and objects
By using CDC APIs provided by database publishers
By reading database logs (when they are readable)
Through partnerships with editors specialized in this type of detection
Once these detections have been made, Stambia automatically takes into account in its mappings the data that has changed in source.
There is no additional management to perform. Any new data will be taken into account and intelligently integrated into the targets.
Link to intelligent integration (incremental, SCD, etc.)
Detecting changes or events is important.
However, a change in a source does not necessarily mean a change in the targets.
In effect, depending on the changed data in the source, as well as what has happened in the target in the meantime, the change may not necessarily need to be applied in the target systems.
In order to take these particular management rules into account, Stambia provides target integration modes that will allow you to intelligently integrate the data.
For example, the "incremental" mode will allow you to compare the source data, resulting from the detected changes, with the already existing target data and to decide whether to insert, update or even historize the data..
The use of web services / SOA architecture
Finally, an effective solution is to react immediately and synchronously, as soon as an event occurs. SOA architecture is an equally insightful approach. Stambia makes it possible to expose web services graphically and without any programming lines.
These web services can be called up by any type of program and can therefore be integrated into the company's management processes.
This not only allows you to react immediately, but also to place yourself in a system of questions/answers between applications.
The construction of complex management processes can therefore be carried out with great simplicity and agility.
1. Designer: development environment 2. Runtime: execution engine for data integration processes, Web services, etc. 3. Production Analytics: consultation of executions and production implementation.
• Any relational database such as Oracle, PostgreSQL, MSSQL, etc. • Any NoSQL database such as MongoDB, Elasticsearch, Cassandra, etc. • Any specific database such as Vertica, Teradata, Netezza, Action Vector, Sybase IQ, Snowflake, etc. • Any Cloud system such as Amazon Web Service (AWS), Google Cloud Platform (GCP), Microsoft Azure, etc. • Any application or ERP such as SAP, Microsoft Dynamics, etc. • Any SAAS application such as Salesforce, Snowflake, Google Big Query, etc. • Any Big Data platform such as Spark, Hadoop, etc. • Any MOM or ESB middleware such as JMS, Apache Active MQ, Kafka, AMQP, etc. • Any hierarchical file such as XML, JSON, etc. • Spreadsheets such as Excel, Google Spreadsheet
○ Reverse: the structure of the database can be reversed thanks to the notion of reverse metadata ○ DDL /DML operations: supports object and data manipulation operations (DML/DDL) such as insertion, update, deletion, etc. (Insert, Update, Select, Delete, Create or Drop) ○ Integration Method: Append, Incremental Update ○ Staging: a database can be used as an intermediate step (staging area) for data transformation, reconcialization, etc. Supported modes are: ○ staging as subquery ○ staging as view ○ staging as table ○ Rejection: Rejection rules can be defined to filter or detect data that does not comply with the conditions defined during integrations. ○ 3 types of rejection can be created: Fatal, Warning, Reject ○ Differentiated treatment according to the type of data for each rejected data ○ Recycling of rejects created in previous runs ○ Replication: database replication is supported from any source such as relational or NoSQL databases, flat files, XML / JSON files, Cloud system, etc.
Data Integration - Advanced Features
○ Slowly Changing Dimension (SCD): integrations can be achieved using Slowly Changing Dimension (SCD) ○ Full support of Teradata utilities such as TPT with possibility to use specific methods such as MLOAD, FastLoad, etc. ○ Optimization functions such as SQL query band for specific steps. ○ Different types of automated integration, such as Slowly Changing Dimension Type II, Undo/Replace, Incremental, Merge, etc. ○ Change Data Capture (CDC) ○ Privacy Protect: module for managing the RGPD with the functionalities ○ Anonymization ○ Pseudonimization ○ Audits ○ etc.
Data Quality Management (DQM): data quality management directly integrated in the metadata and in the Designer
Operating system: ○ Windows XP, Vista, 2008, 7, 8, 10 in 32 or 64-bit mode ○ Linux in 32 or 64 bit mode ○ Mac OS X in 64-bit mode - Memory ○ At least 1 Gb of RAM - Disk space ○ At a minimum, there must be 300 MB of free disk space available. - Java environment ○ FMV 1.8 or higher Notes: for Linux, it is necessary to have a GTK+ 2.6.0 windowing system with all dependencies
Image Docker available for the runtime engines (Runtime) and the operating console (Production Analytics)
Open API specifications (OAS)
Version management system
Any plugin supported by Eclipse: SVN, CVS, Git, etc., as well as any file sharing system (Google Drive, One Drive / Sharepoint, Drop Box, etc.)
Migrating from other solutions
Oracle Data Integrator (ODI) *, Informatica, Datastage, talend, Microsoft SSIS * ability to migrate simply and quickly