.png "Employee"){kind=icon}
Introduction
When Incorta is installed on premises, Incorta effectively uses it's data connectors to load data from both data sources that are located within your internal network, as well as, those cloud based data sources (Oracle EBS, Salesforce, etc) that are accessible in the public network from your internal network. With the introduction of the Incorta Cloud SaaS based solution, the Incorta connectors need to maintain the same connectivity, however, now the Incorta services themselves are running in the public network. How does the Incorta Cloud connect to data resources that reside inside a private network? The answer is the Incorta Data Agent.
Simply stated, the Data Agent is a small Incorta program that is installed within your internal network. The Data Agent communicates with the Incorta Cloud Cluster and is responsible for executing the queries against data sources that are located within your internal network when requested, and then returning the results of those queries to the Cloud via a secure and encrypted data path.
What you need to know before reading this article
- You should be familiar with the Incorta Cloud offering:
- Incorta Cloud Documentation
- Supported Data Connectors
- Creating a Data Source
- Short Data Agent video
Let's Go
Where to locate and installation
As mentioned above, the Data Agent must be installed somewhere within your private network. It should be located in a location on the network that has access to the data sources it will be required to pull data from. If network latency could be an issue within the private network, then more than one Data Agent may be installed within the customer's network, choosing physical locations close to the data sources.
The Data Agent must be installed on a Linux server existing within the your network. For information about the technical requirements & procedure for installing the Data Agent service, please refer to the following link. (Requirements for the Data Agent service host & Install Procedures )
When to Use Multiple Data Agents
As mentioned above, it is possible to install multiple Data Agents within a network. The primary reasons for doing so are:
- Physical Location: it is best to create a Data Agent physical server located in the network close to the data source in order to take network latency out of the query timings. This is NOT required, but in the case where a customer may support multiple physical data centers, it may be a good idea to locate one Data Agent machine within each data center.
- Data Agent Service: The Data Agent may be installed more than once on a Data Agent machine to support the data traffic for more than one cluster (Dev, UAT, and Prod). Each version of the Data Agent must be installed in a different subdirectory and be configured with a separate Authorization File (see the configuration section below for a discussion on Auth files). Once installed, the Data Agent service can be started from each of the installed directories and each data agent service will support 1 and only 1 cluster.
To download the Data Agent software, login to your Customer Cloud Portal account at https://cloud.incorta.com/signin then open your Cluster to see the main page. Then from the "Configurations" Tab in the portal, scroll down to where you see the following:
You can see that there is both a link to installation steps & a download link for the software. Also note, that here as well you have the ability to enable or disable the use of the Data Agent functionality for your cluster. Ensure the it is toggled to the ON position. After the ZIP file is downloaded, it must be copied to the Data Agent server, and uncompressed into the folder where the Data Agent service will be run. This is also where the Auth file will be placed, and where Log files for the Data agent will be written.
Other Networking & Configuration Requirements
In order for the Data Agent service to be able to communicate with the Incorta Cloud, there are a few additional steps that need to be taken.
Customer Side Network
The communication with the Data Agent and the Incorta Cluster is bidirectional. In order for the Data Agent to communicate with the Incorta Cloud, the following steps need to be taken.
First, it will be necessary to whitelist within the customer's network the IP address associated with the URL servicing the cluster's Data Agent service. Login to the CMC and navigate to the Cluster Configurations tab as shown below. Ensure that the "Enable Data Agent" is toggled to the "ON" position. Then copy the URL from one of the "Public Hosts and Ports" boxes as seen below. (do not copy the :#### port, just the URL up to and including the .com) Open a terminal session and run the command "ping <paste url here>. The IP address you see in the results in the IP address you must whitelist in your network for the Data Agent and your Cloud Cluster to communicate.
Secondly, the Loader and Analytics services communicate with the Data Agent over 2 Ports. In many installations, these ports may be set to the Default of 6767 and 6768. However, in some Clusters, you will find alternate port numbers created dynamically by the system. You can find these port numbers as well (see the screen shot above).
These ports need to be open for external bidirectional network traffic at the customer's firewall.
Incorta Application
Setting up the Data Agent from the Incorta UI is quite simple. You must log in to Incorta with a user that has either the Admin or Schema Manager role, which will allow access to the Data Tab in Incorta.
By clicking on the New button at the top right of the screen and selecting "Add Data Agent" you will be presented with the following screen:
Fill in the name and the optional description and then click on Create. You will then be presented with the option to create the Authentication File.
Click on Generate Now, and you will see that a file is downloaded to your machine in the Downloads folder. The filename will end in a .auth file extension. This file will need to be copied to the Data Agent server on the customer's network into the <data agent install folder>/conf directory. Before the data agent service is started, the Data Agent in the Incorta UI will reflect that it is NOT communicating with the Data Agent by a red circled X as shown below:
Note on the Authorization File: The reason for the Authorization file is to give the Data Agent service the information it needs to communicate securely with the Cloud Cluster for which it is being configured. As mentioned previously, each Data Agent service can communicate with 1 and only 1 Incorta Cluster. The Authorization File (which is encrypted itself) contains the certificate and all the other information that the Data Agent service needs to connect to an Incorta Cloud Cluster.
Follow the procedure to start the Data Agent Service on the Data Agent server after having applied the Port, IP Whitelisting, and installation of the Auth file. The status should change to a green X indicating that the Data Agent service is communicating with the Incorta Cloud servers.
Special Considerations
Data Agent Upgrades
The Data Agent is software. Just like the Incorta version of the software that you are using in the Cloud, it has a version number associated with its release. There are periodic updates to the Data Agent software, and it will be necessary for you to update your Data Agent from time to time. Typically, if you are performing an upgrade to Incorta, the release notes will indicate which version of the Data Agent is current for that version of Incorta. Follow the download procedure discussed above to download the latest version of the Data Agent from your cloud portal after an upgrade to Incorta is applied. Then follow the steps to
- Stop the data agent service using the "./agent.sh stop" command
- Take a backup copy of your Auth file
- Apply the new Zip file to the <data agent install directory>
- Place the Auth file in the <data agent install directory>/conf directory if it is not already there
- Start the data agent service using the "./agent.sh start" command
Your Data Agent service in the UI should report green X status. If it does not, contact Support for assistance.
Data Sources that are NOT Data Agent Compatible
It should be noted that some of the Incorta Data Connectors are NOT compatible with the Data Agent. Most of the connectors that are NOT supported by the Data Agent are data sources hosted by other SaaS providers, where our Incorta connector is utilizing a published API to get the data from that data source. Examples of data sources not supported by the Data Agent include:
- Oracle Cloud Applications
- Salesforce
- SAP
- etc.
As SaaS providers themselves, customers are typically accessing these systems using pubic IP addresses or public URLs. In this case there is no need to access the data "through" the customer's private network because the data is available in the public network with the correct connect string and access credentials. If you have any questions about specific data sources NOT supported by Incorta's Data Agent, you can contact your Customer Support Manager, or you Sales Account Manager, and they will be able to direct you to the correct resource to answer your question.
Related Material
Here are some links to assist you in getting started with Incorta:
