You can use Data Cube to connect to the data in an HTTP response from a RESTful API service. After the data source is added, you can restructure the data. For example, you can combine data from two or more data sources into a single data source. For information about the available data restructuring options, see Restructuring Data in Data Cube.
Important
Only JSON formatted responses are supported.
Before You Begin
You need to obtain the following information to configure the HTTP connection:
-
The HTTP method for the request that you want to send, either GET or POST.
-
Request headers (such as security credentials).
-
Parameters and values for paginated sets of data in the HTTP response.
Procedure
-
From the Data Cube dashboard, next to HTTP, click Add New.
Tip
Alternatively, click HTTP to open the Data Sources (HTTP) page, and then click Add HTTP in the upper-right corner.
The New Data Source (HTTP) page appears.
-
In the Data Source Name section, enter the following information:
-
Field Name
Description
Steps
Index Server
The Index Server entity that you want to use for the data source.
The Index Server list is populated by the Index Servers configured for Data Cube in your CommCell Console.
Click the Index Server list and select an Index Server.
Data Source Name
The name of the data source as it will appear in the Data Cube dashboard.
-
Enter a name for the data source.
Note: Only alphanumeric, dash, and underscore characters are supported.
Data Source Description
A short description of the data source that is visible to any user who can view the data source in Data Cube.
- Optional: Enter a description for the data source.
-
-
Click Next.
-
In the URL Options section, enter the following information:
Field Name
Description
Steps
URL
Select an HTTP method and enter the URL to the RESTful API service that you want to connect to. The supported HTTP methods are as follows:
-
GET
-
POST
-
Next to the URL box, click the list and select an HTTP method for the request.
-
Enter the URL for the RESTful API service that you want to use.
Headers
Send additional parameters in the request header.
-
To send header information with the HTTP request, next to Header enter the data in the Key and Value boxes.
Note
If you need to add additional headers, click the plus (+) icon. To remove a header row, click the trash icon.
Authorization
Specify the authentication protocol that you want to include in the request, if any. The supported authentication types are as follows:
-
No Authentication
-
Basic Authentication
-
OAuth 1.0
-
OAuth 2.0
-
If the data source requires authentication, click the list and select an authentication protocol.
-
Complete the required authentication information for the protocol that you selected.
-
-
Click Next.
-
Optional: In the Preview section, you can perform the following functions:
-
For hierarchical data, click [object] to expand and preview the object that you want to preview.
-
To exclude data from crawling, clear the green check box next to the name of the data field. The data field and any records that are nested within the field will be excluded from crawling.
-
-
Click Next.
-
Optional: Under Edit Field Information, configure the response data as follows:
Field Name
Description
Steps
Field Name
Use to change the name of the data field as it will be stored in Data Cube.
- To rename a data field for a row, click the field name and enter a different name.
Field Type
Use to modify the type of data stored in field in Data Cube. The following data types are supported:
-
Boolean
-
Date
-
Double
-
Float
-
Integer
-
Long
-
Long string
-
Lowercase string
-
String
-
Text
-
UNIX timestamp
- To change the data type for a row, click data type, click the list that appears, and then select a different data type.
JSON Path
The URL of the data field.
None.
-
Click Next.
-
In the Pagination section, enter the following information:
Field Name
Description
Steps
Paginate
Use to configure pagination parameters for the data source.
- Click the slider to enable pagination.
Pagination Parameter
Use to specify the parameters in the URL that correspond to pagination. This option only appears if Paginate is enabled.
-
If Paginate is selected, enter the parameters as follows:
-
Name: Enter the URL parameter that determines which page of the total data set to return in the response.
-
Value: Enter the value of the starting page position.
-
Parameter that specifies the length of each page
Use to specify the parameters in the URL that correspond to the number of items to return per page in the response. This option only appears if Paginate is enabled.
-
If Paginate is selected, enter the parameters as follows:
-
Name: Enter the URL parameter that determines the number of records to return for each page in the response.
-
Value: Enter the value of the number of records to return for each page.
-
Step pagination parameter by
Use to configure the pagination method used by the RESTful API.
-
If Paginate is selected, select the type of pagination used by the API as follows:
-
Offset: Select to increase the pagination parameter by the number of records returned on each page.
-
Page: Select to increase the pagination parameter by one.
-
Crawl options
Use to configure how many pages or items to crawl for the data source.
-
If Paginate is selected, select the option for retrieving paginated records as follows:
-
To return a specific number of pages from the response, select By number of iterations and then enter a number in the box.
-
To return a specific number of items from the response, select By number of records to crawl and then click the list and select the data field that corresponds to the total number of records.
-
-
Click Next.
-
In the Advanced Options sections, enter the following information:
-
Field Name
Description
Steps
Primary Key
When incremental crawl is enabled, the primary key is used to uniquely identify items in the data source.
- If incremental crawling is enabled, click the list and select a data field that is a unique primary key for the data source.
Incremental Crawl
Select this option to crawl only updated values in the data source.
- Optional: Click the slider to enable incremental crawling.
Incremental Query
When incremental crawl is enabled, the filter query is used to determine the data source fields that have changed since the previous crawl.
Note
The incremental query should not be encoded.
- If incremental crawling is enabled, enter the query.
-
Click Submit.
The data source name Configuration page appears.