V11 SP8
Loading...

Connecting to HTTP Responses with Data Cube

You can use Data Cube to connect to the data in an HTTP response from a RESTful API service.

Before You Begin

  • You must be able to log in to the Web Console to view Data Cube. See Accessing the Web Console.
  • Only users assigned a role with the Data Connectors permission at the MediaAgent level can access Data Cube in the Analytics section of the Web Console. The associated MediaAgent must have been configured with Analytics Engine for Data Cube.
  • You need to obtain the following information to configure the HTTP connection:
    • The type of HTTP request that you want to send, either GET or POST.
    • Any headers, such as security credentials, or other raw data that must be sent with the request to receive the proper HTTP response.
    • The parameters and values for paginated sets of data in the HTTP response.
  • Note: Only JSON responses are supported.

Procedure

  1. In a Web browser, log in to the Web Console and then click Analytics.
  2. In the left navigation next to HTTP, click Add New.
  3. On the New Data Source (HTTP) page, configure the source as follows:
    1. Under Data Source Name:
      • Click the Analytics Engine list and select the Analytics Engine on which to store the crawling data.
      • In Data Source Name, enter a name for the data source. The name can only contain alphanumeric characters, underscores, and dashes.
      • In Data Source Description, enter a description for the data source.
      • Click Next to proceed to the next section.
    2. Under URL Options:
      • In URL, enter the URL to the RESTful API service to which you want to connect.
      • Click the Request Type list and select the type of HTTP request that you want to send.
      • To send header information with the HTTP request, next to Header enter the data in the Key and Value boxes.

        Note: If you need to add additional headers, click the plus icon. To remove a header row, click the trash icon.

      • Click Next to proceed to the next section.
    3. Under Preview:
      • Click the values in the columns to expand and preview the response data.

        Note: By default, all of the records that appear in the HTTP response preview are included in the data connection. To exclude data from crawling, clear the green check mark icon next to the name of the data field. The data field and any records that are nested within the field will be excluded from crawling.

      • Click Next to proceed to the next section.
    4. Under Edit Field Information:
      • To edit the names of the data fields, click the values under Field Name and make your changes.
      • To edit the data type of a data field, click the list under Field Type and select the new data type.
      • Click Next to proceed to the next section.
    5. Under Pagination:

      If the URL for the HTTP request contains pagination parameters, enable Pagination and proceed as follows:

      1. In Pagination Parameter, enter the parameters as follows:
        • Name: Enter the URL parameter that determines which page of the total data set to return in the response.
        • Value: Enter the value of the starting page position.
      2. In Parameter that specifies the length of each page, enter the parameters as follows:
        • Name: Enter the URL parameter that determines the number of records to return for each page in the response.
        • Value: Enter the value of the number of records to return for each page.
      3. In Step pagination parameter by, select the type of pagination used by the API as follows:
        • Offset: Select to increase the pagination parameter by the number of records returned on each page.
        • Page: Select to increase the pagination parameter by one.
      4. In Crawl Options, select the option for retrieving paginated records as follows:
        • By number of iterations: Select to return a specific number of pages from the response.
        • By number of records to crawl: Select to return a specific number of items from the response.
        • Enter the value for the iterations or number of records that you want to return in the response.
      5. Select Start Crawling Now to start crawling the data source after the data source is saved.
    6. When finished, click Submit.