Skip to main content

Ingesting data using the API

This guide will walk you through the process of importing and processing datasets in Walr.

Prerequisites​

  • A Walr project created where you want your dataset to exist.
  • A valid bearer token for your customer in the request authorization header.

Steps​

1. Import Metadata​

Make a POST request to the following endpoint:

https://api.walr.com/an/ingestion/projects/{projectId}/import/meta

The body of the request should include:

  • file: An XML file (please see Triple-S Specification for more details)
  • name: Display name of the dataset that you want it to be called in the platform

If successful, a 200 OK response will be returned with a metaId:

{
  "metaId": "3975632f-0020-4bd4-af80-6ac6df797e4e",
  "errorMessages": null
}

If the request fails, a 400 BadRequest response will be returned with a list of errors in the error messages to help fix the XML file.

2. Import Case Data​

Make a POST request to the following endpoint:

https://api.walr.com/an/ingestion/projects/{projectId}/import/meta/{metaId}/case

The body of the request should include a file: a .csv or .zip file of a zipped csv file.

ℹ️ The file should be at most 2 GB large in total.

If successful, you will receive a response similar to this:

{
  "totalFilesReceived": 1,
  "totalSizeUploaded": "62MB",
  "uploadedFiles": [
    {
      "uploadedFileName": "my_csv_file.csv",
      "caseId": "864da4a8-293c-4ee8-934e-d884a73f2e6b",
      "lineCount": 382734
    }
  ],
  "notUploadedFiles": []
}

3. Check the Status of the Uploaded File​

Use the caseId from each file to continuously get the status of the uploaded file using this endpoint:

GET https://api.walr.com/an/ingestion/projects/{projectId}/import/meta/{metaId}/cases/{caseId}/status

When the statusCode is 4, the file has been properly validated and ready for processing:

{
  "status": "Validated",
  "statusCode": 4
}

4. Process the Case Data​

To process the case data, use this endpoint:

POST https://api.walr.com/an/ingestion/projects/{projectId}/import/meta/{metaId}/cases/{caseId}/process

This request does not require a body. If successful, it will return a 202 accepted if the background job has started.

5. Follow the Process of the Dataset​

Keep following the process of the dataset by querying the meta logs:

GET https://api.walr.com/an/ingestion/projects/{projectId}/import/meta/{metaId}/logs

This will provide you with the status and progress of your dataset processing.

The full list of statues are described below (for both meta and case).

{
    NotSet,                  // 0 - No status
    Uploaded,                // 1 - File uploaded
    Validating,              // 2 - File validating
    Invalid,                 // 3 - File invalid
    Validated,               // 4 - File valid
    ValidatedWithWarnings,   // 5 - File valid with warnings
    Processing,              // 6 - File processing
    Completed,               // 7 - Processing completed
    Failed,                  // 8 - Processing failed
    Deleted,                 // 9 - File deleted
    HereBeDragons            // 10 - Uncharted territory
}