Troubleshooting Halo Cloud

This page details how troubleshoot various issues with Halo Cloud, including troubleshooting steps for general connection issues and product-specific issues.

As a general rule, when receiving frequent or persistent errors from a practice, we recommend applying a backoff strategy. For example, by doubling the time between queries each time a query fails. This allows you to continue retrying the query with the minimum overhead to all systems.

Some errors could also be avoided by pausing queries for a given time period based on the error message. For example, polling for appointments for a given GP may error if the GP is away for the day, in which case we recommend pausing any queries for that GP. Halo Cloud will pass on database error messages where possible so that integrators can adjust their product's logic for such cases.

Reducing unnecessary queries doesn't just reduce the number of queries you're charged for, it also reduces load on the practice server and on Halo Connect's systems.

Known issues

SQL Passthrough

Failure to convert parameter values

Impact

When sending a query with command of type storedProcedure, if it contains parameters of type VarBinary or NVarChar, the query returns a 400 with the error "Failed to convert parameter value from a String to a Byte[]. --> Invalid cast from 'System.String' to 'System.Byte[]'. Please correct your request and try again."

Cause

Currently, query parameters specified in command.parameters are converted to SQL parameters using implicit conversion. This fails for large values.

For VarBinary: Implicit conversion fails if the byte array exceeds 8,000 bytes.
For NVarChar: Implicit conversion fails if the string exceeds 4,000 characters.

Workaround

To handle longer strings and byte arrays, explicit object setting is required. Use custom SQL to declare the variables required for the stored procedure with the correct types, then execute the stored procedure using those variables.

The command's type should be "text", not "storedProcedure".

When using a stored procedure, please make sure to include all mandatory fields, and to explicitly assign values to the variables by name if not all fields are used.

Example:

{
    "command": {
        "text": "
        DECLARE @Content VarBinary(max);
        SET @Content = 0xEFBBBF54686973206973206120746573742066696C652E
        execute @returnValue = sp_InsertSampleDocument @Content=@Content
        SELECT @ReturnValue
        ",
        "executionMode": "reader",
        "type": "text"
    }
}

FHIR API

The following known issues should be fixed in a future release. Please contact Support if any of these issues negatively impact your ability to use the FHIR API, so we can ensure any fixes accommodate your use case.

Issue	Impact	Status
Incorrect `_include` and `_revinclude` pagination URL	The `link.url` field in Bundles that use `_include` or `_revinclude` may not include the `_include` or `_revinclude` parameter.	Fixed in upcoming release
Only active records are returned	The FHIR API does not return inactive records.	Fix scheduled
Timezone differences across different resources for `lastUpdated`	There may be slight differences in how timezones are calculated and displayed for the `lastUpdated` field for different resources.
Datetimes must be exact	Datetime-based search parameters only accept the formats `YYYY-MM-DD` and `YYYY-MM-DDThh:mm:ss`.

Troubleshooting

Query type differences

API	Query type	Key difference
SQL Passthrough	Immediate queries	Immediate queries are the simplest, as any errors are returned directly in the HTTP response.
SQL Passthrough	Async queries	Async queries errors are return in either the POST response or the query result, depending on if the error occurred before, during, or after query execution. Errors thrown before execution will generally be returned in the POST response. Errors thrown during or after execution will be sent to the cloud as the query result, to be fetched by the integrator.
SQL Passthrough	Registered queries	Same as async queries, registered query errors can be returned in the POST response or the query result. Errors returned in the POST response usually mean the query was not registered. Errors sent to the cloud as the query result indicate that the query was registered but errored during execution. Occasional errors may be due to intermittent issues on the practice server. Frequent errors imply something is wrong with the query or server.
FHIR API	FHIR queries	FHIR queries can return the same errors as immediate queries, plus an extra category. These are FHIR errors, represented by an OperationOutcome.

Error formats

Failed queries will return error data within the query response JSON using one of the following formats, using a 413 as an example:

New format

"errorDetails": {
    "errorType": "http",
    "errorCode": 413,
    "errorMessage": "Query was executed successfully but an error occurred preparing it to be uploaded, results were too large. Immediate query results beyond message size limit (8mb)"
},

Old format

"errorCode": 413,
"errorMessage": "Query was executed successfully but an error occurred preparing it to be uploaded, results were too large. Immediate query results beyond message size limit (8mb)"

If both exist, use errorDetails and its sub-fields over the root ones. The old format has been kept temporarily for backwards compatibility as we shift all queries to the new format.

errorType specifies whether the error is a HTTP error, Halo Link error, or specific to the database the PMS is running, such as MSSQL. This can help narrow down what caused the issue.

For FHIR queries, you may also receive an additional error type represented by FHIR standard OperationOutcome. For example:

OperationOutcome format

{
    "resourceType": "OperationOutcome",
    "issue": [
        {
        "severity": "fatal",
        "code": "exception",
        "details": {
            "coding": [
            {
                "system": "http://hl7.org/fhir/dotnet-api-operation-outcome",
                "code": "5003"
            }
            ],
            "text": "Unable to parse Resource URL"
        }
        }
    ]
}

Halo Link error codes

In addition to standard HTTP errors, integrators may see Halo Link errors, which are specific to how Halo Link operates. How these errors are returned depends on the query type:

For immediate and FHIR queries, Halo Link errors will be returned in the POST response as a 400 error.
For async and registered queries, Halo Link errors will be uploaded to the cloud to be fetched by the integrator. The POST response will be a 200 success, as these errors occur during or after query execution.

Error Code	Type	Exception	Description
0	HTTP	`ClientConnectionFailure`	`Client connection was unexpectedly closed`
1	Halo Link	`SqlExecutionFailed`	Various, including: `Error occured while establishing a connection to SQL Server`, `SQL query string cannot be null or void`
2	Halo Link	`PrepForUploadFailed`	Various, including: `Parameter conversion error (e.g. varchar to datetime)`, `Query executed successfully but an error occurred preparing it for upload`
3	Halo Link	`QueryOrphaned`	Async queries only. Query was orphaned during upload or execution
4	Halo Link	`QueryMissingProperties`	`Query missing properties`
5	Halo Link	`QueryUploadFailed`	Async queries only. Request failed with status code PreconditionFailed

HTTP error codes

The following is a non-exhaustive list of common error codes, and how to mitigate them.

200 success

A HTTP response of 200 means the POST succeeded. However, that means different things for different query types:

Immediate and FHIR queries: the query was successfully received, executed, and sent back to the integrator.
Async and registered queries: the query was successfully received and queued.

Async and registered queries may still error during execution or upload, in which case the error will be made available via the Get query result endpoints.

4xx client errors

4xx errors are usually caused by issues with the query being sent or something being misconfigured. Please follow the steps listed below to resolve the issue, and if it persists, feel free to contact Support.

Code	Exception	Cause	Action
400	`Login failed for user [userID]`	The practice has not authorised the integration in their PMS database.	Contact the practice to ensure the integration is enabled in the PMS software. Apply backoff strategy until resolved. Cease sending queries if the practice is a churned customer.
400	`BadRequest`	The database has returned an error that is directly related to the content of the SQL query.	Review SQL for errors. Check error message for further details, such as "Invalid column name".
401	`SubscriptionKeyInvalid`, `SubscriptionKeyNotFound`	Subscription key supplied by the integrator is invalid for the environment being queried.	Check subscription key and environment are correct.
401	N/A	The Halo GUID being queried is inactive or not available in this environment.	Check the environment is correct. Check the Halo GUID is correct. If not resolved, contact support.
403	`CallerIpNotAllowed`	The query was sent from an unrecognised IP address.	Contact Support.
403	`Forbidden`	Developer site is not allowed to access this resource. PMS developer licenses cannot be used with Halo Connect's production environment.	Switch Halo Link to `Stage`, or use a fully-licensed PMS.
403	N/A	The practice being queried is non-authoritative.	Re-fetch the practice's Halo GUID. If not resolved, contact Support.
404	`OperationNotFound`	The endpoint URI used was invalid or malformed.	Review the query URI. See the API reference for endpoint details.
406	`QueryNotAcceptable`	Query was executed successfully but an error occurred preparing it to be uploaded. Intermittent 406s may be caused by disruptions to the practice server.	If infrequent, retry query. If frequent or persistent, contact Support.
413	`QueryResultsTooLarge`	Immediate query results are larger than message size limit.	Change the query to reduce the result size or use an async query instead.
429	`RateLimitExceeded`	Too many requests sent too quickly according to the integrator's rate limit.	Reduce query frequency or contact us to discuss a higher limit.

5xx server errors

5xx errors are often momentary. We also proactively monitor practices for connectivity issues that last longer than a few minutes, and reach out to practices ourselves to resolve any issues.

However, if a 5xx error is persistent or frequent, please:

Apply backoff strategy sending queries until resolved.
Contact Support.

HTTP Code	Description
500	Generic error caused by various server-side issues.
503	Connectivity issue between Halo Link and Halo Cloud which results in the error `Site could not be contacted.` with no further information.
504	Gateway timeout which results in errors such as `Query took more than 30 seconds to return a result`, `Site cannot be contacted. Site is currently unreachable.`, `Site cannot be contacted. Site is offline.`