- Release Notes
- Requirements
- Installation
- Getting Started
- Projects
- Datasets
- ML Packages
- Pipelines
- ML Skills
- ML Logs
- Document Understanding in AI Fabric
- Basic Troubleshooting Guide
Support
This page details where to find the relevant information to report bugs or troubleshoot problems both at install time and while using the products.
We have a diagnostic tool to help you check AI Fabric health and identify issues on your installation. To execute this diagnostic, simply connect to your AI Fabric Host and run the following command:
bash <(curl https://raw.githubusercontent.com/UiPath/ai-customer-scripts/master/platform/generate-report.sh)
bash <(curl https://raw.githubusercontent.com/UiPath/ai-customer-scripts/master/platform/generate-report.sh)
For airgapped, if you can't access above url from the machine itself, create a new file generate-report.sh and copy paste above file in it then execute command:
bash generate-report.sh
bash generate-report.sh
This will generate a file aifabric-diagnostics-latest.log (example below) with report on status of the different AI Fabric services, if the right ports are actually open on AI Fabric machine, test to upload a file and ML Package, it will show info on your certificates and GPU status.
Fetching Core Services Status
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 867 0 867 0 0 3454 0 --:--:-- --:--:-- --:--:-- 3468
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 862 0 862 0 0 3747 0 --:--:-- --:--:-- --:--:-- 3747
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 470 0 470 0 0 13055 0 --:--:-- --:--:-- --:--:-- 13055
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 569 0 569 0 0 14589 0 --:--:-- --:--:-- --:--:-- 14973
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 444 0 444 0 0 12333 0 --:--:-- --:--:-- --:--:-- 12333
Starting Orchestrator Connection Check
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 492 100 492 0 0 5857 0 --:--:-- --:--:-- --:--:-- 5927
Successfully received response from orchestrator: HTTP/2 200
cache-control: no-store, must-revalidate, no-cache, max-age=0
content-type: application/json; charset=utf-8
x-correlation-id: 655adc9a-df94-47b3-8a35-40ffed513acc
api-supported-versions: 10.0
x-content-type-options: nosniff
x-frame-options: DENY
strict-transport-security: max-age=31536000; includeSubDomains
server:
date: Tue, 08 Dec 2020 14:28:24 GMT
content-length: 492
{"keys":[{"alg":"RS256","e":"AQAB","kid":"BA16
...
Checking aifabric ports availability in the Cluster
aif.snvenkat1.xyz (52.178.221.160:31390) open
aif.snvenkat1.xyz (52.178.221.160:31443) open
aif.snvenkat1.xyz (52.178.221.160:6443) open
Open
Fetching Certificate Details from Orchstrator and AIFabric
depth=0 CN = aifabricqaorchtest.northeurope.cloudapp.azure.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = aifabricqaorchtest.northeurope.cloudapp.azure.com
verify error:num=21:unable to verify the first certificate
verify return:1
DONE
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = AT, O = ZeroSSL, CN = ZeroSSL RSA Domain Secure Site CA
verify return:1
depth=0 CN = aif.snvenkat1.xyz
verify return:1
DONE
Check if GPU is installed in the Cluster!!
Node: dm-onebox
GPU Capacity : 1
GPU Node Found!
-----Analysis Start
Core Services Status:
Deployer : "UP"
Trainer : "UP"
PkgManagaer : "UP"
Helper : "UP"
AppManager : "UP"
RabbitMQ : "UP"
AIFabric Ports Status:
AIFabric Port (31390) : Open
Storage Port (31443) : Open
Kubernetes Port (6443) : Open
Databases Health:
Deployer DB : "UP"
Trainer DB : "UP"
Helper DB : "UP"
PkgManager DB : "UP"
AppManager DB : "UP"
DockerRegistry Health:
Deployer Registry : "UP"
Trainer Registry : "UP"
Orchestrator Connection Status:
Orchestrator connection is Healthy!
Certificates Check:
Your orchestrator certificate is valid for following IP/Hosts, please make sure it matches the host/IP you are using in AIFabric Setup.
DNS:aifabricqaorchtest.northeurope.cloudapp.azure.com
Expiry Date of the Orchestrator Certificate : Jul 22 12:11:30 2021 GMT
Your AIFabric Ingress Host certificate is valid for following IP/Hosts, please make sure it matches the host/IP you are using for AIFabric Setup in Orchestrator
DNS:aif.snvenkat1.xyz
Expiry Date of the AIFabric Certificate : Dec 24 23:59:59 2020 GMT
Storage Checks:
1. Object Storage - File Upload Test Successful
2. Object Storage - File Deletion Test Successful
GPU Drivers Check:
GPU Available and Working Fine. Total no of nodes with GPU - 1
-----Analysis End
**Report Generated on Tue Dec 8 14:28:31 UTC 2020
Fetching Core Services Status
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 867 0 867 0 0 3454 0 --:--:-- --:--:-- --:--:-- 3468
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 862 0 862 0 0 3747 0 --:--:-- --:--:-- --:--:-- 3747
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 470 0 470 0 0 13055 0 --:--:-- --:--:-- --:--:-- 13055
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 569 0 569 0 0 14589 0 --:--:-- --:--:-- --:--:-- 14973
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 444 0 444 0 0 12333 0 --:--:-- --:--:-- --:--:-- 12333
Starting Orchestrator Connection Check
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 492 100 492 0 0 5857 0 --:--:-- --:--:-- --:--:-- 5927
Successfully received response from orchestrator: HTTP/2 200
cache-control: no-store, must-revalidate, no-cache, max-age=0
content-type: application/json; charset=utf-8
x-correlation-id: 655adc9a-df94-47b3-8a35-40ffed513acc
api-supported-versions: 10.0
x-content-type-options: nosniff
x-frame-options: DENY
strict-transport-security: max-age=31536000; includeSubDomains
server:
date: Tue, 08 Dec 2020 14:28:24 GMT
content-length: 492
{"keys":[{"alg":"RS256","e":"AQAB","kid":"BA16
...
Checking aifabric ports availability in the Cluster
aif.snvenkat1.xyz (52.178.221.160:31390) open
aif.snvenkat1.xyz (52.178.221.160:31443) open
aif.snvenkat1.xyz (52.178.221.160:6443) open
Open
Fetching Certificate Details from Orchstrator and AIFabric
depth=0 CN = aifabricqaorchtest.northeurope.cloudapp.azure.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = aifabricqaorchtest.northeurope.cloudapp.azure.com
verify error:num=21:unable to verify the first certificate
verify return:1
DONE
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = AT, O = ZeroSSL, CN = ZeroSSL RSA Domain Secure Site CA
verify return:1
depth=0 CN = aif.snvenkat1.xyz
verify return:1
DONE
Check if GPU is installed in the Cluster!!
Node: dm-onebox
GPU Capacity : 1
GPU Node Found!
-----Analysis Start
Core Services Status:
Deployer : "UP"
Trainer : "UP"
PkgManagaer : "UP"
Helper : "UP"
AppManager : "UP"
RabbitMQ : "UP"
AIFabric Ports Status:
AIFabric Port (31390) : Open
Storage Port (31443) : Open
Kubernetes Port (6443) : Open
Databases Health:
Deployer DB : "UP"
Trainer DB : "UP"
Helper DB : "UP"
PkgManager DB : "UP"
AppManager DB : "UP"
DockerRegistry Health:
Deployer Registry : "UP"
Trainer Registry : "UP"
Orchestrator Connection Status:
Orchestrator connection is Healthy!
Certificates Check:
Your orchestrator certificate is valid for following IP/Hosts, please make sure it matches the host/IP you are using in AIFabric Setup.
DNS:aifabricqaorchtest.northeurope.cloudapp.azure.com
Expiry Date of the Orchestrator Certificate : Jul 22 12:11:30 2021 GMT
Your AIFabric Ingress Host certificate is valid for following IP/Hosts, please make sure it matches the host/IP you are using for AIFabric Setup in Orchestrator
DNS:aif.snvenkat1.xyz
Expiry Date of the AIFabric Certificate : Dec 24 23:59:59 2020 GMT
Storage Checks:
1. Object Storage - File Upload Test Successful
2. Object Storage - File Deletion Test Successful
GPU Drivers Check:
GPU Available and Working Fine. Total no of nodes with GPU - 1
-----Analysis End
**Report Generated on Tue Dec 8 14:28:31 UTC 2020
<machine-ip>:8800
) and click on Troubleshoot on the top navigation bar. Click on the button to generate a new support bundle, and then download that bundle.
Contact UiPath support, they will be able to resolve your issue with the provided bundle.
If, for some reason, creating a support bundle from the admin console does not work, use the command below to create a support bundle from the linux terminal:
curl https://krew.sh/support-bundle | bash
kubectl support-bundle https://kots.io
curl https://krew.sh/support-bundle | bash
kubectl support-bundle https://kots.io
Create file specs.yaml on your machine as below:
apiVersion: troubleshoot.replicated.com/v1beta1
kind: Collector
metadata:
name: collector-sample
spec:
collectors:
- clusterInfo: {}
- clusterResources: {}
- ceph: {}
- exec:
args:
- "-U"
- kotsadm
collectorName: kotsadm-postgres-db
command:
- pg_dump
containerName: kotsadm-postgres
name: kots/admin_console
selector:
- app=kotsadm-postgres
timeout: 10s
- logs:
collectorName: kotsadm-postgres-db
name: kots/admin_console
selector:
- app=kotsadm-postgres
- logs:
collectorName: kotsadm-api
name: kots/admin_console
selector:
- app=kotsadm-api
- logs:
collectorName: kotsadm-operator
name: kots/admin_console
selector:
- app=kotsadm-operator
- logs:
collectorName: kotsadm
name: kots/admin_console
selector:
- app=kotsadm
- logs:
collectorName: kurl-proxy-kotsadm
name: kots/admin_console
selector:
- app=kurl-proxy-kotsadm
- secret:
collectorName: kotsadm-replicated-registry
includeValue: false
key: .dockerconfigjson
name: kotsadm-replicated-registry
- logs:
collectorName: rook-ceph-agent
selector:
- app=rook-ceph-agent
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-ceph-mgr
selector:
- app=rook-ceph-mgr
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-ceph-mon
selector:
- app=rook-ceph-mon
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-ceph-operator
selector:
- app=rook-ceph-operator
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-ceph-osd
selector:
- app=rook-ceph-osd
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-ceph-osd-prepare
selector:
- app=rook-ceph-osd-prepare
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-ceph-rgw
selector:
- app=rook-ceph-rgw
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-discover
selector:
- app=rook-discover
namespace: rook-ceph
name: kots/rook
apiVersion: troubleshoot.replicated.com/v1beta1
kind: Collector
metadata:
name: collector-sample
spec:
collectors:
- clusterInfo: {}
- clusterResources: {}
- ceph: {}
- exec:
args:
- "-U"
- kotsadm
collectorName: kotsadm-postgres-db
command:
- pg_dump
containerName: kotsadm-postgres
name: kots/admin_console
selector:
- app=kotsadm-postgres
timeout: 10s
- logs:
collectorName: kotsadm-postgres-db
name: kots/admin_console
selector:
- app=kotsadm-postgres
- logs:
collectorName: kotsadm-api
name: kots/admin_console
selector:
- app=kotsadm-api
- logs:
collectorName: kotsadm-operator
name: kots/admin_console
selector:
- app=kotsadm-operator
- logs:
collectorName: kotsadm
name: kots/admin_console
selector:
- app=kotsadm
- logs:
collectorName: kurl-proxy-kotsadm
name: kots/admin_console
selector:
- app=kurl-proxy-kotsadm
- secret:
collectorName: kotsadm-replicated-registry
includeValue: false
key: .dockerconfigjson
name: kotsadm-replicated-registry
- logs:
collectorName: rook-ceph-agent
selector:
- app=rook-ceph-agent
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-ceph-mgr
selector:
- app=rook-ceph-mgr
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-ceph-mon
selector:
- app=rook-ceph-mon
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-ceph-operator
selector:
- app=rook-ceph-operator
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-ceph-osd
selector:
- app=rook-ceph-osd
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-ceph-osd-prepare
selector:
- app=rook-ceph-osd-prepare
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-ceph-rgw
selector:
- app=rook-ceph-rgw
namespace: rook-ceph
name: kots/rook
- logs:
collectorName: rook-discover
selector:
- app=rook-discover
namespace: rook-ceph
name: kots/rook
Then run the following command:
kubectl support-bundle /path/to/spec.yaml
kubectl support-bundle /path/to/spec.yaml
Contact UiPath support, they will be able to resolve your issue with the provided bundle.
When reporting Data Manager issues, include the generated logs. To retrieve them, do the following:
- Click the question mark on the top-right corner in Data Manager. The Data Manager help menu is displayed.
- In the Error Reporting section, click Collect recent logs for error reporting. The Recent Logs window is displayed.