5. Run the AI Fabric Application Installer

Run the application installer. This will install AI Fabric and Out of the Box models in case of airgapped installations.

Accessing admin console

Navigate to the admin console, this will always be <machine-ip>:8800. This is where the configuration for the application installer will take place. On navigating to that address, the below page is displayed.

Click Continue to Setup. The below window is displayed.

Configure DNS (optional)

If the Linux machine has DNS configured and you with to use your own certificate when accessing AI Fabric app based on a fully qualified domain name, this configuration screen will allow you to do so.

Enter the hostname and upload both the private key and the SSL certificate. Make sure that the domain is reachable from the network where you want to access AI Fabric over the ports 8800, 31443, 31390. In the case of domain certificates, please make sure the certificate bundle file includes all the chain elements in this specific order: root, intermediate and server certificate. Also, if the public key certificate was not issued with a public RSA key of at least 2048-bit size, the validation will fail.

The format of the certificate is a .pem file, and the private key is a .key file. The files can be obtained from the .pfx certificate.

Logging in to the Admin Console

After configuring the DNS in the step above (or clicking **Skip & Continue" to skip that step), the following page is displayed:

Enter your password from Step 4: Run the AI Fabric Infrastructure Installer. After successfully logging in, you will be prompted to upload a license file:

Upload the yaml file that has your AI Fabric license (contact your AI Fabric representative if you do not have an AI Fabric license).

Choose installation type

Airgapped Installation

If you are following the airgapped installation, in this step you must upload the installation bundle.

Click on Choose a bundle to upload button and select aif_services file extracted on previous step.

Online Installation

If you are following the online installation, click on the link at the bottom of the page that states download AI Fabric from the internet

Configure the installer

Next step is to configure installer using following screen:

The fields in this page need to be completed. Please refer to the explanations below

Ingress

Host (IP or FQDN) IP of the Linux machine. If you configured DNS for this machine and completed the step Configure DNS above then enter the fully qualified domain name of this machine.

Single Database Vs Multiple Databases

Just select right option according to what you did in step 2.

SQL

• Host Enter the IP address of the SQL database where you created the databases in Step 2. SQL Configuration.
• Username Enter the username for the SQL database owner
• Password Enter the password for the SQL database.
  Note: If you are using Azure databases please make sure to use FQDN to connect to the databases. It can be public or private domain name (as long as DNS is configured) but it won't work if you are using the private IP is documented here: https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-dns

Orchestrator

Endpoint Enter the Orchestrator address with port 443. Do not include http, https. The Orchestrator endpoint must be the domain name <domain-name>:443.

Important: Make sure to include the port 443 after the domain-name. Not including the port is a common source of user error during the install.

See the examples below for pitfalls to avoid.

Orchestrator endpoint entered	Correct
aifabric.orchestrator.cloudapp.azure.com:443	✓
https://aifabric.orchestrator.cloudapp.azure.com	✗
https://23.96.154:443	✗
23.96.154:443	✗

Identity Server

This is only necessary if UiPath Identity Server is different from Orchestrator Endpoint.

Identity Server Access Token

This is the identity token from Orchestrator. To generate this token navigate to <orchestrator-address>/identity in a browser. Make sure you are logged in to the host tenant (as opposed to the "default" tenant). The below page is displayed:

If you see a page that does not have the left navigation as above, you are likely logged in to the default tenant. Make sure you log in as the host tenant.

Now click Installation Access Token, click Generate Token and use the two card icon to copy this token to your clipboard.

Paste this token into the field Identity access token.

Important: Note that this token expires in an hour. The configuration should be deployed immediately after you have generated the token to reduce the risk of issues. If you have to edit the configuration and re-deploy, be sure to generate a new token.

Preflight checks

Click Continue in the configuration page. You will be navigated to a page with title Preflight checks. If all your preflight checks pass, you will see a page like this:

The only preflight checks that can be cautiously ignored is Total Memory. Conservatively, the machine should have at least 52GB RAM, to see how provisioning less memory limits the capacities of AI Fabric see the hardware requirements page.

Troubleshooting Preflight Checks

If you do not see a green checkmark on the preflight checks use this guide to fix your errors.

Preflight Check
Orchestrator Check	Verifies that: We can connect to orchestrator domain on the path /api/auth Verifies Web.config is correctly set. Solution: Make sure the network rules are set so the linux machine can connect to Orchestrator. Verify that Web.config is correctly set in 3. Configure Orchestrator.
Identity Server Check	Verifies that we can connect to identity server on path /identity. Solution: Make sure the network rules are set so the linux machine can connect to Identity Server (if different from Orchestrator).
Identity Server Access Token Validation	Verifies that the access token is valid. Solution: If you pass the Identity Server Check, make sure that you have a fresh token as the token validity is 1 hour.
Orchestrator and Identity Server checks fail AND you do not have an external DNS	If you do not have an external DNS through which the Orchestrator domain name or the Identity Server domain name can be resolved, you need an extra application of a configuration file. See Advanced Troubleshooting - DNS Resolution
DB Checks	Verifies that: SQL Server connection and credentials are valid. Checks DB exists Checks user is assigned db_owner role for each DB. Solution: Make sure SQL Server is allowing incoming connections on its port and that SQL Server Authentication is enabled. The DB creation and proper role creation is handled automatically by the script in 2. Configure Database.
All checks below Disk Space fail.	If you have verified the network/firewall rules for Orchestrator and SQL Server but are still seeing all the checks related to connections fail it may be due to a more nuanced network configuration issue called IP Masquerading. This is usually caused when the linux machine and Orchestrator/SQL Server are on different subnets. If this happens, run: sysctl -w net.ipv4.ip_forward=1 iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE On the linux machine, and retry the preflight checks.

If you are unable to resolve preflight check failure please contact support and send them a Support Bundle. See Advanced Troubleshooting - Support Bundle for instructions on how to create one.

Running the application installer

Once all the preflight checks are passing, click Continue. You will see the following page:

This page means that the AI Fabric application installer has started. If all configurations where set correctly, this should execute within 20-30 mins. See below to output the log that will show whether the installation was successful.

Troubleshooting the Application Installer

Tip:

At a high-level troubleshooting the application install by yourself (note you can also just send UiPath technical support your Support Bundle, see below) involves these steps:

• Looking at the provision log to determine what happened. Run kubectl logs -f provision-* to see the latest provision log.
• Fix/Edit your configuration after you learn of the error from the logs.
• Re-trigger a provision by saving the configuration, clicking Go to new new version and then clicking Deploy

Running Logs

To see running logs, you can use the kubernetes commadline interface since the application installer runs on top of kubernetes.

To run any kubernetes command, reload your bash shell (once per terminal session) by running bash -l.

Every time a configuration is changed and deployed (for the first configuration the deploy is done automatically, for subsequent one you must save the configuration and click Deploy), a new job will be executed to install the application.

To see the running logs on the linux machine run:

bash -l
kubectl get pods

you will see something like this:

aif-admin@aifabric-onprem-int0:~$ kubectl get pods
NAME                                  READY   STATUS             RESTARTS   AGE
...
...
provision-rmvfg                       0/1     Running            0          1m

The output shows a name formatted like provision-<identifier>. To see the running application installer do:

kubectl logs -f provision-rmvfg

Where you replace 'rmvgf' with whatever identifier is shown in your terminal (alternatively just type kubectl logs -f provision and hit TAB, this will autocomplete the identifier).

This will show running logs (if the process has not ended) as well as the logs when the process has succeeded or terminated. Most, if not all users can troubleshoot without having to use anything other than the command above.

Successful Install

A successful install will show the screenshot below 15-20 minutes after passing the preflight checks:

If you see this screen, you can proceed to the step 6. Verify Installation .

Known Issues

Contingent on triggering an installation only after you have passed the preflight checks, there is only one known issue that can happen in the AI Fabric installer. If this is the case, you will see the following output from the log:

...
Starting ai-helper deployment ...Release "ai-helper" does not exist. 
Installing it now.
Error: etcdserver: request timed outHelm installation failed for ai-helper in namespace aifabric. 
Exiting !!!onebox provisioning failed. 
Exiting !!!

This is an intermittent etcdserver issue. Retry the installation by going to the admin console on <ip>:8800, clicking on Config, pasting a new Identity server token, hitting Continue to new version, and finally clicking Deploy.

This issue should resolve itself with up to 5 installation attempts. If you face the same issue (Error: etcdserver: request timed out) and retrying does not resolve the issue please contact support and create a support bundle.

Install OOB models (airgapped only)

For airgapped installation you need to manually download OOB models and then "install" them on your AI Fabric machine to be able to use them. For each model that you want to add you will have a tar file that you need to move to your AI Fabric machine. Once on AI Fabric machine just run following commands for each file:

tar -zxvf formextractor-1.tar.gz
cd formextractor
nohup sudo ./setup.sh > formextractor.out 2>&1

Logs will be accessible in formextract.out file. We recommend using nohup for this command because installation can take up to an hour and this would avoid any issue due to losing connection to the machine.

Advanced troubleshooting

Creating a Support Bundle

To create a support bundle, navigate to the admin UI (<machine-ip>:8800) and click on Troubleshoot on the top navigation bar. Click the button to generate a new support bundle, and then download that bundle. When you contact support, include in the ticket that file (support-bundle.tar.gz).

DNS Resolution

As mentioned above, this issue will surface during preflight checks. As a reminder, this issue occurs when there is no external DNS that will resolve the Orchestrator domain or the Identity Server domain.

To address this issue (note if you are an experienced linux user, the issue cannotbe simply solved by adding entries to /etc/hosts), we will need to edit cluster's configmap so the cluster is aware of this DNS. To do do so you need to run the following command:

kubectl -n kube-system edit cm coredns

It will open a vi editor with file looking as follow

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
        }
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2020-11-30T12:25:28Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "17667708"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 2bde7049-eda6-46eb-b523-beb8c421085f

A new section hosts needs to be added after loadbalance in Corefile section (line 19), you can list there as many dns you want to and add fallthrough at the end:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
        hosts example.hosts orchestrator-dns.com {
          1.2.3.4 example.hosts
          5.6.7.8 orchestrator-dns.com
          fallthrough
        }
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2020-11-30T12:25:28Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "17667708"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 2bde7049-eda6-46eb-b523-beb8c421085f

After that the DNS will be configured and working inside your cluster.

Note: Make sure that after your edit coredns is a proper yaml file and don't use tab to indent but spaces only.