- API docs
- CLI
- Integration guides
- Blog
- How machines learn to understand words: a guide to embeddings in NLP
- Prompt-based learning with Transformers
- Efficient Transformers II: knowledge distillation & fine-tuning
- Efficient Transformers I: attention mechanisms
- Deep hierarchical unsupervised intent modelling: getting value without training data
- Fixing annotating bias with Communications Mining
- Active learning: better ML models in less time
- It's all in the numbers - assessing model performance with metrics
- Why model validation is important
- Comparing Communications Mining and Google AutoML for conversational data intelligence
Self-hosted EWS integration
The EWS appliance is delivered as a Docker image. The sections below explain how to configure and deploy the appliance.
The appliance expects a JSON config file to be present. This section explains the contents of the file. Refer to the Deployment section for instructions on how to make the config file available to the appliance.
You can authenticate with client secret or with client certificate.
The token grant flow used is the client credentials flow.
With client secret
{
"ews_endpoint": "https://outlook.office365.com/EWS/Exchange.asmx",
"auth_type": "oauth2",
"auth_oauth_authority": "https://login.microsoftonline.com/<tenant_id>/",
"auth_oauth_client_id": "<client_id>",
"auth_oauth_client_secret": "<client_secret>",
"access_type": "impersonation",
"mailboxes": {
"abc@example.com": {
"bucket": {
"owner": "project-name",
"name": "bucket-name"
},
"start_from": "bucket",
"start_timestamp": "2020-01-01T00:00:00+00:00"
},
"xyz@example.com": {
"bucket": {
"owner": "project-name",
"name": "bucket-name"
},
"start_from": "bucket",
"start_timestamp": "2020-01-01T00:00:00+00:00"
}
}
}
{
"ews_endpoint": "https://outlook.office365.com/EWS/Exchange.asmx",
"auth_type": "oauth2",
"auth_oauth_authority": "https://login.microsoftonline.com/<tenant_id>/",
"auth_oauth_client_id": "<client_id>",
"auth_oauth_client_secret": "<client_secret>",
"access_type": "impersonation",
"mailboxes": {
"abc@example.com": {
"bucket": {
"owner": "project-name",
"name": "bucket-name"
},
"start_from": "bucket",
"start_timestamp": "2020-01-01T00:00:00+00:00"
},
"xyz@example.com": {
"bucket": {
"owner": "project-name",
"name": "bucket-name"
},
"start_from": "bucket",
"start_timestamp": "2020-01-01T00:00:00+00:00"
}
}
}
With client certificate
{
"ews_endpoint": "https://outlook.office365.com/EWS/Exchange.asmx",
"auth_type": "oauth2",
"auth_oauth_authority": "https://login.microsoftonline.com/<tenant_id>/",
"auth_oauth_client_id": "<client_id>",
"auth_oauth_client_credential_private_key": "<private_key>",
"auth_oauth_client_credential_thumbprint": "<thumbprint>",
"access_type": "impersonation",
"mailboxes": {
"abc@example.com": {
"bucket": {
"owner": "project-name",
"name": "bucket-name"
},
"start_from": "bucket",
"start_timestamp": "2020-01-01T00:00:00+00:00"
},
"xyz@example.com": {
"bucket": {
"owner": "project-name",
"name": "bucket-name"
},
"start_from": "bucket",
"start_timestamp": "2020-01-01T00:00:00+00:00"
}
}
}
{
"ews_endpoint": "https://outlook.office365.com/EWS/Exchange.asmx",
"auth_type": "oauth2",
"auth_oauth_authority": "https://login.microsoftonline.com/<tenant_id>/",
"auth_oauth_client_id": "<client_id>",
"auth_oauth_client_credential_private_key": "<private_key>",
"auth_oauth_client_credential_thumbprint": "<thumbprint>",
"access_type": "impersonation",
"mailboxes": {
"abc@example.com": {
"bucket": {
"owner": "project-name",
"name": "bucket-name"
},
"start_from": "bucket",
"start_timestamp": "2020-01-01T00:00:00+00:00"
},
"xyz@example.com": {
"bucket": {
"owner": "project-name",
"name": "bucket-name"
},
"start_from": "bucket",
"start_timestamp": "2020-01-01T00:00:00+00:00"
}
}
}
{
"host": "https://exchange-server.example.com",
"port": 443,
"auth_type": "ntlm",
"auth_user": "ews-service-user@example.com",
"access_type": "delegate",
"mailboxes": {
"abc@example.com": {
"bucket": {
"owner": "project-name",
"name": "bucket-name"
},
"start_from": "bucket",
"start_timestamp": "2020-01-01T00:00:00+00:00"
},
"xyz@example.com": {
"bucket": {
"owner": "project-name",
"name": "bucket-name"
},
"start_from": "bucket",
"start_timestamp": "2020-01-01T00:00:00+00:00"
}
}
}
{
"host": "https://exchange-server.example.com",
"port": 443,
"auth_type": "ntlm",
"auth_user": "ews-service-user@example.com",
"access_type": "delegate",
"mailboxes": {
"abc@example.com": {
"bucket": {
"owner": "project-name",
"name": "bucket-name"
},
"start_from": "bucket",
"start_timestamp": "2020-01-01T00:00:00+00:00"
},
"xyz@example.com": {
"bucket": {
"owner": "project-name",
"name": "bucket-name"
},
"start_from": "bucket",
"start_timestamp": "2020-01-01T00:00:00+00:00"
}
}
}
host
, port
, and
auth_user
with their real values, and change access_type
if required. See the configuration reference for a description of these parameters and their allowed
values.
REINFER_EWS_AUTH_PASS
environment variable - see the Deployment section. The full list
of environment variables that you can set to override values in the config is:
NAME | DESCRIPTION |
---|---|
REINFER_EWS_AUTH_USER | Exchange server user |
REINFER_EWS_AUTH_PASS | Exchange server password |
REINFER_EWS_ACCESS_TYPE | Access type: "delegate" or "impersonation" |
REINFER_EWS_HOST | Exchange server host |
REINFER_EWS_PORT | Exchange server port |
You can specify one or more mailboxes in your configuration. For each mailbox, you have to provide the mailbox address and specify the following parameters:
NAME | DESCRIPTION |
---|---|
bucket.owner | Project of the bucket in which the mailbox should be synced. |
bucket.name | Name of the bucket in which the mailbox should be synced. |
start_from | Whether to start from last synced time ("bucket") or ignore last synced time
and always start from start_timestamp ("config"). Should be set to
"bucket" for normal operation, but "config" can be useful in some cases when
debugging.
|
start_timestamp | Timestamp from which to start syncing email. If not set, all emails will be synced. |
The configuration uses the default values for a number of settings such as polling frequency or batch size. To customize your configuration further, refer to the configuration reference.
Buckets
The Exchange integration syncs raw email data into Communications Mining buckets. Same as other Communications Mining resources, a bucket is created in a project which allows you to control access to the bucket. In order to read from a bucket, upload to a bucket, or manage buckets, the user needs the respective permissions in the project the bucket is in.
You can deploy the EWS appliance either with Kubernetes or with Docker.
Deploying with Kubernetes allows you to run multiple instances of the EWS appliance, with each instance handling a subset of mailboxes to be synced.
kubectl
installed. Please check this documentation if you need help getting started with Kubernetes.
In order to deploy to Kubernetes, you need to create a YAML file describing your application. To start, copy the example below.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: reinfer-ews-appliance
labels:
app: reinfer-ews-appliance
spec:
podManagementPolicy: Parallel
replicas: 1
selector:
matchLabels:
app: reinfer-ews-appliance
serviceName: reinfer-ews-appliance
template:
metadata:
labels:
app: reinfer-ews-appliance
name: reinfer-ews-appliance
spec:
containers:
- args:
- "reinfer-ews"
- "--bind"
- "0.0.0.0:8000"
- "--reinfer-api-endpoint"
- "https://<mydomain>.reinfer.io/api/"
- "--shard-name"
- "$(POD_NAME)"
# This value should match `spec.replicas` above
- "--total-shards"
- "1"
env:
- name: REINFER_EWS_CONFIG
value: "/mnt/config/example_ews_config"
- name: REINFER_API_TOKEN
valueFrom:
secretKeyRef:
key: reinfer-api-token
name: reinfer-credentials
- name: REINFER_EWS_AUTH_PASS
valueFrom:
secretKeyRef:
key: ews-auth-pass
name: reinfer-credentials
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
image: "your.private.registry.com/reinfer/ews-appliance:TAG"
name: reinfer-ews-appliance
resources:
requests:
cpu: 0.05
memory: 128Mi
volumeMounts:
- mountPath: /mnt/config
name: config-vol
volumes:
- configMap:
name: ews-config
items:
- key: example_ews_config
path: example_ews_config
name: config-vol
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: reinfer-ews-appliance
labels:
app: reinfer-ews-appliance
spec:
podManagementPolicy: Parallel
replicas: 1
selector:
matchLabels:
app: reinfer-ews-appliance
serviceName: reinfer-ews-appliance
template:
metadata:
labels:
app: reinfer-ews-appliance
name: reinfer-ews-appliance
spec:
containers:
- args:
- "reinfer-ews"
- "--bind"
- "0.0.0.0:8000"
- "--reinfer-api-endpoint"
- "https://<mydomain>.reinfer.io/api/"
- "--shard-name"
- "$(POD_NAME)"
# This value should match `spec.replicas` above
- "--total-shards"
- "1"
env:
- name: REINFER_EWS_CONFIG
value: "/mnt/config/example_ews_config"
- name: REINFER_API_TOKEN
valueFrom:
secretKeyRef:
key: reinfer-api-token
name: reinfer-credentials
- name: REINFER_EWS_AUTH_PASS
valueFrom:
secretKeyRef:
key: ews-auth-pass
name: reinfer-credentials
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
image: "your.private.registry.com/reinfer/ews-appliance:TAG"
name: reinfer-ews-appliance
resources:
requests:
cpu: 0.05
memory: 128Mi
volumeMounts:
- mountPath: /mnt/config
name: config-vol
volumes:
- configMap:
name: ews-config
items:
- key: example_ews_config
path: example_ews_config
name: config-vol
Before you can deploy the appliance using this YAML file, there are a few additional steps you need to perform.
<mydomain>.reinfer.io
with your tenant API endpoint.
REINFER_TOKEN
and REINFER_EWS_AUTH_PASS
environment variables are populated from Kubernetes secrets. Create the secrets like so:
kubectl create secret generic reinfer-credentials \
--from-literal=reinfer-api-token=<REINFER_TOKEN> \
--from-literal=ews-auth-pass=<MSEXCHANGE_PASSWORD>
kubectl create secret generic reinfer-credentials \
--from-literal=reinfer-api-token=<REINFER_TOKEN> \
--from-literal=ews-auth-pass=<MSEXCHANGE_PASSWORD>
Finally, since we would like to load the appliance config from a local file, we need to mount that file into the pod. We do this by storing the data in a Kubernetes ConfigMap and mounting the ConfigMap as a volume. Create the ConfigMap like so:
kubectl create configmap ews-config \
--from-file=example_ews_config=your-ews-config.json
kubectl create configmap ews-config \
--from-file=example_ews_config=your-ews-config.json
NOTE
As an alternative to storing the config file locally, you can upload it to Communications Mining and let the EWS appliance fetch it via the Communications Mining API. This is described here. If both local and remote config files are specified, the appliance will use the local config file.
You can now create your statefulset and check that everything is running:
kubectl apply -f reinfer-ews.yaml
kubectl get sts
kubectl apply -f reinfer-ews.yaml
kubectl get sts
Alternatively, you can run the EWS appliance in Docker. The command below will start the appliance with the same parameters that are used in the Kubernetes section.
EWS_CONFIG_DIR=
REINFER_API_TOKEN=
MSEXCHANGE_PASSWORD=
TAG=
sudo docker run \
-v $EWS_CONFIG_DIR:/mnt/config \
--env REINFER_EWS_CONFIG=/mnt/config/your_ews_config.json \
--env REINFER_API_TOKEN=$REINFER_API_TOKEN \
--env REINFER_EWS_AUTH_PASS=$MSEXCHANGE_PASSWORD \
eu.gcr.io/reinfer-gcr/ews:$TAG \
reinfer-ews --reinfer-api-endpoint https://<mydomain>.reinfer.io/api/ &> ews_$(date -Iseconds).log
EWS_CONFIG_DIR=
REINFER_API_TOKEN=
MSEXCHANGE_PASSWORD=
TAG=
sudo docker run \
-v $EWS_CONFIG_DIR:/mnt/config \
--env REINFER_EWS_CONFIG=/mnt/config/your_ews_config.json \
--env REINFER_API_TOKEN=$REINFER_API_TOKEN \
--env REINFER_EWS_AUTH_PASS=$MSEXCHANGE_PASSWORD \
eu.gcr.io/reinfer-gcr/ews:$TAG \
reinfer-ews --reinfer-api-endpoint https://<mydomain>.reinfer.io/api/ &> ews_$(date -Iseconds).log
- Replace
<mydomain>.reinfer.io
with your tenant API endpoint. - Replace
your_ews_config.json
by the name of your EWS config JSON file.
The appliance will run continuously syncing emails into the Communications Mining platform. If stopped and started again, it will pick up from the last stored bucket sync state.
The EWS appliance can save extracted emails locally instead of pushing them into the Communications Mining platform.
EWS_LOCAL_DIR=
MSEXCHANGE_PASSWORD=
CONFIG_OWNER=
CONFIG_KEY=
TAG=
sudo docker run \
-v $EWS_LOCAL_DIR:/mnt/ews \
--env REINFER_EWS_AUTH_PASS=$MSEXCHANGE_PASSWORD \
eu.gcr.io/reinfer-gcr/ews:$TAG \
reinfer-ews --local-files-prefix /mnt/ews \
--remote-config-owner $CONFIG_OWNER --remote-config-key $CONFIG_KEY &> ews_$(date -Iseconds).log
EWS_LOCAL_DIR=
MSEXCHANGE_PASSWORD=
CONFIG_OWNER=
CONFIG_KEY=
TAG=
sudo docker run \
-v $EWS_LOCAL_DIR:/mnt/ews \
--env REINFER_EWS_AUTH_PASS=$MSEXCHANGE_PASSWORD \
eu.gcr.io/reinfer-gcr/ews:$TAG \
reinfer-ews --local-files-prefix /mnt/ews \
--remote-config-owner $CONFIG_OWNER --remote-config-key $CONFIG_KEY &> ews_$(date -Iseconds).log
- The appliance expects to find the config in
$EWS_LOCAL_DIR/config/$CONFIG_OWNER/$CONFIG_KEY.json
. You can alternatively provide the path to config by setting the$REINFER_EWS_CONFIG
environment variable. - The appliance will save the sync state to
$EWS_LOCAL_DIR/state
. If stopped and started again, it will pick up from the last stored sync state. - The appliance will save data to
$EWS_LOCAL_DIR/data
.
Instead of providing a local config file to the appliance like you did if you followed the EWS appliance deployment guide, you can instead manage the config file in Communications Mining. Note that if both local and remote config files are specified, the appliance will default to using the local config file.
First, upload your JSON config file to Communications Mining:
curl -H "Authorization: Bearer $REINFER_TOKEN" \
-H "Content-Type: multipart/form-data" \
-F 'file=@your-ews-config.json' \
-XPUT https://<mydomain>.reinfer.io/api/v1/appliance-configs/<project-name>/<config-name>
curl -H "Authorization: Bearer $REINFER_TOKEN" \
-H "Content-Type: multipart/form-data" \
-F 'file=@your-ews-config.json' \
-XPUT https://<mydomain>.reinfer.io/api/v1/appliance-configs/<project-name>/<config-name>
To see the current config:
curl -H "Authorization: Bearer $REINFER_TOKEN" \
-XGET https://<mydomain>.reinfer.io/api/v1/appliance-configs/<project-name>/<config-name>
curl -H "Authorization: Bearer $REINFER_TOKEN" \
-XGET https://<mydomain>.reinfer.io/api/v1/appliance-configs/<project-name>/<config-name>
--remote-config-owner
parameter to the project name, and the --remote-config-key
parameter to the config name.
See the table below for a list of available application parameters. You can learn more about running the EWS appliance here.
PARAMETER | DESCRIPTION |
---|---|
--reinfer-api-endpoint | Endpoint to connect to the Reinfer API. Mutually exclusive with --local-files-prefix .
|
--local-files-prefix | Path to store synced emails and bucket sync state. Mutually exclusive with --reinfer-api-endpoint and REINFER_API_TOKEN .
|
--remote-config-owner | Project that owns the remote EWS appliance config file. |
--remote-config-key | Name of the remote EWS appliance config file. |
--debug-level | Debug level. 0 = No debug, 1 = Service debug, 2 = Full debug. Default: 1. |
--shard-name | Shard name i.e. ews-N to extract shard number from. When running in Kubernetes, you can set it to the pod name.
|
--total-shards | The total number of instances in the appliance cluster. When running in Kubernetes, must be set to the same value as the number of instances in the StatefulSet. |
--restart-on-unrecoverable-errors | If enabled, unrecoverable failures will result in the entire service being restarted without crashing. |
See the table below for a list of available configuration parameters. You can learn more about writing the EWS appliance configuration file here.
NAME | DESCRIPTION |
---|---|
host | Exchange server host. Can be overriden by the REINFER_EWS_HOST environment variable.
|
port | Exchange server port. Default: 80. Can be overriden by the REINFER_EWS_PORT environment variable.
|
auth_type | Only "ntlm" allowed. |
auth_user | Exchange server user. Can be overriden by the REINFER_EWS_AUTH_USER environment variable.
|
auth_password | Exchange server password. Can be overriden by the REINFER_EWS_AUTH_PASS environment variable.
|
access_type | Access type: "delegate" or "impersonation". Default: "delegate". Can be overriden by the REINFER_EWS_ACCESS_TYPE environment variable.
|
ews_ssl_verify | If set to "false", will not verify certificates. Default: "true". |
poll_frequency | How long to wait between batches, in seconds. Default: 15. |
poll_message_sleep | How long to wait between individual emails in a batch, in seconds. Default: 0.1. |
max_concurrent_uploads | Number of concurrent uploads to Communications Mining, between 0 and 32. Default: 8. |
emails_per_folder | Max number of emails to fetch from each folder per batch, between 1 and 100,000. Default: 2520. This setting allows the appliance to make progress on all folders evenly in case there is a very large folder. |
reinfer_batch_size | How many emails to fetch per batch, between 1 and 1000. Default: 80. |
mailboxes | List of mailboxes to fetch. See here for an explanation of how to configure the mailboxes. |
audit_email | If you have configured the appliance with a remote config, Communications Mining will send an email to this address whenever the config is updated. Default: None. |
ews_ssl_ciphers | Make EWS appliance use specific ciphers. The ciphers should be a string in the OpenSSL cipher list format. Default: None. |