- Getting Started with Test Suite
- Studio
- Orchestrator
- Testing robots
- Test Manager
AI-powered evaluation
This feature is currently part of an audit process and is not to be considered part of the FedRAMP Authorization until the review is finalized. See here the full list of features currently under review.
This page lists guidelines for evaluating requirements in Test Manager.
This section provides examples of additional instructions you can provide to AutopilotTM.
Check the following guideline that you can give to Autopilot for analyzing requirements from various perspectives:
"Generate the top 10 enhancement suggestions from diverse perspectives:
- User Flow: Check for detailed information about the user flow and application logic.
- Consistency: Examine for logical coherence and lack of contradictions.
- Relevance: Check if the requirement clearly highlights the user's needs.
- Rationale: Check if it explains why it is necessary and what problem it solves.
- Clarity: Check for precision and clarity in the requirement description.
- Criteria: Check for measurable and achievable acceptance criteria.
- Completeness: Check for inclusion of all critical functionalities.
- Prerequisites: Note missing prerequisites or conditions needed before implementation.
- Security: Check if the requirement addresses security and privacy concerns adequately.
- Compliance: Check for any potential legal and regulatory compliance issues.
- Ethics: Check if the requirements promote fair and responsible behavior.
- Terminology: Verify the use of clear, inclusive, and universally understood language.
- Performance: Check for clear performance metrics and benchmarks."
Check the following guideline that you can give to Autopilot for analyzing requirements for security-related aspects.
"Generate the top 10 enhancement suggestions focusing solely on security aspects:
- Access: Check if it is clear who can access the application and under what conditions.
- Protection: Check if encryption, anonymization and secure storage measures are defined.
- Authentication: Check if identity verification and permission controls are included.
- Vulnerability: Check if measures against known vulnerabilities are addressed.
- Transmission: Check if encrypted data transmissions are mandated.
- Logging: Check if detailed logging and monitoring provisions are included.
- Compliance: Check if adherence to relevant security standards is ensured.
- Errors: Check if secure error handling practices are specified.
- Response: Check if there are protocols for responding to security incidents."
Check the following guideline that you can give to Autopilot for analyzing requirements for performance-related aspects.
"Generate the top 10 enhancement suggestions focusing solely on performance aspects:
- Time: Check if specific response times under various load conditions are detailed.
- Throughput: Check if expected throughput rates are clearly specified.
- Scalability: Check if the requirement addresses scalability under increasing loads.
- Resource usage: Check if limits on CPU, memory, and storage usage are defined.
- Efficiency: Check if the requirement promotes efficient use of resources.
- Load handling: Check if strategies for handling peak load conditions are included.
- Concurrency: Check if behaviors under concurrent user scenarios are specified.
- Latency: Check if maximum acceptable latency figures are clearly stated.
- Stability: Check if stability under stress conditions is addressed.
- Capacity: Check if maximum system capacity limits are specified."
This section lists various types of supporting documents you can provide to AutopilotTM. Consider these documents as additional information that enhances the description of a requirement in Test Manager. Because these documents complement the requirement description, Autopilot also offers suggestions to enhance their quality. Essentially, the definition of your requirement is shaped by both requirement description and all associated supporting documents.
Consider providing security guidelines, policy documents, or security audit findings. Autopilot will review these documents to check for any inconsistencies or discrepancies within them or between these documents and the security aspects outlined in the requirement description to ensure that security criteria are both comprehensive and accurately represented.
Consider providing accessibility guidelines, audit reports, or user accessibility specifications. Autopilot will evaluate these supporting documents to cross-check their alignment with the accessibility criteria stated in the requirement description. This evaluation helps confirm that the guidelines are effectively integrated into the requirement and are free of contradictions.
Consider providing compliance checklists, regulatory summaries, or compliance requirements relevant to your industry. Autopilot will assess these checklists to ensure that they are consistent with the compliance criteria described in the requirement description. This helps in identifying any potential inconsistencies or gaps in regulatory adherence.
This section outlines the current limitations of AutopilotTM.
You can only upload the following file extensions, from which Autopilot processes only the text content:
- DOCX
- XLSX
- TXT
- PNG
- JPG
- BPMN
The input token capacity of Autopilot is 128,000 tokens, which is equivalent to approximately 96,000 words, or 512,000 characters.
Ensure that your requirement description and supporting documents do not exceed these limits.
To check the approximate token count of your documents, open the document as a TXT file and copy the content into the Open AI Tokenizer tool The provided token count is an approximate. The actual token count can be higher.
