Examining functional AI selection criteria and planning compared to your desired outcomes.
Artificial intelligence (AI) is sweeping through every industry. It’s out of control, like the Wild West. AI output is showing up in reports and presentations. The Apple App Store and Google Play offer many free AI apps of varying quality. Many AI software vendors provide access to their chatbot websites. AI output is part of search results. AI capabilities are integrated into desktop software.
What should engineers consider as they select AI solutions to realize AI’s business benefits, often as part of their digital transformation, and exercise some AI oversight?
Companies often digitally transform operations, production or logistics when implementing AI applications. Through advances in generative AI, machine learning, big data analytics, and automation, companies benefit from efficiency, safety, and environmental sustainability.
Most companies will be better off rolling out vendor AI solutions as they become available rather than designing and implementing an entirely new and exceedingly ambitious custom AI application.
Practical AI applications in most industries are not new systems. Instead, AI functionality adds value to existing applications through:
- More confident or accurate results.
- Reduced elapsed time to achieve the results.
- A significant reduction in staff effort to achieve the results.
- Increased scope or scale in factors such as the number of customers, products, inventory items or larger geographic coverage.
- Additional functions that couldn’t be achieved without AI.
More specific and detailed selection criteria are more likely to satisfactorily select an AI solution that will satisfy the scope of the planned AI project. More high-level, general and conceptual selection criteria will make differentiating the available AI software difficult. This situation will lead to selecting AI software based on appearances, such as marketing presentation quality, website appeal or impressions from the AI vendor demo, rather than capability.
The primary selection criteria engineers should consider when selecting AI solutions for their digital transformation projects are these.
Functional requirements
Functionality selection criteria consider the AI software’s fit against the planned digital transformation application’s requirements.
AI application control
When selecting an AI solution, deciding on a Software-as-a-Service (SaaS) or On-premises solution is the first consideration. These two alternatives vary significantly in the amount of control your company will have over the AI application.
A Software-as-a-Service (SaaS) solution offers these advantages:
- Pay-as-you-go on a monthly or annual basis.
- There is no computing infrastructure to operate.
- The AI vendor handles all software and data updates.
An On-premises solution offers these advantages:
- Total control over software, data, performance and application availability.
- Complete data privacy with no opportunity for the AI vendor to use your company’s data for model training or to sell it to others.
- Opportunity to build a smaller, focused, custom AI model for more accurate results, fewer hallucinations and faster performance.
Currently, most companies choose a SaaS AI solution because they don’t have the AI skills and computing infrastructure to build and operate an on-premises AI solution. However, this situation will likely change as companies build AI experience and want to tightly control their AI model and proprietary data for competitive advantage.
Problem fit
Engineers should consider whether the AI software can generate the desired output, typically text, images or audio, that aligns with their specific application requirements. Understanding the capabilities and limitations of proposed AI solutions in relation to the problem at hand is crucial for successful adoption.
Define specific problems or opportunities the AI solution will address as test cases. Perform the test cases for each proposed AI solution to differentiate them.
Accuracy
As engineers assess the accuracy of AI model outputs, they should consider related criteria such as the following:
- Quality of the generated outputs.
- Ability to generalize to different inputs or scenarios.
- Consistency of results.
- Frequency of hallucinations or inaccurate or misleading results.
Specify the desired level of accuracy for the AI solution’s output and test to establish whether or not the candidate AI software can achieve this level.
Non-functional requirements
Non-functional selection criteria consider the AI software’s quality attributes, such as security, usability, and scalability.
Performance
Engineers evaluate the performance of AI software to ensure that it meets their desired response time.
AI software accesses a large language model (LLM) to formulate the required response to prompts. That process can take time. If the desired response time cannot be achieved, the actions to consider include:
- Rerun the test with similar but revised prompts to see if the response time changes materially.
- Rerun the test with another AI software offering.
- Build a smaller LLM that is more focused on the problem space.
- Confirm that there are no network bottlenecks between the workstation and the AI solution.
Scalability
An increasing number of end-users will likely access the AI application over time. Also, the number of prompts a given end-user will issue will increase over time.
To handle that growth, the AI application must scale well. Scalability criteria include:
- Performance remains the same even though the number of prompts increases.
- The quality of outputs does not deteriorate.
- The associated cost increase is acceptable.
Ethical considerations
To responsibly adopt AI solutions, engineers must consider ethical implications. They should consider factors such as:
- Data privacy protection.
- Fairness, as opposed to bias, in the output.
- The risk of potentially harmful or unethical uses.
Ethical considerations can be revealed by testing the candidate AI solutions.
Security and compliance
As engineers assess the various AI solutions, they should evaluate each potential AI solution’s security measures and compliance features.
Vendor criteria
Vendor selection criteria consider the capability and likely performance of the AI solution vendor. Vendor evaluations for AI solutions for digital transformation are risky because:
- Predicting the likelihood of AI vendors being absorbed by a merger or failing is difficult.
- Most vendors are so new that they have little track record with customers.
Vendor expertise
Engineers assess each AI vendor’s experience and capabilities in the relevant AI domain for their planned application.
Vendor support and maintenance
Evaluate each AI vendor’s support and maintenance services, most likely through customer references.
Customization and flexibility
Determine each AI vendor’s willingness and ability to adapt their solution to specific needs. If the evaluation suggests that customization will be required, that AI solution should be dropped from further consideration.
Costs
The cost selection criteria engineers consider when designing AI solutions include licensing, implementation, and operating costs.
License cost
Compare the license cost of the various on-premises AI solutions.
Implementation cost
Compare the implementation cost of the various AI solutions. The implementation cost includes:
- Computing infrastructure upgrades.
- People change management.
- Revisions to business workflows.
If the AI solution uses your company’s proprietary data, there will be significant data preparation costs.
Operating cost
Compare the operating and maintenance costs of the various on-premises AI solutions.
Compare the monthly usage cost of the various SaaS solutions.
Business case
The business case for the alternative AI digital transformation solutions will vary. The easiest choice is always the alternative with the strongest business case based on tangible benefits. However, there are always intangible benefits to consider, such as customer satisfaction or contribution to the strategic plan. The final AI solution recommendation should consider tangible and intangible benefits.
Allocating more effort to select an AI solution with more specific and detailed selection criteria will reduce risk and position the planned AI digital transformation project for success.
