In the previous posts, we talked about taking the first steps towards building an Edge AI solution. Once you frame the problem and have your most crucial raw material data covered, we are ready to focus on the design. Design requires building on the steps of problem framing, dataset collection, technology choices, and responsible AI.
Two main parts of designing an Edge AI application are the Product or the solution itself and the technology architecture to make it work.
Product and Experience Design
For the product to be considered solved, we need to look at
(a) did the product solve the problem (b) did it meet the user expectations (c) did it meet the realities of the world
Design Principles
In the age of AI, where speed, scale, and scary can simultaneously all be components of the products we design, we have to change design from a noun to become a very, deliberate verb. We’re entering a Brave New World. That world requires designers to take on a larger responsibility for the outcomes produced, behavior induced, and effect on humanity the intelligent products we design to have.
—Ovetta Sampson
Good design based on Sampson’s article
- Solves hard problems with limited resources
- Promotes healthy relationships with other people and other products
- Requires malleability and a good amount of customization for people who use it
- Has an accurate understanding of the needs of end users, not the marketing department
- Acknowledges bias as bias is always present
- Prevents dishonesty by being honest about the negative impacts of the product
- Expects unintended consequences that AI systems can harm people and it should build guardrails around that
- Fosters equity as AI systems can harm people
- Considers its effects on collective and connected ecosystem
- Purposefully brings order to chaos
In practical terms: while a bad doctor might harm thousands of patients over the course of their career, a bad medical AI system can harm millions.
Scoping
AI/ML development is inherently unpredictable. Dataset, Algorithm development, Size of deep learning model to produce accurate results and the types of devices where the solution can be deployed can have lots of permutations. Without these earlier hurdles being crossed, one cannot begin the hardware development process, if needed.
Fully autonomous cars seemed to be around the corner a few years back. However, their progress has stalled because it has not been proven to be extremely accurate when there are high-speed interactions with the general public. A good thing out of all that research is ADAS (Advanced driver assistance systems) that ease the physical and mental burden on drivers to improve road safety. Adaptive cruise control can take over acceleration, braking, and lane changes when the car is on the highway. This is a narrowly constrained system and would work well on the highways but not on city streets. Thus we can get remarkable accuracy with narrowly constrained systems. This narrow success allows for the companies to stay in the market. Engage with real customers and real-world situations. Keep iterating to make the success wider and eventually build an accurate self-driving system.
Instead of pursuing grand ideas from the beginning, try to identify the small, useful stepping stones that still deliver real value.
Determine the minimum viable product: a simple, achievable benefit that is genuinely helpful to your users.
Build something out, see how it performs in the real world, and iterate from there.
Design Goals
There are 3 kinds of main goals systemic, technical, and value goals. The definition of these goals needs input from project stakeholders and domain experts.
Systemic
System goals reflect the overall performance of the system. They ensure that we are building the right thing.
Knowledge of the current solution and the improvement that needs to be done lets us design systems. For example, if the shelves in a retail store are restocked 60% of the time we need to improve it to a level of 90%.Then we know that we have an improvement of 30% with technology. This goal produces real business value when the customers can find what they are looking for. This is better than having a technical goal that could say. That the AI system should be able to predict that the shelf needs restocking with 95% accuracy.
Technical
These goals tell us if we are building the system right. In the above example, we could mention that the accuracy of the system should be 99% and within 60s of a shelf being empty. The alert should be triggered then we will put the necessary SLAs on the solution.
Value
This is done in consultation with the stakeholders and domain experts again on what is the minimum level of accuracy expected to derive value. If the solution falls below this value then it means that the ROI on the solution is not justified. In the above example, if the accuracy of our empty shelf recognition system falls below 96% then potentially it is a waste of effort.
Architectural Design
How the components fit together to create an effective solution. Each architecture has trade-offs and those need to be considered. Architecture for an Edge AI solution is a combination of hardware, the edge devices on which the models would run, software beginning with low-level drivers that allow the software to work with the hardware and all DSP, AI algorithms that would run on the device plus the services that are external systems that an edge AI system can interface with. These include communication networks, wireless systems, IoT management platforms, web APIs, cloud APIs, etc.
A typical edge computing architecture might look like this
But we are more interested in the hardware, software, and services needs of the edge device. Zooming into the architecture of the edge device, it would look more like this
The Application loop is the series of repeating steps where signals are captured and processed, AI algorithms are run and their output is interpreted to trigger business conditions. Consider this to be the processing unit of the device that interacts with the sensor unit. This is supported by the Device firmware or OS that allows the application loop to work with the hardware device like reading data from them, receiving and sending instructions, or controlling device movements.
Basic flow
Ensemble flow
Here by combining the output of multiple algorithms, a decision can be made that would be less biased
Parallel or serial flow
The output of 2 algorithms running in parallel could be combined in the postprocessing step or the business logic. Most embedded processors are not capable of more than a single thread of computation so the algorithms may be called out one after the other like below. In a serial flow, the decorated output of one algorithm can be passed as an input to the next one. In parallel flow, the same input is passed to all the algorithms
Cascade flow
Here there could be multiple algorithms each with their own set of postprocessing and business logic. The final business logic value might be fed into the next step of the cascade. For example, instead of doing wakeup word spotting, the first algorithm could just detect speech. If the business logic confirms that it is speech then it can pass on the words to the next algorithm that identifies if it is the wake word or not
Here the first algorithm could be low-powered and take less energy than the second one which is more energy-hungry. The processing unit for the first algorithm could be a basic one and that for the second processing could be a beefed-up one.
- Multi-device cascade – The cascading could happen on multiple devices as well instead of just one with two processors.
- Cascade to cloud – Once the processing is done on the low-powered devices. The next set of processing could be passed onto the cloud where large models that are too large to be deployed on the edge devices can take on the hard work. This is the case with smart speakers that have multi cascade stages, 3 on the device and the final one on the cloud
Sensor fusion flow
There are multiple sensors that pass on their data to the algorithm where it is fused together and passed onto postprocessing. Heart monitoring on smartwatches where data from skin temperature, heart rate, and motion sensors is fused together to predict if the heart is in the safe operating zone or not.
Summary
In this post, we understood that design and development is an iterative process. It is a living consideration throughout the lifecycle of the project. We understood the fundamentals of a good design, how to scope a system and define the design goals in terms of system, technology, and value. We further worked through various permutations of architectural patterns that allowed us to develop an Edge AI solution.
