silo.ai
  • Services
  • Solutions
  • Work
  • Research
  • Contact
  • •••
    • About
    • Careers
    • Learn
Menu
  • Services
  • Solutions
  • Work
  • Research
  • Contact
  • •••
    • About
    • Careers
    • Learn
silo.ai
  • Services
  • Solutions
  • Research
  • About
  • Careers
  • Learn
  • Contact
Menu
  • Services
  • Solutions
  • Research
  • About
  • Careers
  • Learn
  • Contact
Smart Industry

AI on the edge: machine learning in restricted environments

  • November 26, 2021

Machine learning in embedded systems is becoming increasingly important. We are seeing it being utilized in a multitude of different application areas including automotive, manufacturing, robotics as well as as well as in advanced consumer devices, to name a few. To process data close to its acquisition has multiple benefits, for example, lowering latencies or reducing data transmission bandwidth. This is referred to as edge computing, in comparison to moving data to the remote computer or cloud.   

Today, the evolution of processing capabilities provided by the modern embedded computing solutions, from microcontrollers to SoCs (System on a chip) as well as advanced deployment frameworks, are supporting this embedded or edge AI trend. The adoption of these technologies is already happening and will be expanding in the future. In this blog article, I will share my key insights on applying modern AI, such as deep neural networks, into restricted computation environments found in modern embedded and edge devices.

What do we mean by restricted environments?

All computing systems have limitations, but some chipset environments are significantly more challenging than others. Main restrictions rise from hardware specifications, such as limited computing capacity or small memory size. However, other aspects such as power consumption and thermal behavior are also factors that typically need to be taken into account. 

Architectures for embedded AI have different capabilities. In this blog post, we will focus on the following embedded AI architectures:

  • System on a chip (SoC) 
  • 32-bit microcontrollers
  • 8-bit microcontrollers

Traditionally smartphones and tablets devices can be considered as embedded systems, however, their computational capabilities are already approaching the level of common low-end PCs. SoCs used in these devices also contain graphics processing units (GPU) and dedicated AI accelerators. They are able to run advanced operating systems and have rich connectivity support. Nowadays similar SoCs are used also for industrial applications, automotive, and advanced consumer electronics appliances.

In smaller devices like microcontrollers, there is an evolution towards the adoption of 32-bit architectures, and while still significantly less capable than SoCs they provide an interesting platform for modern data-driven AI solutions, especially in low-power, cost-efficient sensor data processing. Taking a closer look at smaller microcontroller devices it is observed that some modern processing architectures, such as Cortex-M4F have also already advanced operations (dual MAC, SIMD, etc.) that can speed up the execution of the machine learning algorithms. 

Nevertheless, we should not overlook the possibility of utilizing advanced data-driven algorithms even in very small older 8-bit architectures. While these systems are naturally quite limited, with proper design and choice of correct machine learning approaches they can be also harnessed to run AI algorithms.  

Machine learning in low resource hardware

In all product designs, but especially when working with restricted hardware, we at Silo AI consider good R&D practices that embrace system-level thinking crucial. With a limited computing environment, there is typically a need to do design compromises. In order to do these in an optimal and controlled way, it is vital to:

  • Understand the use case requirements. This needs to be looked at from both end customers but also from an architecture point of view. Consider factors like functionality, accuracy, speed, latency, maintainability, interfacing, etc. What are the most valuable features that cannot be compromised? 
  • Identify system limitations. Seek to understand the processing power, memory size, and thermal limits but also restrictions due to the software environment. It is important to look at the total system-level behavior, and not to focus only on the machine learning parts. Eventually, the full product functionality is the sum of all its components and AI needs to be integrated seamlessly as part of the total solution. 
  • Identify R&D constraints. Consider cost, time-to-market, resources, available competencies, lifetime of the product, etc. Sometimes with limited resources more optimization and tailoring is needed. If highly optimized and proprietary solutions are developed, can it be guaranteed that they can be maintained throughout the whole product life cycle?
Edge AI – Machine learning in low resource hardware.

AI development for the edge

At the beginning of the practical AI development project, it is important to start with an approximate understanding of the platform’s capabilities and of the complexity of the machine learning solution. It is tempting to jump to development and experimentation with algorithms without any restrictions in an environment like a powerful desktop PC or cloud computing service. This is often justified by the need to first see if the chosen approaches would work for use cases or not. While it is useful to get such quick insight, especially when working with a new problem, there is a risk that a significant R&D effort is spent on solutions that eventually cannot be deployed into target hardware. In order to avoid this, two critical sets of restrictions need to be considered from the beginning:

The first set of restrictions are related to the hardware architecture. For example, in very small devices like microcontrollers, memory, including both program memory used to store the machine learning models and RAM needed for executing those models, are critical. In many use cases with microcontrollers, the processing speed might be less of an issue as the models are relatively small and thus their execution time will not be a bottleneck.

For more capable SoC’s the relevant use cases do often require models that are more complex and input data is large, like images or a video stream. In these cases, the performance is less often restricted by memory, but instead, computational speed, power consumption, or overheating can be the critical limiting factors. Thus, the model complexity must match the hardware capabilities from the start.  Trying to modify overly complex machine learning solutions into a limited computing environment is often doomed to failure.

The second design restriction is coming from software. Typically the machine learning frameworks that are optimized for edge computing do not support all features that full versions of the same frameworks are offering. For example, some neural network layer types or activation functions may not be available in the target environment. The problem arises when there is a need to use network architecture with these unsupported features. Luckily, in many cases, modifying some part of the network so that it supports inference on edge devices is possible, but this should be done before considerable effort and time is spent for training the network.

It should be noted that while deployment frameworks are constantly evolving and new features are being added, AI architectures are also evolving rapidly. Sometimes, documentation is lacking behind the actual implementation. Especially in the case of deploying cutting-edge machine learning architecture to an embedded device, it is advisable to verify the support for required features with practical testing with the target platform.

To summarize, it is crucial to start testing AI solutions already at the beginning of the developed project in the targeted hardware and software platform. From a performance point of view, it is also important that the full use case, including data acquisition and pre-processing and application logic, is part of the testing starting from relatively early stages. This ensures that the chosen AI solution can be executed in the final product with the required accuracy and performance.  

Workflow of a typical machine learning solution.

Hands-on

At Silo AI we have worked on customer projects from 8-bit and 32-bit microcontrollers to advanced SoCs with dedicated machine learning accelerators. While each of the customers’ case is different and platforms have different toolchains for optimal deployment, the basic AI development flow follows similar steps:

  • Machine learning models are developed and trained with common frameworks such as TensorFlow or PyTorch.
  • Models are then converted to the format that is more suitable for efficient execution in the target hardware environment. Some platforms have their own format and APIs for this, but many of them are also supporting more generic deployment formats, for example, TensorFlow Lite. In some cases, especially for microcontrollers, the machine learning models are converted directly to executable code to minimize the memory footprint.
  • Solutions are tested in the target environment, and if needed model parameters and performance are iterated. Typically in the early phase of the development different architectures are tested in order to find the best alternative for the use case requirements.

Conclusions

We are seeing a bright future for embedded and edge AI. In order to tap the entire potential of these resource restricted environments, it is always important to look at problems holistically, understand use case requirements and consider total system-level architecture. AI is a great tool, but it won’t replace good product development practices.  


Image of Pertti Hannelin

Would you like to build machine learning-driven products for restricted environments?

Get in touch with our VP of Business Development Pertti Hannelin to discuss further on LinkedIn or via email.

Share

Share on twitter
Share on facebook
Share on linkedin

Author

  • Jukka Yrjänäinen Jukka Yrjänäinen

Topics

Edge AIMachine Learning

You might be interested in

Silo AI and Mila join forces to connect leading AI experts in the Nordics and Canada 

Pauliina Alanen 2.2.2023

Silo AI, one of Europe’s largest private Artificial Intelligence (AI) labs, is proud to announce a partnership with Mila – Quebec AI Institute, the world’s largest academic deep learning research center. Founded by the leading AI researcher, Yoshua Bengio, Mila brings together more than 1,000 academic researchers in machine learning (ML). The partnership aims to connect state-of-the-art AI research with industry needs. With a strong experience in building AI-driven products, Silo AI has gathered a unique pool of AI talent including 240 AI experts, out of which 120 have a PhD degree. 

Read more

Read More

Hype, hope or revolution: What is ChatGPT and do you need to care?

Peter Sarlin 31.1.2023

The hype is most definitely real. OpenAI’s conversational chatbot ChatGPT has in recent weeks provided hope. But is it a true technological revolution? Put simply, the answer is both yes

Read More

We challenge you to ask why

We don’t only deliver projects but we challenge you to think different.
Contact

Subscribe to Silo AI newsletter

Join 5000+ subscribers who read the Silo AI monthly newsletter

silo.ai
Contact

+358 40 359 1299

info@silo.ai

  • Helsinki, Finland
  • Stockholm, Sweden
  • Copenhagen, Denmark
Menu
  • Home
  • Services
  • Solutions
  • Research
  • Work
  • About
  • Careers
  • Contact
Menu
  • Home
  • Services
  • Solutions
  • Research
  • Work
  • About
  • Careers
  • Contact
Resources
  • Learn
  • Inference podcast
  • For media
  • MLOps
  • Predictive maintenance
  • Nordic State of AI report
Menu
  • Learn
  • Inference podcast
  • For media
  • MLOps
  • Predictive maintenance
  • Nordic State of AI report
Linkedin Facebook-square Twitter Instagram Spotify
©2017-2023 All Rights Reserved.

|

Website Privacy Policy / Cookie Policy / Newsletter Privacy Policy / Recruitment Privacy Policy

Manage cookies
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent. Read Cookie Policy
Cookie SettingsAccept All
Manage cookies

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT
Powered by CookieYes Logo