\chapter{Evaluation}
\label{chap:eval}

 The goal of the project was to implement a portable SmartCGMS concept. To achieve this, the SmartCGMS codebase has been modified to use static linking during compilation, and C++ features that are not widely available in \glspl{rtos} have been removed. With these modifications, it is possible to compile the SmartCGMS concept as a native application for FreeRTOS or any platform that supports C++17. 
 
To evaluate the SmartCGMS concept, it has been compiled for five different platforms. The five platforms are:
 
 \begin{itemize}
     \item Raspberry Pi Zero W running FreeRTOS (ARMv6)
     \item ESP32 running FreeRTOS (Xtensa)
     \item GNU/Linux (x86-64)
     \item \gls{wamr} (compiled using \gls{wasi} \gls{sdk})
     \item SmartCGMS demo webpage (compiled using Emscripten)
 \end{itemize}

The first three platforms use a native application for specified \gls{isa} and operating system, while the last two are compiled to \gls{wasm}. A set of four filters has been developed (described in section \ref{sec:filters}. The filters can be compiled for any of the mentioned platforms. 

The evaluation is divided into two parts. First, the correct functionality of the modified SmartCGMS framework and the developed filters is verified. Second, the operational characteristics of the SmartCGMS concept applications are measured and compared for different platforms. The next sections outline the experimental setup for each measurement, followed by the presentation and discussion of the gathered results.

\section{Correctnes of Execution}
\label{subs:correctnes}

To verify the correct functionality of the modified SmartCGMS framework, a test scenario consisting of several interactions with the SmartCGMS framework was created. The test scenario was executed on each platform, and the results were collected and compared. The goal of this test was to evaluate if the modified SmartCGMS framework can load the developed filters, build a filter chain, and process events with the same results as the original SmartCGMS. This test also aimed to verify that the implemented filters behave as intended.

\subsection{Experimental Setup}

The experimental setup includes an array of 100 double-precision floating-point values. The values were pseudo-randomly generated and do not represent any real data. The same array was used on all platforms. These values were then used to create level events and were processed by the SmartCGMS concept applications. The filter chain processing the events was the same for all platforms. It includes these filters in the specified order:

\begin{itemize}
    \item Data reading filter
    \item Data transformation filter
    \item Data visualization filter
\end{itemize}

First, the filters were compiled as Microsoft Windows dynamic-link libraries for the original SmartCGMS. The filters' source code files were then preprocessed by the implemented filter preprocessor. The modified source code files of the filters were then compiled for the SmartCGMS concept platforms.

The configuration remained identical for all platforms except for the \texttt{create\_task} parameter of the data reading filter. On the ESP32, a task to listen for incoming data on UART was created. The data from the array was then transmitted to the ESP32 using UART. The other platforms used the provided SmartCGMS concept \gls{api} to create events directly in the same task/thread that built the filter chain. The results were collected using the data visualization filter. After each execution of the test scenario, the printed event levels were saved to a file. The same test scenario was carried out using the SmartCGMS console application on Microsoft Windows 10. After all the results were gathered, the events' level values were compared.

The watchdog filter has been tested separately. The watchdog has been implemented to indicate that the filter chain is inactive. To test its correct functionality a filter chain was built but no events were sent into the chain. After the timeout period, the watchdog was expected to trigger. For the test, the \texttt{Trigger} function has been implemented as a print statement to the console, signaling the chain is inactive. The test scenario was executed, and the console was observed to see whether the watchdog was triggered or not. This test has been carried out only on the SmartCGMS concept platforms.

\subsection{Results and Discussion}

The events' level values were printed with a precision of six decimal places. The level values were identical on all platforms, including the original SmartCGMS. This suggests that the modifications to the SmartCGMS codebase didn't influence its functionality. Additionally, the performed tests indicate that the proposed and implemented filter preprocessor functions correctly as it was used during the setup for the tests. One of the biggest limitations of the SmartCGMS concept is its lack of filesystem support. Multiple filters that are developed for the original SmartCGMS use files. Implementing a simple filesystem for the SmartCGMS concept-supported platforms would allow us to bring the concept even closer to the original SmartCGMS. Emscripten provides a virtual filesystem that allows us to use more features from the original SmartCGMS (e.g., deferring model parameters to files), see section \ref{sec:webmodel}


\begin{figure}[tpb]
\centering
\begin{minipage}{0.3\textwidth}
  \includegraphics[width=\linewidth]{figures/ESP32.png}
  \caption{ESP32}
  \label{fig:taskshort}
\end{minipage}
\hspace{0.05\textwidth}
\begin{minipage}{0.6\textwidth}
  \includegraphics[width=\linewidth]{figures/EMCC.png}
  \caption{The SmartCGMS webpage}
  \label{fig:tasklong}
\end{minipage}
\caption{The event level values being collected on different platforms}
\end{figure}

The filters' behavior (described in section \ref{sec:filters}) verification was also successful. The data filter's task successfully received/generated the data on platforms that used the \texttt{create\_task} parameter set to \texttt{true} (ESP32 and Raspberry Pi Zero W). The correct calculation of the \gls{ema} by the data transformation filter has been presented by the modified level values printed to the console. The data visualization filter's functionality was verified by collecting the level values from the console. The watchdog filter was successfully triggered on all SmartCGMS concept platforms except \gls{wamr}. The reason was that \gls{wamr} support for \glspl{pthread} is only experimental, and attempts to create a separate \gls{pthread} for the watchdog timer were unsuccessful. The SmartCGMS web demo requires the browser to support \texttt{SharedArrayBuffer} for correct functionality of the watchdog timer thread.

SmartCGMS defines multiple kinds of entities that can be used within the framework. The SmartCGMS concept focused on SmartCGMS filters as the top-level entity. Support for other entities was not implemented in the SmartCGMS concept and, therefore, was not tested. The gathered results suggest that the SmartCGMS concept is capable of building a filter chain (potentially with asynchronous filters) and executing events on all supported platforms. Only the most basic functionality of the SmartCGMS concept has been tested. Thorough testing will be required along with further development of the concept.


\section{Memory Usage}

The memory usage of an application for low-power devices is an important characteristic. Low-power devices often have a limited amount of \gls{ram}. When writing an application for a low-power device, it is important to keep the memory requirements as low as possible. On the other hand, predictability is also an important factor, and it may be preferable to have an application with a constant higher memory consumption than an application with very low memory requirements that has sudden peaks of memory usage. In this test, the memory footprint of the SmartCGMS concept application is measured. This test has two objectives. The first objective is to provide insight into how the SmartCGMS concept application manages memory allocations during runtime. The second objective of the test is to compare the SmartCGMS concept's memory footprint compiled as a native application and a \gls{wasm} application.

\subsection{Experimental Setup}

The memory consumption has been measured on four different platforms:

\begin{itemize}
     \item Raspberry Pi Zero W running FreeRTOS (ARMv6)
     \item ESP32 running FreeRTOS (Xtensa)
     \item GNU/Linux (x86-64)
     \item \gls{wamr} (running on GNU/Linux (x86-64))
 \end{itemize}

The first two platforms were used to gather data about how much memory the SmartCGMS concept allocates when building a filter chain and processing events. The last two platforms were used to compare the memory footprint of the native and \gls{wasm} applications.

The same array of event levels from the test scenario in section \ref{subs:correctnes} was used. All four implemented filters were used. In the following order:

\begin{itemize}
    \item \#1 Data reading filter
    \item \#2 Data transformation filter
    \item \#3 Data visualization filter
    \item \#4 Watchdog filter
\end{itemize}

A FreeRTOS \texttt{xPortGetFreeHeapSize} \gls{api} was used to gather the dynamic memory usage on the Raspberry Pi Zero W. The \gls{espidf} provides its own \gls{api} for requesting heap information \texttt{heap\_caps\_print\_heap\_info\(\)}. Both of these functions provide information about the current free heap size. The data from these functions were collected before and after building the filter chain. Then, the heap information was sampled for each second during the execution of the test scenario. The events were injected every two seconds into the filter chain.

To compare the native and \gls{wasm} applications, GNU/Linux was used. Originally, the goal was to use ESP32 for the comparison, but the compiled SmartCGMS \gls{wasm} module was too large for its limited \gls{ram}. On the GNU/Linux platforms, both versions of the SmartCGMS concept application were measured using the same tools. For measuring the heap memory usage, the Valgrind Massif heap profiler tool was used. Valgrind measures how much memory the program allocates from the heap \parencite{valgrind}. The static memory footprint of the SmartCGMS concept application was measured using the \texttt{size} GNU utility. The \texttt{size} tool displays the sizes of each section and the total size of a specified object \parencite{manpage}. The total memory footprint of the application was then computed as a sum of the total size of the executable and the maximum heap usage. For the \gls{wasm} version, both the size of the \gls{wasm} module and the \gls{wasm} runtime were summed. To make the comparison as fair as possible, the SmartCGMS concept application has been compiled using the same build environment. The only difference was that the \gls{wasm} module was compiled using \gls{wasi}\gls{sdk} cross-compile toolchain file. The cross-compile toolchain file specifies the correct compiler, linker, and additional tools required for the compilation to \gls{wasm}. 
 
\subsection{Results and Discussion}

Figure \ref{fig:memory-rpiesp} presents the allocated heap space during the SmartCGMS concept application runtime on the Raspberry Pi Zero W and ESP32 platforms. The actual value of the allocated heap space is not relevant. Rather, the focus is on the shape of the graph lines. The concrete values can change depending on the specific parameters of the filters (e.g., the watchdog task's stack size) and their implementation. From the graph, it can be observed that once the filter chain is built and the first event is injected into the filter chain, the allocated heap space does not change. The constant memory usage is desired on embedded devices. It is achieved by the SmartCGMS framework's device event pool. SmartCGMS implements its own statically allocated pool of device events. When an event is created, SmartCGMS first tries to create it from the event pool, and only if it fails will it allocate the event on the heap.

\begin{figure}[htpb]
  \centering
  \includegraphics[width=1\textwidth]{figures/memory_rpiesp.png}
  \caption{Dynamic memory usage of SmartCGMS concept application executed as a FreeRTOS native application on Raspberry Pi Zero W and ESP32. Only the first four events are displayed since the allocated memory remains constant afterward.} \label{fig:memory-rpiesp}
\end{figure}

The initial rise of allocated memory is caused by the construction of the filter chain. The SmartCGMS concept applications are using dynamic task allocation. The tasks' stacks are therefore allocated from the heap and this causes a relatively big increase in memory usage. While static allocation is generally preferred for programming low-power devices due to its predictability, dynamic allocations occur only during the initialization phase of the application - when configuring the filters in the filter chain. Ensuring the application shuts down safely in case of insufficient memory during the filter chain configuration is relatively easy to control. Compared to encountering a depletion of free heap space during event processing. Regardless, there are not any constraints that would prevent allocating the tasks' stacks statically (FreeRTOS offers the \texttt{xTaskCreateStatic} \gls{api}).

The figure \ref{fig:memory-native} displays the dynamic memory usage of the SmartCGMS concept native application on GNU/Linux. The \gls{wasm} module's memory usage is presented in figure \ref{fig:memory-wasm}. The maximum values are presented in table \ref{tab:memory_footprint}. As the goal was to compare the memory requirements of the two approaches, the exact values are relevant. The heap memory usage of the \gls{wasm} version of the SmartCGMS concept application is approximately 85x higher than the usage of the native version. The main reason is that the \gls{wasm} standalone runtime must load the whole \gls{wasm} module into a buffer, and then additional memory is required to load the module sections and instantiate the module as the operations are not performed in place. The native application's memory usage consists mainly of allocations performed by the standard C++ library. 

\begin{figure}[htpb]
  \centering
  \includegraphics[width=1\textwidth]{figures/memory-native.png}
  \caption{Dynamic memory usage of SmartCGMS concept application executed on GNU/Linux as a native application, only the total memory consumption is displayed. The time displayed on the x-axis is measured in executed instructions.} \label{fig:memory-native}
\end{figure}

\begin{figure}[htpb]
  \centering
  \includegraphics[width=1\textwidth]{figures/memory-wasm.png}
  \caption{Dynamic memory usage of SmartCGMS concept application executed as a \gls{wasm} module in \gls{wamr} on GNU/Linux. The time displayed on the x-axis is measured in executed instructions.} \label{fig:memory-wasm}
\end{figure}


When comparing the static memory footprint of the two versions, the native SmartCGMS application has a smaller footprint. The specific values are presented in table \ref{tab:memory_footprint}. The footprint of the \gls{wasm} module and the \gls{wamr} application combined is almost twice as big as the footprint of the native application. When comparing the total memory footprint of the two approaches (table \ref{tab:memory_footprint}), the SmartCGMS native application's memory footprint is approximately 9x lower than that of the \gls{wasm} version. The results for the \gls{wasm} module were gathered from \gls{wamr} in the fast interpreter mode.


\begin{table}[htbp]
\centering
\caption{Memory Footprint of the SmartCGMS Concept Application}
\label{tab:memory_footprint}
\begin{tabular}{lrr}
\toprule[1.5pt]
& \textbf{WASM} & \textbf{Native} \\
\midrule
Executable Size & 1223KB & 880KB \\
Runtime Size & 360KB & - \\
Maximum Heap Usage & 7085KB & 83KB \\
Total Size & 8668KB & 963KB \\
\bottomrule[1.5pt]
\end{tabular}
\end{table}


\gls{wamr} also offers AOT mode that requires the \gls{wasm} code to be compiled for a specific architecture before it can be loaded and executed. This mode offers increased execution speed as the \gls{jit} LLVM mode but without the added memory overhead of the LLVM compilation during runtime. This mode appears to be the best fit for the anticipated usage of the SmartCGMS concept, but attempts to compile and run it on the GNU/Linux platform were unsuccessful.

As mentioned earlier, the \gls{wasm} memory footprint of the current approach rules out low-power devices with less than 10 MB of \gls{ram} (e.g., ESP32). The \gls{wasm} module size could be further reduced with code optimizations and different compile options, but without a major redesign of the SmartCGMS Concept, it is impossible to fit within the memory limit of ESP32 (320 KB of Data \gls{ram}). 

The \gls{wasm} module was linked with the C/C++ standard library. When compiled with the option \texttt{-nostdlib} (and then using the host environment's C/C++ \texttt{stdlib} implementation), the size of the executable is reduced to 168KB. However, from observing the memory allocation of \gls{wamr} in figure \ref{fig:memory-wasm}, the ESP32 would still be unable to execute the SmartCGMS concept \gls{wasm} module. \gls{wamr} allocates at least twice the memory of the \gls{wasm} module when loading and instantiating it. Alternative runtime targeted at memory-constrained devices (such as the one proposed in \parencite{wait}) might allow execution of the SmartCGMS concept application compiled to \gls{wasm} on the ESP32. The Raspberry Pi Zero W is a viable candidate for the \gls{wasm} version of the SmartCGMS concept application as it has sufficient \gls{ram} (512 MB). At the writing of the project, a ready-to-execute \gls{wasm} standalone runtime was not available for the Raspberry Pi Zero W, and creating a custom port of available \gls{wasm} runtimes was out of the scope of the project. Therefore, it is only a suggestion for further research.


\section{Execution Time}

Execution time might not appear as a critical parameter when considering the use case of the SmartCGMS concept application as an insulin pump controller. However, it is directly related to energy consumption. The more time the device spends processing events, the more energy it consumes. Given that \gls{wasm} can run on embedded devices, a test scenario was created to compare the SmartCGMS concept application compiled to \gls{wasm} for \gls{wamr} and a native application in terms of the time required to process a specific number of events. This test aims to provide more insight into the \gls{wasm} modules versus native applications debate and help steer the future development of the SmartCGMS concept.

\subsection{Experimental Setup}

The execution speed of the SmartCGMS concept application was evaluated on GNU/Linux. A native SmartCGMS concept application and \gls{wasm} module for \gls{wamr} were compared. The tested application consisted of building a filter chain and then executing a specified number of events. The events were created using the level values from the array from previous experiments. The filter chain included these filters:

\begin{itemize}
    \item \#1 Data reading filter
    \item \#2 Data transformation filter
    \item \#3 Watchdog filter
\end{itemize}

The data visualization filter was not used in this evaluation as the I/O operations would probably be the main bottleneck of the execution speed. To obtain accurate results, the event set was iterated multiple times. Different numbers of iterations through the set were measured: 1000, 10 000, and 100 000. The native SmartCGMS concept application was compared to the \gls{wasm} application running in \gls{wamr} in the fast interpreter mode and the LLVM \gls{jit} mode. Each test was performed 10 times, and the times were averaged. Multiple runs of the test should ensure better accuracy of the data by reducing the influence of random variations caused by, for example, cache, I/O operations, memory management, or background processes. The execution time was measured using the \texttt{clock\_gettime} function from the GNU C library, defined in \texttt{time.h} header. The function takes two parameters. The ID of the clock that it should obtain the time from and a pointer to a \texttt{timespec} struct, in which the current time of the clock is returned. A high-resolution clock (\texttt{CLOCK\_PROCESS\_CPUTIME\_ID}) was used for measuring the execution time of the tests. The time required for the execution was computed by subtracting two timestamps. In the native application, the initial timestamp was recorded before building the filter chain, and the ending timestamp was recorded after all the events were processed. The \gls{wasm} modules execution time was measured from \gls{wamr}. The initial timestamp was recorded before calling the main function of the \gls{wasm} module, and the ending timestamp was recorded right after the main function returned. The main function of the \gls{wasm} module was identical to the main function of the native application and consisted only of building the chain and then executing the defined set of events.

\subsection{Results and Discussion}

The averaged results are displayed in Figure \ref{fig:exectime}, with precise numeric values provided in Table \ref{tab:execution_times}. The findings are similar to the ones published in \parencite{wamr_bench}. The execution is approximately 10 times slower in \gls{wamr} fast interpreter mode compared to the native application. The \gls{jit} mode performs the best when the \gls{wasm} module performs a large number of operations. The disadvantage of \gls{jit} mode is that the initialization of the \gls{wasm} module includes its compilation for the target architecture. This creates a large memory and execution time overhead when the module is loaded. From the table \ref{tab:execution_times}, it can be seen that the execution time of the \gls{wasm} module executed in \gls{jit} mode increased in similar increment to the native approach between the two largest number of iterations. The \gls{wamr} \gls{aot} mode could be used as an alternative to the \gls{jit} mode in the SmartCGMS concept. It has the advantage of faster execution with platform-specific code and the compilation overhead does not affect the execution as it is compiled \gls{aot}.

\begin{figure}[htpb]
  \centering
  \includegraphics[width=1\textwidth]{figures/exectime.png}
  \caption{Comparison of the SmartCGMS concept application execution times on different platforms: Native GNU/Linux, WAMR fast interpreter, and WAMR JIT mode. The y-axis displays time on a logarithmic scale.} \label{fig:exectime}
\end{figure}



\begin{table}[htbp]
\centering
\caption{Execution Times of the SmartCGMS concept application on different platforms (in seconds). The top line refers to the number of processed events by the filter chain. }
\label{tab:execution_times}
\begin{tabular}{lccc}
\toprule
\textbf{Execution Type} & $ 10^{5}$  events & $ 10^{6}$  events & $ 10^{7}$  events \\
\midrule
Native Application & 0.14 & 1.41 & 10.38 \\
WAMR Fast Interprer & 1.39 & 14.32 & 145.54 \\
WAMR JIT Mode & 57.67 & 65.87 & 74.94 \\

\bottomrule
\end{tabular}
\end{table}








 

 



