{"id":14577,"date":"2026-11-01T06:00:35","date_gmt":"2026-11-01T06:00:35","guid":{"rendered":"https:\/\/krikis.lt\/run-ai-models-edge\/"},"modified":"2026-11-01T06:00:35","modified_gmt":"2026-11-01T06:00:35","slug":"run-ai-models-edge","status":"publish","type":"post","link":"https:\/\/krikis.lt\/en\/run-ai-models-edge\/","title":{"rendered":"Run AI Models for Edge Application Development with Just a Few Commands"},"content":{"rendered":"
NVIDIA is expanding the capabilities of AI beyond the data center by providing developers with tools that allow them to quickly build, test, and deploy models on embedded systems. The key tool is NVIDIA TensorRT Edge\u2011LLM, a lightweight C++ inference engine designed for embedded workloads. It runs on NVIDIA DGX Spark as well as other platforms such as Jetson Thor, DRIVE AGX Thor, and IGX Thor. In this article, we\u2019ll look at how a few simple commands allow you to run a Nemotron\u20113.5 Lightning text model directly from the terminal and how you can speed up generation using DSpark speculative decoding.<\/p>\n
NVIDIA DGX Spark as a prototyping platform<\/h2>\n<\/figure>\n
DGX Spark with 64 GB memory configuration provides embedded developers with the ability to prototyping, fine-tuning and testing in a single environment. The platform supports NVIDIA CUDA 13, TensorRT 10.16.1.11 (or later) and Python 3.10-3.12. This software combination ensures that TensorRT Edge-LLM runs stably and can exploit the full potential of the hardware.<\/p>\n
Getting started with trtedge CLI<\/h2>\n<\/figure>\n
The model can be run via trade<\/code> command-line tool (CLI). First, we open a terminal on DGX Spark, create and activate a virtual environment, and then install the TensorRT Edge-LLM package. The virtual environment allows you to isolate newly installed packages, but provides access to all Python packages on the system, including TensorRT.<\/p>\n
After installation, we verify that the TensorRT Edge\u2011LLM runtime is successfully loaded. If compatible TensorRT connections are missing, you should follow the NVIDIA TensorRT pip installation instructions and install the appropriate 10.x version.<\/p>\n
Downloading the model and running it for the first time<\/h2>\n
With the virtual environment active, we start a conversation with Nemotron\u20113.5 Lightning:<\/p>\n
The first run takes longer as the model (about 21 GB) is downloaded and the TensorRT engine is created. Files are stored ~\/.cache\/tensorrt-edgellm<\/code> directory, so you need to ensure sufficient disk space. Subsequent launches use cached data from already created engines, making the process almost instantaneous.<\/p>\n
Launching an API server for prototyping<\/h2>\n
To integrate the model into an application, you can run a local OpenAI-compatible API server:<\/p>\n
The server creates or reuses the engine, and after starting it outputs a JSON response with \""status":"ready"\"<\/code>. Another terminal session can send requests to http:\/\/localhost:8000\/v1\/chat\/completions<\/code>. Responses are returned as streaming JSON events, allowing you to monitor the generation in real time.<\/p>\n
TensorRT Edge\u2011LLM supports DSpark speculative decoding, which uses a draft model to propose multiple tokens to test the main model at once. To enable this feature, stop the server (Ctrl+C) and start it with an additional parameter specifying how many draft tokens to use, e.g. three:<\/p>\n
With DSpark enabled, the measured data shows that the time to first token (TTFT) and the overall token generation rate are significantly improved. Testing on a DGX Spark 64 GB with DSpark shows that the maximum system memory usage is 27.9 GB, including the operating system. This shows that even with the additional speculative logic, the device remains well balanced in terms of memory and performance.<\/p>\n
Practical advice and conclusions<\/h2>\n
\n
Container<\/strong> \u2013 use default ~\/.cache\/tensorrt-edgellm<\/code> directory to avoid repeated downloads.<\/li>\n
Versions<\/strong> \u2013 make sure that the CUDA, TensorRT and Python versions meet the requirements; incompatibility may cause runtime errors.<\/li>\n
DSpark configuration<\/strong> \u2013 experiment with num_speculative_tokens<\/code> and other parameters to find the optimal speed for a specific workload.<\/li>\n
Variety of devices<\/strong> \u2013 TensorRT Edge\u2011LLM works not only on DGX Spark, but also on Jetson, DRIVE, and IGX series, so developers can move their prototype to real edge devices without major changes.<\/li>\n<\/ul>\n
In summary, NVIDIA TensorRT Edge-LLM provides a simple yet powerful means of running AI models on edge devices. A few commands allow for rapid download, creation of an inference engine, launch of an interactive chat, and even access to an API server, while DSpark speculative decoding provides an additional performance advantage. This technology paves the way for a new generation of embedded AI solutions, from driver attention monitoring to robot environment awareness and production line coordination.<\/p>\n
![\"Computer](\"https:\/\/krikis.lt\/wp-content\/uploads\/2026\/11\/run-ai-models-edge-inline-1-37113175.webp\")
<\/figure>\n![\"A](\"https:\/\/krikis.lt\/wp-content\/uploads\/2026\/11\/run-ai-models-edge-inline-2-11700195.webp\")
<\/figure>\n