Qwen3.5-2B NPU

User:<image>Describe the image.

Answer: This is a surreal, digitally manipulated image that blends science fiction with everyday life in an unexpected setting. Here’s a detailed description:

Main Subject:

• A person wearing a white astronaut suit reclines on what appears to be a lunar surface or spacecraft interior.
• Their legs are propped up against the edge of a green cooler box, which has faint graffiti-style writing (“SOMETHING”) visible on its side.
• In their right hand, they hold a bright green glass beer bottle — likely a craft beer — as if enjoying a drink in zero gravity.

Setting & Background:

• The environment is clearly extraterrestrial: gray, cratered terrain resembling the Moon’s surface.
• Behind the astronaut, Earth dominates the sky — its blue oceans and swirling white clouds are visible against the black void of space dotted with stars.
• To the right, part of a metallic ladder or structure leans into the frame, suggesting this might be inside a spacecraft or on a lunar module.

Atmosphere & Style:

• The lighting is dramatic — strong highlights on the astronaut’s helmet visor and suit, contrasting sharply with the dark background.
• There’s a sense of irony or absurdity: an astronaut relaxing with beer next to Earth, defying gravity and normal human behavior in space.
• The image has a hyper-realistic yet dreamlike quality, enhanced by digital effects like lens flares and atmospheric glow around Earth.

Overall Impression: It’s a humorous, thought-provoking piece that plays on the contrast between isolation in space and comfort found in mundane pleasures — drinking beer while floating above our own planet. It invites reflection on human resilience, curiosity, and the strange ways we adapt to extreme environments.

Paper: Qwen3 Technical Report

LLMs (Large Language Models) are neural networks trained on large text datasets to understand and generate language.
VLMs (Vision-Language Models) add a visual encoder so the model can process images and text together.
A combined VLM+LLM system is often referred to as a multimodal model.

These models can be large—hundreds of millions to billions of parameters—which impacts accuracy, memory use, and runtime speed.
On edge devices like the RK3588, available RAM and compute are limited, and even the NPU has strict constraints on supported operations.
Because of this, models typically need to be quantised or simplified to fit.

Performance is usually expressed in tokens (words) per second.
Once converted to RKNN, parts of the model can run on the NPU, improving speed.
Despite these limits, models like Qwen3-2B run well on the RK3588 because the NPU efficiently accelerates the heavy math, and the vision encoder can be optimised. This makes advanced multimodal AI feasible on small, power-efficient devices.

All models, with C++ examples, can be found on the Q-engineering GitHub.

All LLM models are quantized to w8a8, while the VLM vision encoders use fp16.

¹ The total used memory; LLM plus the VLM.
² When an llm/vlm model is loaded for the first time from your disk to RAM or NPU, it is called a cold start.
The duration depends on your OS, I/O transfer rate, and memory mapping.
³ Subsequent loading (warm start) takes advantage of the already mapped data in RAM. Mostly, only a few pointers need to be restored.

To run the application, you have to:

• OpenCV 64-bit installed.
• rkllm library.
• rknn library.
• Optional: Code::Blocks. ($ sudo apt-get install codeblocks)

Start with the usual

$ sudo apt-get update 
$ sudo apt-get upgrade
$ sudo apt-get install cmake wget curl

To install OpenCV on your SBC, follow the Raspberry Pi 4 guide.

Or, when you have no intentions to program code:

$ sudo apt-get install libopencv-dev 

$ git clone https://github.com/Qengineering/Qwen3-VL-2B-NPU

To run InternVL3, you need to have the rkllm-runtime library version 1.3.0 installed, as well as the rknpu driver version 0.9.8.
If you don't have these on your machine, or if you have a lower version, you need to install them.
We have provided the correct versions in the repo.

$ cd ./Qwen3-VL-2B-NPU/aarch64/library
$ sudo cp ./*.so /usr/local/lib
$ cd ../include
$ sudo cp ./*.h /usr/local/include

Your rkllm model must match the library.



If you use a model synthesized with the previous 1.2.3 rkllm library and run it with the latest 1.3.0, you will get a malfunction. The internal Byte-Pair Encoding (BPE) dictionary parsing gets misaligned.

The next step is downloading the models.
Both can be downloaded from our Hugging Face page.

• qwen3.5-2b_w8a8_rk3588.rkllm
• qwen3.5-2b-vision_rk3588.rknn

Copy both into this folder.

Once you have the two models, it is time to build your application.
You can use Code::Blocks.

• Load the project file *.cbp in Code::Blocks.
• Select Release, not Debug.
• Compile and run with F9.
• You can alter command line arguments with Project -> Set programs arguments...

Or use Cmake.

$ mkdir build
$ cd build
$ cmake ..
$ make -j4

The application switches dynamically between Single Image Mode and Video Sequence Mode based on how many image files you pass into the arguments.

./VLM_NPU RKNN_model RKLLM_model file1.jpg [file2.jpg file3.jpg ...]

In the context of the Rockchip RK3588 LLM (Large Language Model) library, NewTokens and ContextLength control the boundaries for text generation and memory allocation.

In main.cpp you will find the line:

RKLLM.LoadModel(vlm_model, llm_model, NewTokens, ContextLength);

Here you set your context based on available memory.

NewTokens This sets the maximum number of tokens (pieces of text, typically sub-word units) that the model is allowed to generate in response to a prompt during a single inference round. For example, if set to 300, the model will not return more than 300 tokens as output, regardless of the prompt length. It is important for controlling generation length to avoid run-on responses and manage resource use.

ContextLength (Dynamic KV Cache) This specifies the maximum total number of tokens the model can hold in its memory at once, which includes the system prompt, the massive image/video embeddings, your text questions, and all previous generated answers.
We have synthesized the models with a larger KV Cache than normal. Ours can hold up to 16384 tokens!

• For 32GB Boards (e.g., Rock 5C 32GB): You can safely push the KV Cache to 8192 or 16384 to support processing long video sequences and maintaining deep, multi-turn conversations without the model forgetting the image. RKLLM.LoadModel(vlm_model, llm_model, 2048, 16384);

• For 8GB/16GB Boards: The KV cache is highly memory-intensive. You should keep this at 2048 or 4096. If you set this higher than your physical RAM can handle, the Linux Out-Of-Memory (OOM) killer will crash the application. RKLLM.LoadModel(vlm_model, llm_model, 2048, 4096);

Typical Command Line Examples:

Single Image Mode:

./VLM_NPU ./models/qwen3.5-2b-vision_rk3588.rknn ./models/qwen3.5-2b_w8a8_rk3588.rkllm ./frame1.jpg 

Video Sequence Mode (Passing multiple frames):

./VLM_NPU ./models/qwen3.5-2b-vision_rk3588.rknn ./models/qwen3.5-2b_w8a8_rk3588.rkllm ./frame1.jpg ./frame2.jpg ./frame3.jpg

To process video input, individual frames are first extracted. The VLM converts each frame into embeddings, which are then transformed into vision tokens.
Even on a desktop PC, this process places a heavy load on memory and CUDA resources. It’s therefore no surprise that the Rock 5C, with its more limited hardware, struggles even more.
Each vision token corresponds to about 200 tokens, strongly limiting the number of frames processed per video. Therefore, the video is subsampled, and evenly spaced frames are extracted for processing by Qwen3.5.
For reference, each vision token occupies around 20 MB of RAM — a detail worth keeping in mind when working on systems with limited memory.
Keep also in mind that after loading the models, each frame has to be tokenised, which takes about 0.9 seconds per frame.

Using the application is simple. Once you provide the model and the media files, you can ask anything you want.
Remember, we are on a bare Rock 5C, so don't expect the same speed or quality as massive server-grade models like ChatGPT. On the other hand, as you will see in the examples below, the app performs amazingly well on the edge!

Interacting with Media:

• If you passed a single image and want to talk about it, you must include the <image> tag in your prompt. (e.g., "Describe this <image> in detail.")
• If you passed multiple images (video sequence), you must use the <video> tag instead. (e.g., "What action is happening in this <video>?")

Chat Controls:

• The app remembers the dialogue context continuously. To wipe the model's memory and start a fresh conversation about the loaded media, type clear.
• To leave the application, type exit.

Below, you find the surprisingly little code of main.cpp.

#include "RK35llm.h"
#include <vector>

int main(int argc, char** argv)
{
    // Usage: ./VLM_VIDEO_NPU vlm_model llm_model frame1.jpg [frame2.jpg frame3.jpg ...]
    if (argc < 4) {
        std::cerr << "Usage: " << argv[0] << " vlm_model llm_model file1.jpg [file2.jpg file3.jpg ...]\n"; 
        return -1;
    }

    std::string vlm_model = argv[1];
    std::string llm_model = argv[2];

    RK35llm RKLLM;
    RKLLM.SetInfo(true);
    RKLLM.SetSilence(false);

    RKLLM.LoadModel(vlm_model, llm_model, 2048, 16384);

    // Collect all image frames from arguments
    std::vector<cv::Mat> frames;
    for (int i = 3; i < argc; ++i) {
        cv::Mat frame = cv::imread(argv[i]);
        if (!frame.empty()) {
            frames.push_back(frame);
        } else {
            std::cerr << "Warning: Could not load image file: " << argv[i] << "\n";
        }
    }

    if (frames.empty()) {
        std::cerr << "Error: No valid images loaded. Exiting.\n";
        return -1;
    }

    // Dynamic routing: single image vs. video sequence
    if (frames.size() == 1) {
        std::cout << "\n[Info] Loading single image mode...\n";
        RKLLM.LoadImage(frames[0]);
    } else {
        std::cout << "\n[Info] Loading video sequence mode (" << frames.size() << " frames)...\n";
        RKLLM.LoadVideoFrames(frames);
    }

    std::string input_str;
    std::string output_str;

    while (true) {
        printf("\nUser: ");
        std::getline(std::cin, input_str);
        if (input_str == "exit") break;

        // Reminder: the user must input the correct tag (<image> or <video>) based on the mode!
        output_str = RKLLM.Ask(input_str);
//        std::cout << "\nLLM Reply: " << output_str << std::endl;    }

    return 0;
}

Most code speaks for itself. One remark.
The LLM generates the answer to your request in little pieces of text. You see them as if the LLM is typing on your terminal.
RKLLM.SetSilence controls this behaviour. When set to true, it will suppress this output. RKLLM.Ask() also returns the complete answer.
In main.cpp uncomment this line:

std::cout << "\nLLM Answer: " << output_str << std::endl;

Image	Dialog
	User: <image>Which city is this? Answer: This is Singapore. The image shows a famous view of Singapore’s skyline at dusk or dawn, featuring: - The Marina Bay Sands hotel (with its distinctive three-tower structure and sky park) on the left. - The Singapore Flyer, one of the world’s largest observation wheels, to the right. - A calm body of water in the foreground reflecting the city lights — likely Marina Bay or Gardens by the Bay. These landmarks are iconic symbols of Singapore, a city-state known for its modern architecture, blend of heritage and innovation, and vibrant urban life. The long exposure effect on the water gives it a smooth, dreamy quality often seen in travel photography. ✅ So, this is definitely Singapore.
	User: <image>Tell me something about the history of this place. Answer: The image you’ve shared depicts a section of the Great Wall of China, specifically in what appears to be the Northwestern Frontier region — likely near Tianjin, Hebei Province, or possibly further north toward Shandong. This area is historically significant for several reasons: 🏛️ Historical Context & Significance 1. Strategic Importance of the Northern Border The Great Wall was built primarily to protect China from invasions by nomadic tribes such as the Xiongnu, Yuan, and later the Mongols. The northern frontier faced constant threats, so fortifications like this one were essential for defense. 2. Location: Tianjin/Hebei Border This particular stretch of wall is located along the border between modern-day Tianjin (China) and Manchuria (Russia). It was part of the ancient Liaodong Peninsula campaign, where Chinese forces attempted to hold off Russian expansion into Manchuria during the late Qing Dynasty. In 1860, after the Opium Wars, Russia had already begun encroaching on Chinese territory in Manchuria. The Great Wall here served as a symbolic and practical barrier against further invasion — though it was never fully completed or maintained due to resource constraints and shifting geopolitical priorities. 3. Architectural Style & Construction The wall shown is built of brick and stone, with watchtowers, gatehouses, and ramparts typical of Ming Dynasty construction. The use of local materials (clay bricks) reflects adaptation to the region’s geography. Some sections show signs of later modifications or repairs, indicating its long service over centuries. 4. Cultural & Symbolic Role Beyond military function, the Great Wall became a powerful symbol of Chinese sovereignty and resilience. It was also used as a border marker between China and Russia for much of the 20th century — even after formal treaties ended direct territorial disputes in Manchuria by the early 1950s. 📜 Key Historical Events Nearby - Late Qing Dynasty (1860–1890s): The wall was used to resist Russian advances into Northeast China. - Sino-Soviet Border Disputes: During the Cold War, both sides claimed parts of this border region, leading to tension and occasional skirmishes along the Great Wall. - Modern Era: Today, much of the original structure has been preserved as a tourist attraction or historical site, though some sections have been restored for public access. 🌿 Natural Setting & Preservation The lush green hillsides surrounding the wall suggest this area was once part of fertile agricultural land — ideal for sustaining large populations and supporting military operations. Today, conservation efforts focus on preserving these historic structures while allowing visitors to experience their grandeur safely. 💡 Fun Fact There’s a famous legend that says the Great Wall was built by Emperor Gaozu of Han (202 BC) — though historians debate whether this is fact or folklore. Regardless, the wall’s enduring presence testifies to its strategic value and cultural importance. If you’re visiting this location today, it’s a great place to see how history meets nature — standing as a silent guardian of centuries past against the backdrop of rolling mountains and clear skies. Let me know if you’d like more details about specific eras or nearby sites!
	User: <image>What are the ingredients I need to make this? Answer: To make a pizza like the one in the image — which appears to be a classic Margherita pizza with a rustic, charred crust and fresh basil — you’ll need these core ingredients: 🍕 Core Ingredients: 1. Pizza Dough (or pre-made dough) - Flour (all-purpose or bread flour) - Water - Salt - Optional: yeast (for leavened dough), or store-bought pizza dough > Tip: For authentic Neapolitan-style crust, use a simple 2:1 ratio of flour to water and let it rise naturally. 2. Tomato Sauce - San Marzano tomatoes (preferably canned in their own juice) - Garlic (optional but common) - Olive oil - Salt & pepper > Note: You can also make your own sauce from fresh tomatoes, or use a pre-made marinara. 3. Mozzarella Cheese - Fresh mozzarella (preferably buffalo mozzarella for stretchiness) - Or part-skim mozzarella if fresh isn’t available >Tip: Mozzarella should be sliced into rounds and placed on top before baking — it melts beautifully. 4. Fresh Basil Leaves - Fresh basil leaves (optional, but essential for this style) >Place them gently over the cheese after baking to preserve their color and aroma. Optional Additions: - Olive oil – drizzled on top before baking or brushed onto dough - Garlic – minced garlic added to sauce or dough - Extra virgin olive oil – for brushing the crust during baking (especially important for that charred look) Optional Toppings: - Ham, salami, pepperoni → not in this image - Mushrooms, onions, peppers → also not present ✅ So, to summarize — you need: Flour, water, salt, tomato sauce, mozzarella cheese, fresh basil leaves, and optionally garlic or olive oil. This combination creates the simple yet delicious Margherita-style pizza shown in your image! 🍅🧀🌿 User:What is the first step? Answer:The first step to making this pizza — especially if you’re starting from scratch with dough — is: ✅ 1. Make or Prepare the Pizza Dough Start by mixing your flour, water, and salt (and optionally yeast or baking powder) in a bowl. Knead until smooth, then let it rest for 30–60 minutes to relax the gluten before shaping. > 💡 Pro tip: For authentic Neapolitan-style crust, use minimal kneading — just enough to form a ball — and let it rise naturally at room temperature. 🔄 Then proceed with: - Shaping the dough into a round (or oval) - Preheating your oven (especially if using wood-fired or stone oven) - Adding sauce, cheese, basil - Baking until crust is golden and charred But since you asked for the very first step, it’s always: > 🥣 Mix and knead the dough. Let me know if you want a recipe or tips for any part!
	User: <image>Which movie is this? Answer: This image is from the 2017 film Titanic, directed by James Cameron. The scene features Leonardo DiCaprio as Jack Dawson and Kate Winslet as Rose DeWitt Bukater standing on the bow of the ship Titanic during a dramatic sunset or sunrise. Their poses — arms outstretched, embracing tightly — capture the emotional climax of their romance aboard the sinking vessel. The warm lighting and ocean backdrop are iconic to this pivotal moment in the movie’s narrative. While it could be mistaken for other romantic dramas like The Notebook, the specific costumes (19th-century style), the ship’s rigging, and the cinematic composition strongly point to Cameron’s adaptation of Stephenie Meyer’s novel. ✅ Movie: Titanic (2017) 🎬 Director: James Cameron 💥 Iconic scene: Jack and Rose on the bow at sunset

User: <video> Describe the video.
Answer: The video captures a dynamic scene inside what appears to be a martial arts dojo or training hall, characterized by light blue padded walls adorned with framed calligraphy and posters. The floor is covered in large, green mats typical for such environments.

In the foreground, an individual dressed in dark traditional attire—likely a black top and pants—is captured mid-motion. They are executing a powerful, acrobatic kick, possibly a high roundhouse or spinning kick, which creates significant motion blur due to the speed of the movement. Their body is twisted, with one leg extended high into the air while their other arm reaches out for balance.

In the background, another person wearing similar dark clothing stands upright near the wall, observing the action. This observer appears relatively still compared to the fast-moving practitioner. The overall atmosphere suggests a demonstration of martial arts skills, focusing on agility and technique within a controlled training space.

Notice the nice Markdown makeup of the answer!

User: Can you make me a C++ call that embeds the Pythagorean theorem?

Answer: Certainly! Here’s a clean, modern C++ function that computes the length of the hypotenuse using the Pythagorean theorem: a² + b² = c², so c = √(a² + b²).

#include <iostream>
#include <cmath>  // for std::sqrt

// Function to compute the hypotenuse given two legs of a right triangle
double getHypotenuse(double a, double b) {
    return std::sqrt(a * a + b * b);
}

int main() {
    double leg1 = 3.0;
    double leg2 = 4.0;

    double hypotenuse = getHypotenuse(leg1, leg2);

    std::cout << "Legs: " << leg1 << ", " << leg2 << std::endl;
    std::cout << "Hypotenuse: " << hypotenuse << std::endl;

    return 0;
}

• We use std::sqrt() to compute the square root of the sum of squares.
• The function is modular and reusable — you can pass any two positive numbers as legs, and it will return the hypotenuse.
• In main(), we test with classic 3-4-5 triangle values for clarity.

Legs: 3, 4
Hypotenuse: 5

Let me know if you’d like to extend this — e.g., add input validation, handle floating-point precision issues, or visualize the result!

To get a taste, try our professional Qwen3 AI-chatbot running on a Rock 5C: https://rock5gpt.qengineering.eu