Configure and Initialize Python.NET in a C# Console App on Linux containerized with Docker #2296

Was able to get mine working, albeit by bloating my container size probably. For my situation, ultimately I found that when PythonNet tries to load the DLL set by by Runtime.PythonDLL, it calls into a win32 interop lib (kernel32) to call a LoadLibrary function, which returns an empty pointer. The Win32Exception I shared in my last post (error code = 126) stated that "The specified module could not be found" which can indicate a number of things:

DLL is not there on specified path or
DLLs is not supported with platform or
dependent dlls are not loaded before you build your native DLL

I created a small powershell script to quickly test this precise issue when loading the python312.dll, so for the most part excluding PythonNet out of the reproducible steps.

.\TestLoadingDLL.ps1 "C:\python\python312.dll"

$dllToLoad = $args[0]

if ($dllToLoad -eq "" -or !(Test-Path $dllToLoad)) {
    throw "File doesn't exist: $dllToLoad"
}

$MethodDefinition = @"
private const string NativeDll = "kernel32.dll";

[DllImport(NativeDll, SetLastError = true)]
public static extern IntPtr LoadLibrary(string dllToLoad);
"@

$Kernel32 = Add-Type -MemberDefinition $MethodDefinition -Name 'Kernel32' -Namespace 'Win32' -PassThru

$ptr = $Kernel32::LoadLibrary($dllToLoad);
$lastError = [System.Runtime.InteropServices.Marshal]::GetLastWin32Error()

Write-Host $ptr

if ($ptr -eq 0)
{
    Write-Error $lastError

    throw new-object System.ComponentModel.Win32Exception -ArgumentList $lastError
}

Out of curiosity, I loaded up a windows container that already had python installed (python:3.12.1-windowsservercore-1809), and ran the above powershell script against the python dll in it's default install directory, and it worked no issue. So probably what is happening here is that the python installer that runs during the building of that python docker image (shown here), installs some system level dependencies that are required when loading that python DLL, which are simply just missing when I'm using the embedded python runtime. Looking through the python docs under this embedded section, it references that you must make sure to have the Microsoft C Runtime installed. It also states that if the ucrtbase.dll exists in the system directory, then it is already installed, which it was for the initial servercore image I tried, so unsure if it was related. Running python.exe under either image worked w/o issue, so this was a specific issue to loading that DLL via that Kernel32::LoadLibrary() function.

Ultimately my problems stemmed from utilizing the embed python runtime as I mentioned in my last post, so what I ended up doing was using the python docker image as my base and installing the dotnet runtime via this dotnet-install script. This allowed me to move forward with actually getting PythonNet to start. There were several other headaches I had to deal with such as modifying the PATH environment var for dotnet, as well as figuring out how and what to set PythonEngine.PythonPath (it didn't behave the same as my local dev machine). But after several hours of tweaking I did finally get it all working (basically have the python runtime installed twice though).

Dockerfile

FROM python:3.12.1-windowsservercore-1809

SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

RUN &powershell -NoProfile -ExecutionPolicy unrestricted -Command "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; &([scriptblock]::Create((Invoke-WebRequest -UseBasicParsing 'https://dot.net/v1/dotnet-install.ps1'))) -Runtime dotnet -Version 8.0.0"

RUN [ "powershell", "-command", "[Environment]::SetEnvironmentVariable(\"Path\", $env:Path + \";C:\\Users\\ContainerAdministrator\\AppData\\Local\\Microsoft\\dotnet\", \"Machine\")" ]

# Set the working directory
WORKDIR /app

ADD MyProject /app/

ENTRYPOINT ["dotnet", "MyProject.dll"]

C# PythonNet Startup

var pythonDir = Path.Combine(Environment.CurrentDirectory, @"python");

// hardcoded for now, but can probably be made to a bit more dynamic if we search through the `python` output directory
var pyVer = "312";

Runtime.PythonDLL = Path.Combine(pythonDir, $"python{pyVer}.dll");
PythonEngine.PythonHome = pythonDir;

// Mimicking what was in `sys.path` when running on local dev machine (different when running under docker container for some reason)
string[] dirs = [
    $"python\\python{pyVer}.zip",
    "python\\DLLs",
    "python\\Lib",
    "",
    "python",
    "python\\Lib\\site-packages"
];

var pythonPaths = string.Join(';', dirs.Select(e => Path.Combine(Environment.CurrentDirectory, e)));

PythonEngine.PythonPath = pythonPaths;

PythonEngine.Initialize();

This is far from the most optimal way to resolve this, but it gets the job done for now.

Sorry @jimnoble, I don't think any of this applies to your situation since my issue seemed like a pretty windows specific issue. I initially thought our two issues were identical due to the DllNotFoundException, however after digging into it, it was really the Win32Exception that was the main indicator on my issue. For your issue, if you're able to paste the full stack trace, then perhaps we can reason through to a solution.

I have the same issue with the PYTHONNET_PYDLL path setup. I've installed the python3.11. The pythonnet works well in windows after I specified the dll path to C:\Users\103296\AppData\Local\Programs\Python\Python311\Python311.dll in the system environment variables.

I tried to find the equivalent library file in docker environment. As the document mentioned, I found the libpython3.11.so in the folder begin with "config", I've specified the PYTHONNET_PYDLL environment variable in the docker file.
ENV PYTHONNET_PYDLL=/usr/lib/python3.11/config-3.11-x86_64-linux-gnu/libpython3.11.so.

Unfortunately, the same error still occurred.

Unhandled exception. Python.Runtime.BadPythonDllException: Runtime.PythonDLL was not set or does not point to a supported Python runtime DLL. See https://github.com/pythonnet/pythonnet#embedding-python-in-net
2024-02-02T21:02:36.526974682Z ---> System.MissingMethodException: Failed to load symbol Py_IncRef: dotnet: undefined symbol: Py_IncRef
2024-02-02T21:02:36.526980888Z at Python.Runtime.Platform.PosixLoader.GetFunction(IntPtr dllHandle, String name)
2024-02-02T21:02:36.526984945Z at Python.Runtime.Runtime.Delegates.GetFunctionByName(String functionName, IntPtr libraryHandle)
2024-02-02T21:02:36.526988900Z --- End of inner exception stack trace ---
2024-02-02T21:02:36.526992497Z at Python.Runtime.Runtime.TryUsingDll[T](Func`1 op)

I really appreciate if someone can provide any help.