Lua FFI

20 August 2019 Link



Intro

Trying to learn and standardize the Foreign Function Interface (FFI) mechanism for Lua so I can use it for my programs

Libraries

Here are the Libraries I have found till now:
  1. Alien - Seems to be active and is based on a C library independently maintained so is a good candidate.
  2. luaffi - Facebook FFI library. Little doubtful with Torch moving to Python whether this will be maintained in the future. Already in the facebookarchive account and not taking issues. When I ran the test file I had to comment out the section including the test_stdcall and test_fastcall which I could not figure out. Also the test gave an error. The add_i8 function (first function called gives the wrong result as 1+1=6!). Dibyendu has forked this and done work on it to revive it. His repository is here
  3. luaffi - Not maintained anymore. This is what was forked to form the Facebook FFI library

Installing luaffi Facebook

  1. Just pull the latest code in the project directory in My Programs/C_C++/ffi/github
  2. Compile the new code with the code blocks project.

Installing Alien

  • Need to install libffi first. To do that follow the instructions on its github repository page here. I run the commands from the mingw32 terminal from the MSYS2 package. Don't put it in a path which has spaces. Basically in the terminal go to the directory and run:
    • ./configure
    • make
  • After that install Alien using luarocks after putting the libffi dll at a particular path/lib (name it libffi.dll) and executing:
luarocks install alien FFI_DIR=C:/Path/to/the/libffi/dll/file(minus lib)
Note that libffi.dll should be placed in the directory C:/Path/to/the/libffi/dll/file/lib.
  • This will generate an error saying:
Installing https://luarocks.org/alien-0.7.1-2.src.rock
./bootstrap && ./configure LUA=lua.exe CPPFLAGS='-IC:/Users/mgupta/Documents/Programs/luarocks/include -IC:/Users/mgupta/Documents/Programs/luarocks/include' LDFLAGS=-LC:/Users/mgupta/Documents/Programs/luarocks/lib --prefix=c:\users\mgupta\documents\programs\luarocks\/lib/luarocks/rocks/alien/0.7.1-2 --libdir=c:\users\mgupta\documents\programs\luarocks\/lib/luarocks/rocks/alien/0.7.1-2/lib --data
dir=c:\users\mgupta\documents\programs\luarocks\/lib/luarocks/rocks/alien/0.7.1-2/lua && make clean && make
'.' is not recognized as an internal or external command,operable program or batch file.

Error: Build error: Failed building.
  • Since the command is in linux format. So now I will run this same command from the MSYS2 mingw32 terminal
  • To do that
    • Download the alien github project in a directory with no spaces in the path
    • Place the lib files (libffi.dll and lua53.dll) in a subdirectory lib. Also place lua.exe here
    • Place the lua include files in a subdirectory inc
    • Place ffi.h and ffitarget.h (from libffi compile) in a subdirectory ffi inside the inc directory
    • Run the following commands:
PATH=$PATH:/c/Path/To/lib/directory/lib     # To point to lua.exe
./bootstrap
./configure LUA=lua.exe CPPFLAGS='-IC:/Path/To/inc/directory/inc -IC:/Path/To/inc/directory/inc/ffi' LDFLAGS=-LC:/Path/To/lib/directory/lib
  • Configure generated the config.h file. Note this is the reason we have to go this route and we cannot have simple saved configuration in CodeBlocks since config.h referred from alien.c is generated using these steps
  • From here I was trying to run make clean and make as described below in my unsuccessful attempts.
  • So I just created a codeblocks project used the config.h file generated from the above steps and compiled it using codeblocks and it worked!
  • The dll should be named alien_c
  • Also libffi.dll should be in lua executable directory for the module to load successfully.
NOT ABLE TO COMPILE WITH THE FOLLOWING PROCEDURE
Now ran the following commands
make clean
make LDFLAGS='-no-undefined'         # Giving error about undefined reference to all lua functions
  • The option of LDFLAGS='-no-undefined' came after getting an error without that option and getting the guidance from here

No success doing this. Kept getting lua C API functions undefined reference error.

Alien API functions

Here
alien = require("alien")
dll = alien.load("dllname")
func1 = dll.func1
func2 = dll.func2
buf = alien.buffer(byte_size)
arr = alien.array(type,length)
struc = alien.defstruct(tab)
st = struc:new(pointer)

General API

  • alien.version - returns the version of the alien library, for example "alien library for Lua 5.3 / 0.7.1"
  • alien.platform - Can be 1 of the following:
    • linux
    • bsd
    • darwin
    • unknown
    • windows
  • alien.memmove - Works like the C function
  • alien.memset - Works like the C function
  • alien.sizeof(type) - Returns the size of types alien supports
  • alien.align(type) - Returns the structure alignment information
  • alien.tag("tag") - creates a new metatable with tag tag if one does not exist, or returns the metatable with this tag. The namespace for tags is global, so a good pattern is to prefix the tag name with the name of your module (such as mymod_mytag)
  • alien.wrap("tag",...) - creates a full userdata, tags it with the metatable associated with tag, stores the values you passed, then returns the full userdata. Valid values are nil, integers and other userdata.
  • alien.rewrap("tag",obj,...) - replaces the elements on obj with new values. If you pass more values than obj had previously the extra values are silently ignored. If you pass less then obj is filled with nil.
  • alien.unwrap("tag",obj) - tests if obj is tagged with tag, throwing an error if it is not, then returns the values previously stored in it.
  • alien.isnull
  • alien.table - lua_createtable exposed.
  • alien.errno - Returns the errno which several operating system function return on errors.
  • alien.byval
  • alien.loaded - Table of all the libraries loaded by this alien session
  • alien.pack - From the modified struct library
  • alien.unpack - From the modified struct library
  • alien.size - From the modified struct library
  • alien.offset - From the modified struct library


Dynamic Library API

  • alien.load(<library name>) - Loads the library if not loaded and returns the library reference that can be used to access its functions
  • alien.default contains the reference to the currently running module
  • func1:types(ret_type, arg_types...) or func1:types{ret=ret_type,abi=abi_type,arg_types...} - Defines the function types for use in Lua. types are one of the following strings:
    • "void"
    • "byte" = signed char
    • "char"
    • "short"
    • "ushort"
    • "int"
    • "uint"
    • "long"
    • "ulong"
    • "ptrdiff_t"
    • "size_t"
    • "float"
    • "double"
    • "string" = const char*
    • "pointer" = void* - So can be usd to represent any pointer
    • "ref char" = reference of char = char* - Can be used for pass by reference as shown in the scanf example here
    • "ref int" = reference of int = int*
    • "ref uint" = reference of unit = unit*
    • "ref double" = reference of double = double*
    • "longlong"
    • "ulonglong"
    • "callback" = generic function pointer
  • alien.funcptr(function_pointer) - Converts the pointer to a function a alien function that can be called after defining its types.

Buffers API

  • alien.buffer(byte_size) - Allocates a new buffer of BUFSIZE if byte_size is nil.
  • alien.buffer(pointer) - Allocates a new buffer and sets its base pointer to pointer. The size in this case is 0.
  • #buf - Returns size of buffer = byte_size
  • buf:strlen() - Returns the result of calling strlen on the buffer
  • buf[i] - Accesses the i-th character (byte) of the buffer for getting or setting the value
  • buf:get(offset,type) - Get the value of type at the offset in the buffer buf. The offset is in bytes. For the type in the beginning the offset is 1. This does not do bound checking.
  • buf:set(offset,val,type) - Set the value of type at the offset in the buffer buf. This does not do bound checking.
  • buf:tostring(len,offset) - Retrieve part of a buffer as string. Both arguments optional.
  • buf:topointer(offset) - Get a pointer to a buffer. offset is optional defaulting to 1.
  • buf:tooffset(pointer) - Turn a pointer into a buffer offset.
  • buf:realloc(newsize) - Re allocate a buufer to a new size

Arrays API

This is just a abstraction done in the alien.lua file over the Buffers API
  • alien.array(type,length) - Creates array of type elements of length number of elements
  • alien.array(type, tab) - Creates an array as the same length as tab and initializes it with its values
  • alien.array(type,length,buf) - Creates an array with buf as the underlying buffer. Can be useful to map an pointer to an array returned by a library to an array abstraction in alien.
  • arr.type - Contains the type of the elements of the array
  • arr.length - Number of elements the array has
  • arr.size - The size of the array in bytes
  • arr.buffer - The underlying buffer that the array is abstracting
  • arr[i] - Accesses the i-th element to get or set it.
  • arr:ipairs() - Iterate over the array's contents
  • arr:realloc(newlen) - Resize the array to the new length

Structs API

This is just a abstraction done in the alien.lua file over the Buffers API
  • alien.defstruct(tab) - Creates a new struct type. Here tab is a table describing the structure components in order. Each component described by a sub table containing the field name and type at indices 1 and 2
  • struc:new(pointer/buffer) - Creates a new instance of the structure defined in struc. If pointer is given then uses that pointer to map the struct instance to that memory location.
  • st() - Returns the underlying buffer the struct is abstracting.
  • struc.<key> - Attributes about structure. Here key can be the following:
    • names - table containing names of all the components
    • offsets - table containing offsets for each component in the structure buffer
    • types - table containing types of all the components
    • size - size of the entire structure

Callback API

  • alien.callback(<Lua Function>,ret_type,arg_types...) or alien.callback(<Lua Function>,{ret=ret_type,abi=abi_type,arg_types...}) - Returns a callback object that can be passed as an argument of callback type.

Pointer API

  • alien.tostring(pointer) - Converts location pointed to a Lua string and returns it
  • alien.tochar(pointer)
  • alient.toptrdiff_t(pointer)
  • alien.tosize_t(pointer)
  • alien.to<type>(pointer) - Converts location pointed to a Lua number and returns it. type can be:
    • int
    • uint
    • short
    • ushort
    • long
    • ulong
    • float
    • double