Zig -> C Memory matching

Or, how to deal with translate-c's inability to work with bitfields

Disclaimer: Any libraries and technologies discussed in this article may have since gotten updates

tldr; When you try adding a C struct to your Zig project that uses bit fields, it will result in an opaque struct. Such was the case when I tried using ZPL for JSON5 parsing. The base node class zpl_adt_node had a bit field, so I couldn't actually access its members in Zig. The solution is to mimick the memory layout of the original C struct using a combination of the packed and export keywords. Here's a quick writeup as to how I discovered this.

JSON5 sure does exist, huh?

I was looking into ways to parse JSON5 in order to write some tools for GameMaker. Since the project file format and resource format that the engine uses is JSON5 I had no choice but to find a parser that could handle it. Another hidden requirement was that it needed to be able to parse the specific version of JSON5 that GameMaker was using too. So I went looking and found that:

There weren't any Zig libraries that I could find that met my requirements. But that shouldn't be a problem, after all Zig has C interoperability and I won't even need to make the bindings myself. So I broadened my search to include C libraries. It didn't take long before I stumbled into ZPL, and its JSON5 parser.

Getting ZPL to work in Zig

ZPL can be used as a single header library. In fact that's the recommended way to use it as listed on their github. 

curl -L https://zpl.pw/ > zpl.h

Including this library in the project wasn't immediately obvious to me though. I tried just adding the header file as a C source like so: 

// in build.zig
if (target.result.os.tag == .windows) {
    exe.root_module.linkSystemLibrary("ws2_32", .{.needed = true});
}
exe.root_module.addIncludePath(
    .{.src_path = .{.sub_path = "src/lib/zpl", .owner = b}});
exe.root_module.addCSourceFile(
    .{.file = .{.src_path = .{.sub_path = "src/lib/zpl/zpl.c", .owner = b}}});

And then doing a @cImport. Although this seemed to work at first, it resulted in two problems:

  1. Only ZPL declarations were added to the project, not implementations.
  2. zpl_adt_node - which is a generic struct used for various data structure things in the ZPL library - was made opaque. The reason being that translate-c doesn't support the conversion of C bitfields.

Adding the implementations

Issue '1' is caused by the ZPL_IMPLEMENTATION not being defined. Which means that the compiler has no way to include the implementations in the file, they'd be preprocessor-macrod out. There are two ways to fix this.

  1. zig translate-c with a -D flag that defines ZPL_IMPLEMENTATION
  2. Create zpl.c, define ZPL_IMPLEMENTATION, include zpl.h, and add it as a C source file in build.zig

The latter is preferred over the former because the former would would run translate-c on the entire ZPL code. Which is unnecessary since all we need are the bindings.

Anyway, the solution is very simple: 

// zpl.c
#define ZPL_IMPLEMENTATION
#define ZPL_PARSER_DISABLE_ANALYSIS
#include "zpl.h"

Testing if ZPL actually works for me

At this point issue '2' was bugging me. I could immediately spend my time replicating the memory layout of zpl_adt_node in Zig, however I wanted to make sure that ZPL JSON5 can actually parse a YYP file. Yeah sure, I may not be able to allocate an instance of zpl_adt_node in order to receieve data from zpl_json_parse. However, I could just allocate a buffer, cast the pointer, and pass it as if it were a zpl_adt_node. As I'm somewhat fresh to these conscepts, things like these are not always immediately obvious to me. At this level it is certainly cool to be able to think of everything in terms of memory, instead of having to deal with abstract data structures.

So I wrote the following code:

const std = @import("std");
const Allocator = std.mem.Allocator;
const c = @cImport({@cInclude("zpl.h");});
const print = std.debug.print;

fn file_read_to_end_alloc(path: []const u8, alloc: Allocator) ![:0]const u8 {
    const f = try std.fs.cwd().openFile(path, .{});
    return try f.readToEndAllocOptions(
        alloc,
        std.math.maxInt(usize),
        null,
        @alignOf(u8), 0
    );
}

pub fn main() !void {
    const a = std.heap.page_allocator;
    // gamemaker project file containing JSON5
    const k3plus_string = try file_read_to_end_alloc("k3plus.yyp", a);
    const node_buffer = try a.alloc(u8, 32);
    const zpl_a = c.zpl_heap_allocator();
    const err = c.zpl_json_parse(
        @ptrCast(node_buffer.ptr),
        @constCast(@ptrCast(k3plus_string.ptr)),
        zpl_a);
    print("{d}\n", .{err});
}

And indeed, the error code was 0, aka. ZPL_JSON_NO_ERROR Well, it was zero no matter how little memory I allocated for node_buffer. Which is a little concerning considering its probably writing into other parts of memory beyond my buffer without triggering a segmentation fault. My theory was that the page_allocator probably asks for more than 32 bits of memory, even though I only ask for that much. And running the following command in the terminal:

getconf PAGE_SIZE

Writes the number 4096, meaning that it most likely allocated 4096 bytes behind the scenes. Which would make sense given that it's literally called page_allocator.

Idea: just recreate the struct

The solution I came up with for tackling the issue of zpl_adt_note not being translated by translate-c was to just create it myself. Zig has bitfields and packed structs, so I should theoretically be able to recreate the C struct exactly, right?
https://ziglang.org/documentation/0.15.1/#extern-struct

To start out with I added the struct a C file and sizeof'd it:

typedef struct zpl_adt_node {
char const *name;
struct zpl_adt_node *parent;
uint8_t type        :4;
uint8_t props       :4;
union {
    char const *string;
    struct zpl_adt_node *nodes;
    struct {
        union {
            double real;
            double integer;
        };
    };
};
} zpl_adt_node;

The result in the console was 32, which makes sense when compiling for 64bit:

char const* = 8 bytes
struct zpl_adt_node* = 8 bytes
uint8_t :4;
uint8_t :4; = 1 byte + 7 bytes offset to align (8 bytes)
union {
    char const*
    struct zpl_adt_node* struct {union {double real; double integer;};};
}; = 8 bytes

Recreating this in Zig was a matter of using a combination of extern/packed structs/unions.

const ZplAdtNode = extern struct {
    name: [*:0]u8,
    parent: *ZplAdtNode,
    properties: packed struct {
        type: u4,
        props: u4,
    },
    data: extern union {
        string: [*:0]u8,
        nodes: [*]ZplAdtNode,
        value: extern union {
            real: f64,
            integer: f64
        }
    }
};

With this I was able to parse the first two nodes of my JSON5 sample code

var json5 =
    \\{
    \\  "foo": [
    \\    null,
    \\    true,
    \\    false,
    \\    "bar",
    \\    {
    \\      "baz": -13e+37
    \\    }
    \\  ]
    \\}
.*;

pub fn main() !void {
    var node: ZplAdtNode = undefined;
    const node_ptr: *c.zpl_adt_node = @ptrCast(&node);
    const a = c.zpl_heap_allocator();
    _ = c.zpl_json_parse(node_ptr, &json5[0], a);
    _ = @as([*]ZplAdtNode, @ptrCast(node.data.nodes))[0];
}

And I was successfully able to view the contents of zpl_adt_node in Zig. It was enlightening to deal with the glue directly. Having this experience taught me some things about how programming languages are built to communicate with one another. With this working I could finally start using ZPL for real to create the program I wanted.

Sidenote: zpl_array_count() blew my mind when I found out how it works

Thanks for reading. I bet people who are more experienced with this sort of thing are rolling their eyes at how basic this all was. But I found it very exciting to delve into. Learning Zig has been insanely rewarding and it's all thanks to great documentation and readable standard library source code. The latter, has not only inspired me to look deeper into the underlying details but has also taught me various patterns that I would have otherwise learn slowly through trial and error. Kudos to Andrew and the Zig community for that! And that is possibly the most important takeaway: Read source code!!! (but only if it's actually good)

Then I went on to make these two tools: