The C File Storage Meta Language (CFSML)
----------------------------------------

Specification of version 0.8


This documentation is provided WITHOUT WARRANTY of any kind.



IMPLEMENTATION NOTES regarding the reference implementation
-----------------------------------------------------------
== READ THIS CAREFULLY ==
The currrent reference implementation stores everything in a text file,
as name/value pairs, with the names given by the variable names specified
in the cfsml file. This means that renaming a variable will break
compatibility with earlier versions, until this is taken care of by
some sort of clever or semi-clever hack.



Purpose of the CFSML
--------------------
It's damn hard to write functions that read and write C records to or from
a text file and keep them in sync when changes are made to that record.
The CFSML strives to help here at least to some extent, by providing
special preprocessor commands that take care of those problems.


Comments and syntax
-------------------
Comments are shell-style; they start with a hash ("#") sign and extend
to the end of the line. Declarations may be followed by semicolons to
keep the EMACS autoformatter satisfied, but this is not required.


CFSML declarations
------------------
Each CFSML file contains one CFSML block, starting with a single line stating
"%CFSML", and ending with a single line stating "%END CFSML". This is the
CFSML declaration block; the cfsml preprocessor will replace it with functions
for saving/reading the data types declared there.
CFSML implementation need not support more than one CFSML declaration block
per file.

The declaration block contains only type declarations and type meta-
declarations. A declaration looks like this:

TYPE [typename] "C representation" USING [writer] [reader];

for example

TYPE string "char *" USING stringwriter stringreader;

to declare the type "string", which is used for char *. The functions
stringwriter and stringreader must have been declared before the CFSML
block, and they must look like this:

void
stringwriter(FILE *fd, char **foo)

and

void
stringreader(FILE *fd, char **foo, char *lastval, int *line, int *hiteof)

Generally, they must take one FILE* and one pointer to a variable of
the specified type as parameters. The reader function also takes
the parameters "char *lastval", a dynamically allocated string containing
the line that was read last and should be interpreted before reading
more data from the file (Reading ahead is required in order to find the
ends of records and arrays).

For example, if something like
foobar = [3][
  1
  2
]

was being parsed (foobar being an int array), then the int reader function
would be called with "1" as a parameter (leading and trailing whitespace is
stripped). It would not need to read anything from the file. (The file
descriptor is passed for multi-line structures like arrays or records).


However, CFSML comes with two built-in variable types: int and
string. This doesn't only remove the need to declare those, it also allows
one special exception to the TYPE declaration: "LIKE xxx" instead of "USING
foo bar",  where xxx is one of those types (int should be most common). This
simply means that the variable is written and read like one of those variables;
CFSML will automagically create reading and writing functions for
them.
Please note that int includes the C type "long int"; you can write and
read "long" variables 'LIKE int'.

Example:
TYPE short "short" LIKE int;

would generate

void
_cfsml_write_short(FILE *fd, short *foo)
{
	...
}

void
_cfsml_read_short(FILE *fd, short *foo, char *lastval, int *line, int *hiteof)
{
	...
}



Records and Arrays
------------------
It is also possible to declare records, like this:

RECORD [recordname] "C representation" EXTENDS [super_record] {
	[member #1];
	[member #2];
	...
}

with the members looking like this:

[declared type] [identifier];

for example:

RECORD baserec {
	int foo;
	int bar;

	int frobmeharder[4];
}

This example introduces two additional features: The possibility to omit the
C representation if it is equal to the identifier, and arrays. Arrays are
used similarly to the way they are treated in C, except that special keywords
must be used to describe dynamic arrays.

Dynamically allocated block arrays are also possible:

RECORD baserec {

	int myarray1[DYNAMIC 4];

	int length;
	int myarray2[DYNAMIC length];
}


Here, length elements of myarray2 will be stored; during restore time,
myarray2 will be set to the result of a malloc(sizeof(int) * length),
and each of the elements will be restored. This is fairly straightforward.

The following keywords can be used inside arrays:
- STATIC/DYNAMIC: Use to define array types. If ommited, static is assumed.
  Dynamic arrays will get the memory they requre allocated during restoration
  time.
- MAXWRITE followed by a int-like variable member of the underlying record
  that is used to limit the number of elements to be written / read.

If the dynamic/static keyword is followed by a string instead of a numeric
value, the length is assumed to be variable and equal to that string token
if interpreted as a member of the underlying structure (see the example if
this sounds too weird).


Here is another example for a dynamically resized stack:

RECORD dynstack {

	int memsize; # Amount of memory allocated
	int tos;     # Top-of stack element

	int data[memsize, MAXWRITE tos+1];

}

This would write only the data up to and including tos+1, unless tos was
negative (in this case, no entries would be written) or greater than
memsize (in that case, only memsize entries would be written).


It is currently required that all elements inside the brackets must be
declared before the record using them.


Records built on top of other records
-------------------------------------
Sometimes, when doing OOish stuff, several separate types share a common
ancestor that defines their contents. In C, this is typically facilitated
by using some kind of macros; CFSML supports this via the EXTENDS keyword.
Example:

RECORD foo {
    int x;
    int y;
}

RECORD bar "extended_bar_t" EXTENDS foo {
    int z;
}

The declaration of bar has the same semantics as the explicit

RECORD bar "extended_bar_t" {
    int x;
    int y;
    int z;
}

but, if foo is changed, bar need not be changed explicitly, making
maintenance easier.


Error reporting during file recovery
------------------------------------
During read time, errors are reported using a built-in function
(_cfsml_error()). This function prints any error reports to stderr.
It is not yet possible to replace that function or to add additional
information (like the file name) to its output. This may be added to
the specifications at a later time.


Pointers
--------
It is possible to store relative pointers. This feature should
be used with care; for portability, only pointers relative to other
pointers pointing to the same data type should be stored.

Example:

RECORD name "struct name" {

	string fullname;
	POINTER firstname RELATIVETO fullname;
	POINTER lastname RELATIVETO fullname;
}

This will only work if firstname and lastname are, indeed, relative to the
string pointed to by "fullname".

Also, absolute pointers are supported. The syntax is similar, but not identical
to the one employed in C:

	int * foo;

(Note the asterisk must be separated from type and variable name by whitespace).
Saving these pointers is relatively straightforward; it will, however, cause
problems if foo points to an invalid memory address. Empty pointers MUST be NULL
for this to work.

While restoring, it is assured that no memory is lost. All allocations are kept
track of, and all memory allocated dynamically is automatically freed if
restoring failed. This holds until the next atomic restoration operation is
encountered.



Reading and writing
-------------------
To read data into a variable of a previously defined data type, use
%CFSMLREAD type dataptr FROM filehandle [ERRVAR err_variable]
	 [FIRSTTOKEN token] [LINECOUNTER counter]
where
	- type is the type of *dataptr
	- dataptr is a pointer to a variable or struct of the specified type
	- filehandle is the file handle of an open and readable FILE* that
	  should be parsed.
	- err_variable is an optional variable which will be set to != 0 if
	  reading was aborted because of any error.
	- token is an optional first token that can be supplied to the reader
	  function; this is useful if you want to use %CFSMLREAD inside a
	  function that might be called by CFSML generated code and passed a
	  token that it wants to pass on
	- counter is an optional variable that will be increased each time a
	  newline is hit. If not supplied, a CFSML generated variable will be
	  used for this task.

For an atomic read, use %CFSMLREAD-ATOMIC. Atomic reads guarantee that, if
they fail, all memory allocated by them will be freed, unless another atomic
read was hit by recursion.
Only after a %CFSMLREAD-ATOMIC command is it guaranteed that the previous
command did not leave any allocated memory. Typically, this call would be
the actual call invoked from the rest of the program, while other %CFSMLREAD
commands would be invoked by recursion.

To write data, use %CFSMLWRITE similarly (with a writeable file
handle, of course):
%CFSMLWRITE type dataptr INTO filehandle



Artificial limitations
----------------------
- Not all data type possibilities can be covered cleanly (like static strings
  or multi-dimensional records).
  Oh, and be careful with doubly-linked lists ;-)


---
This is not yet version 1.0 of the spec, so additions or changes are
possible until the final release. Feel free to suggest improvements;
however, their chance of being accepted is much more likely with code
to back them up.


Changes since 0.7:
- Removed "*" in "[*]", replaced it with the normal unary pointer
  operator
Changes since 0.6:
- Added EXTENDS
- Added "*"
Changes since 0.5.1:
- Added FIRSTTOKEN
- Added LINECOUNTER
Changes since 0.5:
- Renamed "EOFVAR" to "ERRVAR"
Changes since 0.4:
- Added "EOFVAR" construct for %CFSMLREAD.


The author can be contacted at jameson@linuxgames.com.

  Christoph Reichenbach, 01/2000

