path: root/utils/docs/lib/binary.wiki

All platforms must support two types for interacting with binary data: ByteArray and ByteString.  The ByteArray type resembles the interface of Array in that it is mutable, extensible, and indexing will return number values for the byte in the given position, zero by default, or undefined if the index is out of bounds.  The ByteString type resembles the interface of String in that it is immutable and indexing returns a ByteString of length 1.  These types are exported by the 'binary' top-level module and both types are subtypes of Binary, which is not instantiable but exists only for the convenience of referring to both ByteArray and ByteString.  (The idea of using these particular two types and their respective names originated with Jason Orendorff in the [http://groups.google.com/group/serverjs/msg/89808c05d46b92d0 Binary API Brouhaha] discussion.)

= Philosophy =

This proposal is not an object oriented variation on pack and unpack with notions of inherent endianness, read/write head position, or intrinsic codec or charset information.  The objects described in this proposal are merely for the storage and direct manipulation of strings and arrays of byte data.  Some object oriented conveniences are made, but the exercise of implementing pack, unpack, or an object-oriented analog thereof are left as an exercise for a future proposal of a more abstract type or a 'struct' module (as mentioned by Ionut Gabriel Stan on [http://groups.google.com/group/serverjs/msg/592442ba98c6c70e the list]).  This goes against most mentioned [[ServerJS/Binary|prior art]].

This proposal also does not provide named member functions for any particular subset of the possible charsets, codecs, compression algorithms, or consistent hash digests that might operate on a byte string or array.  Instead, convenience member functions are provided for interfacing with any named charset, with the IANA charset name space, and with the possibility of eventually employing a system of modular extensions for other codecs or digests, requiring that the given module exports a specified interface. (As supported originally by Robert Schultz, Davey Waterson, Ross Boucher, and tacitly myself, Kris Kowal, on the [http://groups.google.com/group/serverjs/browse_thread/thread/be72ef3d8146731d/06c27162b698eef5?lnk=gst First proposition] thread on the mailing list).  This proposal does not address the need for stream objects to support pipelined codecs and hash digests (mentioned by Tom Robinson and Robert Schultz in the same conversation).

This proposal also reflects both group sentiment and a pragmatic point about properties.  This isn't a decree that properties like "length" should be consistently used throughout the ServerJS APIs.  However, given that all platforms support properties at the native level (to host String and Array objects) and that byte strings and arrays will require support at the native level, pursuing client-side interoperability is beyond the scope of this proposal and therefore properties have been specified.  (See comments by Kris Zyp about the implementability of properties in all platforms, comments by Davey Waterson from Aptana about the counter-productivity of attempting to support this API in browsers, and support properties over accessor and mutator functions by Ionut Gabriel Stand and Cameron McCormack on the [http://groups.google.com/group/serverjs/browse_thread/thread/be72ef3d8146731d/06c27162b698eef5?lnk=gst mailing list]).

The byte types provide functions for encoding, decoding, and transcoding, but they are all shallow interfaces that defer to a charset manager module, and may in turn use a system level charset or use a pair of pure JavaScript modules to transcode through an array or stream of canonical Unicode code points.  This behavior may be specified further in the future.

= Specification =

The "binary" top-level module must export "Binary", "ByteArray" and "ByteString".


== ByteString ==

A ByteString is an immutable, fixed-width representation of a C unsigned char (byte) array.  ByteString supports the String API, and indexing returns a byte substring of length 1.

=== Constructor ===

; ByteString()
: Construct an empty byte string.
; ByteString(byteString)
: Copies byteString.
; ByteString(byteArray)
: Use the contents of byteArray.
; ByteString(arrayOfNumbers)
: Use the numbers in arrayOfNumbers as the bytes.
: If any element is outside the range 0...255, an exception (''TODO'') is thrown.
; ByteString(string, charset)
: Convert a string. The ByteString will contain string encoded with charset.

=== Constructor methods ===

; join(array, delimiter)
: Like Array.prototype.join, but for Binarys. Returns a ByteString.

=== Instance properties ===

; length
: The length in bytes. Immutable.

=== Instance methods (in prototype) ===

; toByteArray()
: Returns a byte for byte copy in a ByteArray.
; toByteArray(sourceCharset, targetCharset)
: Returns a transcoded copy in a ByteArray.
; toByteString()
: Returns itself, since there's no need to copy an immutable ByteString.
; toByteString(sourceCharset, targetCharset)
: Returns a transcoded copy.
; toArray()
: Returns an array containing the bytes as numbers.
; toArray(charset)
: Returns an array containing the decoded Unicode code points.
; toString()
: Returns a debug representation like "[ByteString 10]", where 10 is the length of the Array. Alternative debug representations are valid too, as long as (1) this method will never fail, (2) the length is included.
; decodeToString(charset)
: Returns the decoded ByteArray as a string.
; indexOf(byte)
; indexOf(byte, start)
; indexOf(byte, start, stop)
: Returns the index of the first occurance of byte (a Number or a single element ByteString or ByteArray) or -1 if none was found. If start and/or stop are specified, only elements between the the indexes start and stop are searched.
; lastIndexOf(byte)
; lastIndexOf(byte, start)
; lastIndexOf(byte, start, stop)
: Returns the index of the last occurance of byte (a Number or a single element ByteString or ByteArray) or -1 if none was found. If start and/or stop are specified, only elements between the the indexes start and stop are searched.
; charCodeAt(offset)
; <u>get(offset)</u>
: Return the byte at offset as a Number.
; byteAt(offset) ByteString
; charAt(offset) ByteString
: Return the byte at offset as a ByteString.
; split(delimiter, [options])
: Split at delimiter, which can by a Number, a ByteString, a ByteArray or an Array of the prior (containing multiple delimiters, i.e., "split at any of these delimiters"). Delimiters can have arbitrary size.
: Options is an optional object parameter with the following optional properties:
* ''count'' - Maximum number of elements (ignoring delimiters) to return. The last returned element may contain delimiters.
* ''includeDelimiter'' - Whether the delimiter should be included in the result.
: Returns an array of ByteStrings.
; slice()
; slice(begin)
; slice(begin, end)
: See [https://developer.mozilla.org/en/Core_JavaScript_1.5_Reference/Global_Objects/Array/slice Array.prototype.slice]


; substr(start)
:
; substr(start, length)
:
; substring(first)
:
; substring(first, last)

; [] ByteString
: the immutable [] operator returning ByteStrings
; toSource()
: which would return "ByteString([])" for a null byte string

ByteString does not implement toUpperCase() or toLowerCase() since they are not meaningful without the context of a charset.

== ByteArray ==

A ByteArray is a mutable, flexible representation of a C unsigned char (byte) array.

=== Constructor ===

; ByteArray()
: New, empty ByteArray.
; ByteArray(length)
: New ByteArray filled with length zero bytes.
; ByteArray(byteArray)
: Copy byteArray.
; ByteArray(byteString)
: Copy contents of byteString.
; ByteArray(arrayOfBytes)
: Use numbers in arrayOfBytes as contents.
: Throws an exception if any element is outside the range 0...255 (''TODO'').
; ByteArray(string, charset)
: Create a ByteArray from a Javascript string, the result being encoded with charset.

Unlike the Array, the ByteArray is not variadic so that its initial length constructor is not ambiguous with its copy constructor.

All values within the length of the array are numbers stored as bytes that default to 0 if they have not been explicitly set.  Assigning beyond the bounds of a ByteArray implicitly grows the array, just like an Array.  Retrieving a value from an index that is out of the bounds of the Array, lower than 0 or at or beyond the length, the returned value is "undefined".  Assigning an index with a value that is larger than fits in a byte will be implicitly and silently masked against 0xFF.  Negative numbers will be bit extended to a byte in two's complement form and likewise masked.

=== Instance properties ===

; mutable length property
: extending a byte array fills the new entries with 0.

=== Instance methods (in prototype) ===

; toArray()
: n array of the byte values
; toArray(charset)
: an array of the code points, decoded
; toString()
: A string debug representation like "[ByteArray 10]". Alternative debug representations are valid too, as long as (1) this method will never fail, (2) the length is included.
; decodeToString(charset)
: returns a String from its decoded bytes in a given charset.
; toByteArray()
: just a copy
; toByteArray(sourceCharset, targetCharset)
: transcoded
; toByteString()
: byte for byte copy
; toByteString(sourceCharset, targetCharset)
: transcoded
; <u>byteAt(offset) ByteString</u>
: Return the byte at offset as a ByteString.
; <u>get(offset) Number</u>
: Return the byte at offset as a Number.
; concat(other:ByteArray|ByteString|Array)
; pop() byte:Number
:
; push(...variadic Numbers...) -> count(Number)
:
; <u>extendRight(...variadic Numbers / Arrays / ByteArrays / ByteStrings ...)</u>
: 
; shift() byte:Number
:
; unshift(...variadic Numbers...) count:Number
:
; <u>extendLeft(...variadic Numbers / Arrays / ByteArrays / ByteStrings ...)</u>
: 
; reverse() in place reversal
:
; slice()
:
; sort()
:
; splice()
:
; <u>indexOf()</u>
:
; <u>lastIndexOf()</u>
:
; <u>split()</u> 
: Returns an array of ByteArrays instead of ByteStrings.
; <u>filter()</u>
:
; <u>forEach()</u>
:
; <u>every()</u>
:
; <u>some()</u>
:
; <u>map()</u>
:
; <u>reduce()</u>
:
; <u>reduceRight()</u>
:
; <u>displace(begin, end, values/ByteStrings/ByteArrays/Arrays...) -> length</u>
: begin/end are specified like for slice. Can be used like splice but does not return the removed elements.
; toSource()
: Returns a string like "ByteArray([])" for a null byte-array.
; [] Number
: The mutable [] operator for numbers

== String ==

The String prototype will be extended with the following members:

; toByteArray(charset)
: Converts a string to a ByteArray encoded in charset.
; toByteString(charset)
: Converts a string to a ByteString encoded in charset.
; charCodes()
: Returns an array of Unicode code points (as numbers).

== Array ==

The Array prototype will be extended with the following members:

; toByteArray(charset)
: Converts an array of Unicode code points to a ByteArray encoded in charset.
; toByteString(charset)
: Converts an array of Unicode code points to a ByteString encoded in charset.

== General Requirements ==

None of the specified prototypes or augmentations to existing prototypes are enumerable.

<u>Any operation that requires encoding, decoding, or transcoding among charsets may throw an error if that charset is not supported by the implementation.  All implementations MUST support "us-ascii" and "utf-8".</u>

Charset strings are as defined by IANA http://www.iana.org/assignments/character-sets.

<u>Charsets are case insensitive.</u>

= Tests =

* [http://github.com/tlrobinson/narwhal/tree/master/tests/serverjs ServerJS tests] compatible with [http://github.com/tlrobinson/narwhal/tree/master/lib/test this test framework].

= Relevant Discussions =

* [http://groups.google.com/group/serverjs/browse_thread/thread/f8ad81201f7b121b ByteArray and ByteString proposal]
* [http://groups.google.com/group/serverjs/browse_thread/thread/a8d3a91af37fd355 ByteArray: byteAt method]
* [http://groups.google.com/group/serverjs/browse_thread/thread/45190ac89d7b422a Binary/B Extension Proposals and Implementation Notes]