Skip to content

Pipelines: read, change, build or write ​

A pipeline connects a reader to a tree builder or a writer, optionally with stages in between that change the events passing through. Pipelines cover building a tree to query, writing a new file, and copying a document with content removed or altered on the way.

ts
import { 
StringSink
,
XMLOutputFactory
} from '@exemplarjs/writer';
import {
EventType
,
XMLInputFactory
} from '@exemplarjs/xml';
import {
createFilteredReader
} from '@exemplarjs/xml/filters';
const
xml
= new
XMLInputFactory
();
const
source
= '<catalog><!-- draft --><book id="b1"><title>Dune</title></book></catalog>';
// Read... const
reader
=
xml
.
createStreamReader
(
source
);
// ...change... const
withoutComments
=
createFilteredReader
(
reader
, (
r
) =>
r
.
eventType
() !==
EventType
.
COMMENT
);
// ...write. A StringSink collects the output as a string. // The indent option lays the output out, one element to a line. const
sink
= new
StringSink
();
const
writer
= new
XMLOutputFactory
({
indent
: ' ' }).
createStreamWriter
(
sink
);
writer
.
copy
(
withoutComments
);
console
.
log
(
sink
.
toString
());
xml
<?xml version="1.0" encoding="UTF-8"?>
<catalog>
  <book id="b1">
    <title>Dune</title>
  </book>
</catalog>

No stage here holds the whole document. Each event is read, tested and written before the next is read, so memory use does not grow with the size of the input.

This page is an overview of what connects to what. The linked pages cover each part.

The three parts ​

Every pipeline has the same three parts, and the example above has one of each.

  1. A reader. Here it is the cursor. It could equally be a SAX parser or an event reader, the other two streaming APIs from Choosing how to read.
  2. An optional change. Here it is createFilteredReader, which takes one or more predicates and reports an event only if every predicate returns true for it. Without this part, the pipeline is a plain read or a plain copy.
  3. An ending, which consumes the events and builds a result. Here it is a writer; it could be a tree builder instead.

The parts are independent: the ending can be replaced without touching the change, and the reverse.

Swap the ending ​

Passing the same filtered reader to a tree builder produces a tree with no comments, ready to query with XPath:

ts
import { 
XdmDocumentBuilder
} from '@exemplarjs/xdm';
import {
XPathCompiler
} from '@exemplarjs/xpath';
const
catalog
= new
XdmDocumentBuilder
().
parse
(
withoutComments
);
const
comments
= new
XPathCompiler
().
compile
('count(//comment())');
console
.
log
(
comments
.
evaluateNumber
(
catalog
));
// 0

A DOM builder, a SAX parser and canonicalize, which writes canonical XML, accept a reader in the same way. Every connection below lists the calls.

Each of them consumes from the reader's current position. At the start of a document, that is the whole document. At a start tag, it is that one element, and the reader is left at its end tag, which is how a large file is processed one record at a time.

INFO

Every ending checks its input. If a change produces events that do not form a document, such as an end tag with no start tag or two root elements, the ending throws at that event: a tree builder an XMLError, a writer a WriterError. It never produces a malformed tree or file.

Choose how to make the change ​

A filter can only drop events. Each reading API has its own mechanisms for changing them, with different capabilities. Choose by the change required.

ChangeUse
Drop comments, elements or other contentA cursor filter: the least code
Change text, attribute values or namesA delegate
Add attributesA SAX filter
Add, move or duplicate elements and textA generator over event objects, or a SAX filter
Make many scattered changes to a document that fits in memoryBuild a DOM tree, change it, and save it
Make no change at allPass the source straight to the builder or parser

A delegate wraps a cursor and overrides selected accessors. A SAX filter sits between the parser and a handler: it receives each callback and forwards it, modified or not. A generator function takes event objects and yields the events to pass on: the originals, replacements or additions.

Every connection ​

Each reading API reaches the same endings through its own calls. Every row is a supported connection. In the calls, xdmBuilder is an XdmDocumentBuilder, domBuilder a DocumentBuilder, saxParser a SAXParser and output an XMLOutputFactory, which creates writer and eventWriter.

FromChange withIntoWith
A cursorFilters, delegatesAn XDM tree, for XPathxdmBuilder.parse(reader)
A DOM treedomBuilder.parse(reader)
Written XMLwriter.copy(reader)
Canonical XMLcanonicalize(reader)
A SAX handlersaxParser.parse(reader, handler)
SAX callsSAX filtersAn XDM tree, for XPathxdmBuilder.createSAXHandler()
A DOM treedomBuilder.createSAXHandler()
Written XMLoutput.createSAXHandler(destination)
Event objectsA generator, or event filtersAn XDM tree, for XPathxdmBuilder.build(events)
A DOM treedomBuilder.build(events)
Written XMLeventWriter.add(events)

SAX filters and generators need an ending too. SAX filters and Changing events with a generator show theirs.

Mix the ways of reading ​

The reading APIs can be combined. A SAX parser and an event reader can both read from a cursor, so a filter or delegate applied to the cursor carries through:

ts
const 
reader
=
createFilteredReader
(
xml
.
createStreamReader
(
source
),
noComments
);
// Event objects from a filtered cursor: const
events
=
xml
.
createEventReader
(
reader
);
// Or SAX calls from one (a reader can be read only once, so choose one): // new SAXParser(xml).parse(reader, handler);

A job can then be split: a cursor filter drops the unwanted content, and a SAX filter or a generator adds the new attributes or elements.

When the input is a stream ​

Every connection above has an asynchronous form, for a ReadableStream or async iterable source such as an HTTP response body. The structure of the code is the same.

FromStart withInto
A cursorxml.createAsyncStreamReader(stream)await builder.parseAsync(reader), await writer.copyAsync(reader), await canonicalizeAsync(reader), await saxParser.parseAsync(reader, handler)
SAX callsawait saxParser.parseAsync(stream, handler)The same handlers as before
Event objectsxml.createAsyncEventReader(stream)await builder.buildAsync(events), await eventWriter.addAsync(events)

See Large files and streams for the asynchronous cursor.

When writing to a file or a network stream, finish with await writer.close(). Its promise settles when the stream has accepted all of the writer's output.

Next steps ​

The rest of the guide covers one API at a time, and each group can be read independently.

Released under the Apache License 2.0.