Compare commits

..
26 Commits
Author SHA1 Message Date
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 2acec9c0dc Bump @types/node from 17.0.45 to 18.0.2 (#140)
Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 17.0.45 to 18.0.2.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-06 19:04:46 +02:00
Jimmy Wärting 20ee08f33e feat: new way of creating temporary files backed up by fs (#137)
* feat: new way of creating temporary files backed up by fs

* add readme
2022-07-06 19:01:17 +02:00
Jimmy Wärting f077471a23 fix: avoid reading files when the size is empty (#134)
* fix: avoid reading files when the size is empty

* also use stat.size for checking if it is modified

* bump minor version number

* Don't push empty chunks into the array (creates faster reads)

* reverted back to avoid making a if check
2022-03-10 15:11:10 +01:00
Jimmy Wärting dc29588366 fix: misc updates (#132)
* misc updates

* bring back c8 for test coverage

* use new loader

* don't use engines

* simplify ci

* fix double slice problem
2022-01-20 14:00:15 +01:00
Jimmy Wärting 8ab587d340 removed xo 2021-10-30 13:52:13 +02:00
Jimmy Wärting f61ee01955 bumped minor patch 2021-10-30 13:50:21 +02:00
Jimmy Wärting 5dbcbb91b4 standard.js style 2021-10-30 13:50:21 +02:00
Jimmy Wärting c5c2b52154 fix(warning): Supress warnings when importing experimental web stream from NodeJS (#125)
* Suppress warnings when importing web stream
* added node: prefix
* updated changelog
* apply `standard streams.cjs --fix`
2021-10-30 13:04:02 +02:00
Ben McCann 836709fc65 fix: prefix stream/web import with node: (#122) 2021-10-24 17:27:58 +02:00
Nick K 2c11647222 fix(blob, file, tests): Minor improvements for Blob and File implementations (#120)
* Use private class fields in parts initialization directly.

* Move encoder initialization outside the loop

* Improve errors reported by Blob constructor.

* Simulate WebIDL type casting for lastModified in File constructor.

* Add cancel method for Blob.stream() underlying source.

* Fix indentation inconsistency.

* Use getReader in blob stream cancellation test instead of Symbol.asyncIterator
2021-10-24 14:56:15 +02:00
Jimmy Wärting 9022f8fc70 Update index.js 2021-09-29 17:14:07 +02:00
Jimmy Wärting 0e07457246 Added v12 to test 2021-09-07 17:16:24 +02:00
Jimmy Wärting 46476faf49 Update CHANGELOG.md 2021-07-31 11:30:20 +02:00
Jimmy Wärting b86b1409de found/fixed some errors while testing agains WPT (#109)
* found/fixed some errors while testing agains WPT
* only run test.js with ava
2021-07-31 11:24:15 +02:00
Jimmy Wärting a08f394f1b Create PULL_REQUEST_TEMPLATE.md 2021-07-31 11:08:57 +02:00
Nick K c2903ca530 Allow usage of iterable object in Blob constructor. (#108)
* Allow usage of iterable object in Blob constructor.

* Fix ESLint errors
2021-07-29 15:42:42 +02:00
Jimmy Wärting 0b028432d0 bumped patch 2021-07-16 11:59:39 +02:00
Jimmy Wärting f64c20694f added streams.cjs 2021-07-16 11:58:08 +02:00
Jimmy Wärting 613160ba58 bumped patch 2021-07-16 11:57:08 +02:00
Jimmy Wärting efdad69698 bumped patch 2021-07-16 11:55:56 +02:00
Jimmy Wärting e00f849f6d improve typing (#105)
* improve typing

* bug fix

* don't test cjs
2021-07-16 11:51:34 +02:00
Jimmy Wärting 52f296c283 Conditional loading (#104)
use good old cjs to conditionally load stuff in node only
2021-07-16 00:33:00 +02:00
Jimmy Wärting 701103ffd2 Update CHANGELOG.md 2021-07-15 22:55:57 +02:00
Jimmy Wärting ada23da6ee Update package.json 2021-07-15 22:50:37 +02:00
Jimmy Wärting 16f312c543 use whatwg streams (#103)
* use whatwg streams
* could not use optional changing in v12
* could not use fs/promise in v12
2021-07-15 22:49:58 +02:00
Jimmy Wärting 5ba554efd2 Lowered targeted node version to those who support ESM
closes #98
2021-06-19 11:51:54 +02:00
15 changed files with 1023 additions and 736 deletions
+24
View File
@@ -0,0 +1,24 @@
<!-- Thanks for contributing! -->
## The purpose of this PR is:
...
## This is what had to change:
...
## This is what like reviewers to know:
...
-------------------------------------------------------------------------------------------------
<!-- Mark what you have done, Remove unnecessary ones. Add new tasks that may fit (like TODO's) -->
- [ ] I prefixed the PR-title with `docs: `, `fix(area): `, `feat(area): ` or `breaking(area): `
- [ ] I updated ./CHANGELOG.md with a link to this PR or Issue
- [ ] I updated the README.md
- [ ] I Added unit test(s)
-------------------------------------------------------------------------------------------------
<!-- Add a `- fix #_NUMBER_` line for every Issue this PR solves. Do not comma separate them -->
- fix #000
+5 -29
View File
@@ -14,42 +14,18 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macOS-latest]
node: ["16", "15", "14", engines]
exclude:
# On Windows, run tests with only the LTS environments.
- os: windows-latest
node: engines
- os: windows-latest
node: "14"
# On macOS, run tests with only the LTS environments.
- os: macOS-latest
node: engines
- os: macOS-latest
node: "14"
node: ["17.3"]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v2
- name: Get Node.JS version from package.json
if: matrix.node == 'engines'
id: get-version
run: echo ::set-output name=node::$(npx --q minimum-node-version)
- uses: actions/setup-node@v2-beta
if: matrix.node != 'engines'
- uses: actions/setup-node@v2
with:
node-version: ${{ matrix.node }}
- uses: actions/setup-node@v2-beta
if: matrix.node == 'engines'
with:
node-version: ${{steps.get-version.outputs.node}}
node-version: '17.3'
- run: npm install
- run: npm test
- run: npm run report -- --colors
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1
with:
-20
View File
@@ -1,20 +0,0 @@
name: CI
on:
pull_request:
paths:
- "**.js"
- "**eslint**"
- "package.json"
- ".github/workflows/lint.yml"
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Use Node.js
uses: actions/setup-node@v2-beta
with:
node-version: 14
- run: npm install
- run: npm run lint
+6 -2
View File
@@ -16,6 +16,8 @@ pids
*.seed
*.pid.lock
# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov
@@ -47,8 +49,8 @@ typings/
# Optional npm cache directory
.npm
# Optional eslint cache
.eslintcache
# Optional cache
.cache
# Optional REPL history
.node_repl_history
@@ -63,3 +65,5 @@ typings/
.env
*.d.ts
*.d.cts
.DS_Store
+28 -2
View File
@@ -1,5 +1,27 @@
Changelog
=========
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/) and this
project adheres to [Semantic Versioning](http://semver.org/).
## v3.1.3
- Allow usage of iterable object in Blob constructor. [#108]
- Run test WPT test against our impl [#109]
- File name are now casted to string [#109]
- Slicing in the middle of multiple parts added more bytes than what what it should have [#109]
- Prefixed `stream/web` import with `node:` to allow easier static analysis detection of Node built-ins [#122]
- Added `node:` prefix in `from.js` as well [#114]
- Suppress warning when importing `stream/web` [#114]
## v3.1.2
- Improved typing
- Fixed a bug where position in iterator did not increase
## v3.1.0
- started to use real whatwg streams
- degraded fs/promise to fs.promise to support node v12
- degraded optional changing to support node v12
## v3.0.0
- Changed WeakMap for private field (require node 12)
@@ -70,3 +92,7 @@ Changelog
## v1.0.0
- Major: initial release
[#108]: https://github.com/node-fetch/fetch-blob/pull/108
[#109]: https://github.com/node-fetch/fetch-blob/pull/109
[#114]: https://github.com/node-fetch/fetch-blob/pull/114
+35 -47
View File
@@ -22,10 +22,8 @@ npm install fetch-blob
- internal Buffer.from was replaced with TextEncoder/Decoder
- internal buffers was replaced with Uint8Arrays
- CommonJS was replaced with ESM
- The node stream returned by calling `blob.stream()` was replaced with a simple generator function that yields Uint8Array (Breaking change)
(Read "Differences from other blobs" for more info.)
All of this changes have made it dependency free of any core node modules, so it would be possible to just import it using http-import from a CDN without any bundling
- The node stream returned by calling `blob.stream()` was replaced with whatwg streams
- (Read "Differences from other blobs" for more info.)
</details>
@@ -36,48 +34,12 @@ npm install fetch-blob
- This blob version is more arbitrary, it can be constructed with blob parts that isn't a instance of itself
it has to look and behave as a blob to be accepted as a blob part.
- The benefit of this is that you can create other types of blobs that don't contain any internal data that has to be read in other ways, such as the `BlobDataItem` created in `from.js` that wraps a file path into a blob-like item and read lazily (nodejs plans to [implement this][fs-blobs] as well)
- The `blob.stream()` is the most noticeable differences. It returns a AsyncGeneratorFunction that yields Uint8Arrays
- The `blob.stream()` is the most noticeable differences. It returns a WHATWG stream now. to keep it as a node stream you would have to do:
The reasoning behind `Blob.prototype.stream()` is that NodeJS readable stream
isn't spec compatible with whatwg streams and we didn't want to import the hole whatwg stream polyfill for node
or browserify NodeJS streams for the browsers and picking any flavor over the other. So we decided to opted out
of any stream and just implement the bear minium of what both streams have in common which is the asyncIterator
that both yields Uint8Array. this is the most isomorphic way with the use of `for-await-of` loops.
It would be redundant to convert anything to whatwg streams and than convert it back to
node streams since you work inside of Node.
It will probably stay like this until nodejs get native support for whatwg<sup>[1][https://github.com/nodejs/whatwg-stream]</sup> streams and whatwg stream add the node
equivalent for `Readable.from(iterable)`<sup>[2](https://github.com/whatwg/streams/issues/1018)</sup>
But for now if you really need a Node Stream then you can do so using this transformation
```js
import {Readable} from 'stream'
const stream = Readable.from(blob.stream())
```
But if you don't need it to be a stream then you can just use the asyncIterator part of it that is isomorphic.
```js
for await (const chunk of blob.stream()) {
console.log(chunk) // uInt8Array
}
```
If you need to make some feature detection to fix this different behavior
```js
if (Blob.prototype.stream?.constructor?.name === 'AsyncGeneratorFunction') {
// not spec compatible, monkey patch it...
// (Alternative you could extend the Blob and use super.stream())
let orig = Blob.prototype.stream
Blob.prototype.stream = function () {
const iterator = orig.call(this)
return new ReadableStream({
async pull (ctrl) {
const next = await iterator.next()
return next.done ? ctrl.close() : ctrl.enqueue(next.value)
}
})
}
}
```
Possible feature whatwg version: `ReadableStream.from(iterator)`
It's also possible to delete this method and instead use `.slice()` and `.arrayBuffer()` since it has both a public and private stream method
</details>
## Usage
@@ -100,12 +62,8 @@ const blob = new Blob(['hello, world'])
await blob.text()
await blob.arrayBuffer()
for await (let chunk of blob.stream()) { ... }
// turn the async iterator into a node stream
stream.Readable.from(blob.stream())
// turn the async iterator into a whatwg stream (feature)
globalThis.ReadableStream.from(blob.stream())
blob.stream().getReader().read()
blob.stream().getReader({mode: 'byob'}).read(view)
```
### Blob part backed up by filesystem
@@ -129,6 +87,34 @@ console.log(blob.size) // ~4 GiB
`blobFrom|blobFromSync|fileFrom|fileFromSync(path, [mimetype])`
### Creating a temporary file on the disk
(requires [FinalizationRegistry] - node v14.6)
When using both `createTemporaryBlob` and `createTemporaryFile`
then you will write data to the temporary folder in their respective OS.
The arguments can be anything that [fsPromises.writeFile] supports. NodeJS
v14.17.0+ also supports writing (async)Iterable streams and passing in a
AbortSignal, so both NodeJS stream and whatwg streams are supported. When the
file have been written it will return a Blob/File handle with a references to
this temporary location on the disk. When you no longer have a references to
this Blob/File anymore and it have been GC then it will automatically be deleted.
This files are also unlinked upon exiting the process.
```js
import { createTemporaryBlob, createTemporaryFile } from 'fetch-blob/from.js'
const req = new Request('https://httpbin.org/image/png')
const res = await fetch(req)
const type = res.headers.get('content-type')
const signal = req.signal
let blob = await createTemporaryBlob(res.body, { type, signal })
// const file = createTemporaryBlob(res.body, 'img.png', { type, signal })
blob = undefined // loosing references will delete the file from disk
```
`createTemporaryBlob(data, { type, signal })`
`createTemporaryFile(data, FileName, { type, signal, lastModified })`
### Creating Blobs backed up by other async sources
Our Blob & File class are more generic then any other polyfills in the way that it can accept any blob look-a-like item
An example of this is that our blob implementation can be constructed with parts coming from [BlobDataItem](https://github.com/node-fetch/fetch-blob/blob/8ef89adad40d255a3bbd55cf38b88597c1cd5480/from.js#L32) (aka a filepath) or from [buffer.Blob](https://nodejs.org/api/buffer.html#buffer_new_buffer_blob_sources_options), It dose not have to implement all the methods - just enough that it can be read/understood by our Blob implementation. The minium requirements is that it has `Symbol.toStringTag`, `size`, `slice()` and either a `stream()` or a `arrayBuffer()` method. If you then wrap it in our Blob or File `new Blob([blobDataItem])` then you get all of the other methods that should be implemented in a blob or file
@@ -146,3 +132,5 @@ See the [MDN documentation](https://developer.mozilla.org/en-US/docs/Web/API/Blo
[install-size-image]: https://flat.badgen.net/packagephobia/install/fetch-blob
[install-size-url]: https://packagephobia.now.sh/result?p=fetch-blob
[fs-blobs]: https://github.com/nodejs/node/issues/37340
[fsPromises.writeFile]: https://nodejs.org/dist/latest-v18.x/docs/api/fs.html#fspromiseswritefilefile-data-options
[FinalizationRegistry]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/FinalizationRegistry
+31 -18
View File
@@ -1,36 +1,49 @@
import Blob from './index.js';
import Blob from './index.js'
export default class File extends Blob {
#lastModified = 0;
#name = '';
const _File = class File extends Blob {
#lastModified = 0
#name = ''
/**
* @param {*[]} fileBits
* @param {string} fileName
* @param {{lastModified?: number, type?: string}} options
*/ // @ts-ignore
constructor(fileBits, fileName, options = {}) {
*/// @ts-ignore
constructor (fileBits, fileName, options = {}) {
if (arguments.length < 2) {
throw new TypeError(`Failed to construct 'File': 2 arguments required, but only ${arguments.length} present.`);
throw new TypeError(`Failed to construct 'File': 2 arguments required, but only ${arguments.length} present.`)
}
super(fileBits, options);
super(fileBits, options)
const modified = Number(options.lastModified);
this.#lastModified = Number.isNaN(modified) ? Date.now() : modified
this.#name = fileName;
if (options === null) options = {}
// Simulate WebIDL type casting for NaN value in lastModified option.
const lastModified = options.lastModified === undefined ? Date.now() : Number(options.lastModified)
if (!Number.isNaN(lastModified)) {
this.#lastModified = lastModified
}
this.#name = String(fileName)
}
get name() {
return this.#name;
get name () {
return this.#name
}
get lastModified() {
return this.#lastModified;
get lastModified () {
return this.#lastModified
}
get [Symbol.toStringTag]() {
return "File";
get [Symbol.toStringTag] () {
return 'File'
}
static [Symbol.hasInstance] (object) {
return !!object && object instanceof Blob &&
/^(File)$/.test(object[Symbol.toStringTag])
}
}
export { File };
/** @type {typeof globalThis.File} */// @ts-ignore
export const File = _File
export default File
+134 -73
View File
@@ -1,54 +1,98 @@
import {statSync, createReadStream} from 'fs';
import {stat} from 'fs/promises';
import {basename} from 'path';
import File from './file.js';
import Blob from './index.js';
import {MessageChannel} from 'worker_threads';
import {
realpathSync,
statSync,
rmdirSync,
createReadStream,
promises as fs
} from 'node:fs'
import { basename, sep, join } from 'node:path'
import { tmpdir } from 'node:os'
import process from 'node:process'
import DOMException from 'node-domexception'
const DOMException = globalThis.DOMException || (() => {
const port = new MessageChannel().port1
const ab = new ArrayBuffer(0)
try { port.postMessage(ab, [ab, ab]) }
catch (err) { return err.constructor }
})()
import File from './file.js'
import Blob from './index.js'
const { stat, mkdtemp } = fs
let i = 0, tempDir, registry
/**
* @param {string} path filepath on the disk
* @param {string} [type] mimetype to use
*/
const blobFromSync = (path, type) => fromBlob(statSync(path), path, type);
const blobFromSync = (path, type) => fromBlob(statSync(path), path, type)
/**
* @param {string} path filepath on the disk
* @param {string} [type] mimetype to use
* @returns {Promise<Blob>}
*/
const blobFrom = (path, type) => stat(path).then(stat => fromBlob(stat, path, type))
/**
* @param {string} path filepath on the disk
* @param {string} [type] mimetype to use
* @returns {Promise<File>}
*/
const fileFrom = (path, type) => stat(path).then(stat => fromFile(stat, path, type))
/**
* @param {string} path filepath on the disk
* @param {string} [type] mimetype to use
*/
const blobFrom = (path, type) => stat(path).then(stat => fromBlob(stat, path, type));
/**
* @param {string} path filepath on the disk
* @param {string} [type] mimetype to use
*/
const fileFrom = (path, type) => stat(path).then(stat => fromFile(stat, path, type));
/**
* @param {string} path filepath on the disk
* @param {string} [type] mimetype to use
*/
const fileFromSync = (path, type) => fromFile(statSync(path), path, type);
const fileFromSync = (path, type) => fromFile(statSync(path), path, type)
// @ts-ignore
const fromBlob = (stat, path, type = '') => new Blob([new BlobDataItem({
path,
size: stat.size,
lastModified: stat.mtimeMs,
start: 0
})], {type});
path,
size: stat.size,
lastModified: stat.mtimeMs,
start: 0
})], { type })
// @ts-ignore
const fromFile = (stat, path, type = '') => new File([new BlobDataItem({
path,
size: stat.size,
lastModified: stat.mtimeMs,
start: 0
})], basename(path), { type, lastModified: stat.mtimeMs });
path,
size: stat.size,
lastModified: stat.mtimeMs,
start: 0
})], basename(path), { type, lastModified: stat.mtimeMs })
/**
* Creates a temporary blob backed by the filesystem.
* NOTE: requires node.js v14 or higher to use FinalizationRegistry
*
* @param {*} data Same as fs.writeFile data
* @param {BlobPropertyBag & {signal?: AbortSignal}} options
* @param {AbortSignal} [signal] in case you wish to cancel the write operation
* @returns {Promise<Blob>}
*/
const createTemporaryBlob = async (data, {signal, type} = {}) => {
registry = registry || new FinalizationRegistry(fs.unlink)
tempDir = tempDir || await mkdtemp(realpathSync(tmpdir()) + sep)
const id = `${i++}`
const destination = join(tempDir, id)
if (data instanceof ArrayBuffer) data = new Uint8Array(data)
await fs.writeFile(destination, data, { signal })
const blob = await blobFrom(destination, type)
registry.register(blob, destination)
return blob
}
/**
* Creates a temporary File backed by the filesystem.
* Pretty much the same as constructing a new File(data, name, options)
*
* NOTE: requires node.js v14 or higher to use FinalizationRegistry
* @param {*} data
* @param {string} name
* @param {FilePropertyBag & {signal?: AbortSignal}} opts
* @returns {Promise<File>}
*/
const createTemporaryFile = async (data, name, opts) => {
const blob = await createTemporaryBlob(data)
return new File([blob], name, opts)
}
/**
* This is a blob backed up by a file on the disk
@@ -58,46 +102,63 @@ const fromFile = (stat, path, type = '') => new File([new BlobDataItem({
* @private
*/
class BlobDataItem {
#path;
#start;
#path
#start
constructor(options) {
this.#path = options.path;
this.#start = options.start;
this.size = options.size;
this.lastModified = options.lastModified
}
constructor (options) {
this.#path = options.path
this.#start = options.start
this.size = options.size
this.lastModified = options.lastModified
this.originalSize = options.originalSize === undefined
? options.size
: options.originalSize
}
/**
* Slicing arguments is first validated and formatted
* to not be out of range by Blob.prototype.slice
*/
slice(start, end) {
return new BlobDataItem({
path: this.#path,
lastModified: this.lastModified,
size: end - start,
start
});
}
/**
* Slicing arguments is first validated and formatted
* to not be out of range by Blob.prototype.slice
*/
slice (start, end) {
return new BlobDataItem({
path: this.#path,
lastModified: this.lastModified,
originalSize: this.originalSize,
size: end - start,
start: this.#start + start
})
}
async * stream() {
const {mtimeMs} = await stat(this.#path)
if (mtimeMs > this.lastModified) {
throw new DOMException('The requested file could not be read, typically due to permission problems that have occurred after a reference to a file was acquired.', 'NotReadableError');
}
if (this.size) {
yield * createReadStream(this.#path, {
start: this.#start,
end: this.#start + this.size - 1
});
}
}
async * stream () {
const { mtimeMs, size } = await stat(this.#path)
get [Symbol.toStringTag]() {
return 'Blob';
}
if (mtimeMs > this.lastModified || this.originalSize !== size) {
throw new DOMException('The requested file could not be read, typically due to permission problems that have occurred after a reference to a file was acquired.', 'NotReadableError')
}
yield * createReadStream(this.#path, {
start: this.#start,
end: this.#start + this.size - 1
})
}
get [Symbol.toStringTag] () {
return 'Blob'
}
}
export default blobFromSync;
export {File, Blob, blobFrom, blobFromSync, fileFrom, fileFromSync};
process.once('exit', () => {
tempDir && rmdirSync(tempDir, { recursive: true })
})
export default blobFromSync
export {
Blob,
blobFrom,
blobFromSync,
createTemporaryBlob,
File,
fileFrom,
fileFromSync,
createTemporaryFile
}
+224 -184
View File
@@ -1,214 +1,254 @@
/*! fetch-blob. MIT License. Jimmy Wärting <https://jimmy.warting.se/opensource> */
// TODO (jimmywarting): in the feature use conditional loading with top level await (requires 14.x)
// Node has recently added whatwg stream into core
import './streams.cjs'
// 64 KiB (same size chrome slice theirs blob into Uint8array's)
const POOL_SIZE = 65536;
const POOL_SIZE = 65536
/** @param {(Blob | Uint8Array)[]} parts */
async function * toIterator (parts, clone = true) {
for (let part of parts) {
if ('stream' in part) {
yield * part.stream();
} else if (ArrayBuffer.isView(part)) {
if (clone) {
let position = part.byteOffset;
let end = part.byteOffset + part.byteLength;
while (position !== end) {
const size = Math.min(end - position, POOL_SIZE);
const chunk = part.buffer.slice(position, position + size);
yield new Uint8Array(chunk);
position += chunk.byteLength;
}
} else {
yield part;
}
} else {
// For blobs that have arrayBuffer but no stream method (nodes buffer.Blob)
let position = 0;
while (position !== part.size) {
const chunk = part.slice(position, Math.min(part.size, position + POOL_SIZE));
const buffer = await chunk.arrayBuffer();
position += buffer.byteLength;
yield new Uint8Array(buffer);
}
}
}
for (const part of parts) {
if ('stream' in part) {
yield * (/** @type {AsyncIterableIterator<Uint8Array>} */ (part.stream()))
} else if (ArrayBuffer.isView(part)) {
if (clone) {
let position = part.byteOffset
const end = part.byteOffset + part.byteLength
while (position !== end) {
const size = Math.min(end - position, POOL_SIZE)
const chunk = part.buffer.slice(position, position + size)
position += chunk.byteLength
yield new Uint8Array(chunk)
}
} else {
yield part
}
/* c8 ignore next 10 */
} else {
// For blobs that have arrayBuffer but no stream method (nodes buffer.Blob)
let position = 0, b = (/** @type {Blob} */ (part))
while (position !== b.size) {
const chunk = b.slice(position, Math.min(b.size, position + POOL_SIZE))
const buffer = await chunk.arrayBuffer()
position += buffer.byteLength
yield new Uint8Array(buffer)
}
}
}
}
export default class Blob {
const _Blob = class Blob {
/** @type {Array.<(Blob|Uint8Array)>} */
#parts = []
#type = ''
#size = 0
#endings = 'transparent'
/** @type {Array.<(Blob|Uint8Array)>} */
#parts = [];
#type = '';
#size = 0;
/**
* The Blob() constructor returns a new Blob object. The content
* of the blob consists of the concatenation of the values given
* in the parameter array.
*
* @param {*} blobParts
* @param {{ type?: string, endings?: string }} [options]
*/
constructor (blobParts = [], options = {}) {
if (typeof blobParts !== 'object' || blobParts === null) {
throw new TypeError('Failed to construct \'Blob\': The provided value cannot be converted to a sequence.')
}
/**
* The Blob() constructor returns a new Blob object. The content
* of the blob consists of the concatenation of the values given
* in the parameter array.
*
* @param {*} blobParts
* @param {{ type?: string }} [options]
*/
constructor(blobParts = [], options = {}) {
let size = 0;
if (typeof blobParts[Symbol.iterator] !== 'function') {
throw new TypeError('Failed to construct \'Blob\': The object must have a callable @@iterator property.')
}
const parts = blobParts.map(element => {
let part;
if (ArrayBuffer.isView(element)) {
part = new Uint8Array(element.buffer.slice(element.byteOffset, element.byteOffset + element.byteLength));
} else if (element instanceof ArrayBuffer) {
part = new Uint8Array(element.slice(0));
} else if (element instanceof Blob) {
part = element;
} else {
part = new TextEncoder().encode(String(element));
}
if (typeof options !== 'object' && typeof options !== 'function') {
throw new TypeError('Failed to construct \'Blob\': parameter 2 cannot convert to dictionary.')
}
size += ArrayBuffer.isView(part) ? part.byteLength : part.size;
return part;
});
if (options === null) options = {}
const type = options.type === undefined ? '' : String(options.type);
const encoder = new TextEncoder()
for (const element of blobParts) {
let part
if (ArrayBuffer.isView(element)) {
part = new Uint8Array(element.buffer.slice(element.byteOffset, element.byteOffset + element.byteLength))
} else if (element instanceof ArrayBuffer) {
part = new Uint8Array(element.slice(0))
} else if (element instanceof Blob) {
part = element
} else {
part = encoder.encode(`${element}`)
}
this.#type = /[^\u0020-\u007E]/.test(type) ? '' : type;
this.#size = size;
this.#parts = parts;
}
const size = ArrayBuffer.isView(part) ? part.byteLength : part.size
// Avoid pushing empty parts into the array to better GC them
if (size) {
this.#size += size
this.#parts.push(part)
}
}
/**
* The Blob interface's size property returns the
* size of the Blob in bytes.
*/
get size() {
return this.#size;
}
this.#endings = `${options.endings === undefined ? 'transparent' : options.endings}`
const type = options.type === undefined ? '' : String(options.type)
this.#type = /^[\x20-\x7E]*$/.test(type) ? type : ''
}
/**
* The type property of a Blob object returns the MIME type of the file.
*/
get type() {
return this.#type;
}
/**
* The Blob interface's size property returns the
* size of the Blob in bytes.
*/
get size () {
return this.#size
}
/**
* The text() method in the Blob interface returns a Promise
* that resolves with a string containing the contents of
* the blob, interpreted as UTF-8.
*
* @return {Promise<string>}
*/
async text() {
// More optimized than using this.arrayBuffer()
// that requires twice as much ram
const decoder = new TextDecoder();
let str = '';
for await (let part of toIterator(this.#parts, false)) {
str += decoder.decode(part, { stream: true });
}
// Remaining
str += decoder.decode();
return str;
}
/**
* The type property of a Blob object returns the MIME type of the file.
*/
get type () {
return this.#type
}
/**
* The arrayBuffer() method in the Blob interface returns a
* Promise that resolves with the contents of the blob as
* binary data contained in an ArrayBuffer.
*
* @return {Promise<ArrayBuffer>}
*/
async arrayBuffer() {
const data = new Uint8Array(this.size);
let offset = 0;
for await (const chunk of toIterator(this.#parts, false)) {
data.set(chunk, offset);
offset += chunk.length;
}
/**
* The text() method in the Blob interface returns a Promise
* that resolves with a string containing the contents of
* the blob, interpreted as UTF-8.
*
* @return {Promise<string>}
*/
async text () {
// More optimized than using this.arrayBuffer()
// that requires twice as much ram
const decoder = new TextDecoder()
let str = ''
for await (const part of toIterator(this.#parts, false)) {
str += decoder.decode(part, { stream: true })
}
// Remaining
str += decoder.decode()
return str
}
return data.buffer;
}
/**
* The arrayBuffer() method in the Blob interface returns a
* Promise that resolves with the contents of the blob as
* binary data contained in an ArrayBuffer.
*
* @return {Promise<ArrayBuffer>}
*/
async arrayBuffer () {
// Easier way... Just a unnecessary overhead
// const view = new Uint8Array(this.size);
// await this.stream().getReader({mode: 'byob'}).read(view);
// return view.buffer;
/**
* The Blob stream() implements partial support of the whatwg stream
* by only being async iterable.
*
* @returns {AsyncGenerator<Uint8Array>}
*/
async * stream() {
yield * toIterator(this.#parts, true);
}
const data = new Uint8Array(this.size)
let offset = 0
for await (const chunk of toIterator(this.#parts, false)) {
data.set(chunk, offset)
offset += chunk.length
}
/**
* The Blob interface's slice() method creates and returns a
* new Blob object which contains data from a subset of the
* blob on which it's called.
*
* @param {number} [start]
* @param {number} [end]
* @param {string} [type]
*/
slice(start = 0, end = this.size, type = '') {
const {size} = this;
return data.buffer
}
let relativeStart = start < 0 ? Math.max(size + start, 0) : Math.min(start, size);
let relativeEnd = end < 0 ? Math.max(size + end, 0) : Math.min(end, size);
stream () {
const it = toIterator(this.#parts, true)
const span = Math.max(relativeEnd - relativeStart, 0);
const parts = this.#parts;
const blobParts = [];
let added = 0;
return new globalThis.ReadableStream({
// @ts-ignore
type: 'bytes',
async pull (ctrl) {
const chunk = await it.next()
chunk.done ? ctrl.close() : ctrl.enqueue(chunk.value)
},
for (const part of parts) {
const size = ArrayBuffer.isView(part) ? part.byteLength : part.size;
if (relativeStart && size <= relativeStart) {
// Skip the beginning and change the relative
// start & end position as we skip the unwanted parts
relativeStart -= size;
relativeEnd -= size;
} else {
let chunk
if (ArrayBuffer.isView(part)) {
chunk = part.subarray(relativeStart, Math.min(size, relativeEnd));
added += chunk.byteLength
} else {
chunk = part.slice(relativeStart, Math.min(size, relativeEnd));
added += chunk.size
}
blobParts.push(chunk);
relativeStart = 0; // All next sequential parts should start at 0
async cancel () {
await it.return()
}
})
}
// don't add the overflow to new blobParts
if (added >= span) {
break;
}
}
}
/**
* The Blob interface's slice() method creates and returns a
* new Blob object which contains data from a subset of the
* blob on which it's called.
*
* @param {number} [start]
* @param {number} [end]
* @param {string} [type]
*/
slice (start = 0, end = this.size, type = '') {
const { size } = this
const blob = new Blob([], {type: String(type).toLowerCase()});
blob.#size = span;
blob.#parts = blobParts;
let relativeStart = start < 0 ? Math.max(size + start, 0) : Math.min(start, size)
let relativeEnd = end < 0 ? Math.max(size + end, 0) : Math.min(end, size)
return blob;
}
const span = Math.max(relativeEnd - relativeStart, 0)
const parts = this.#parts
const blobParts = []
let added = 0
get [Symbol.toStringTag]() {
return 'Blob';
}
for (const part of parts) {
// don't add the overflow to new blobParts
if (added >= span) {
break
}
static [Symbol.hasInstance](object) {
return (
typeof object?.constructor === 'function' &&
(
typeof object.stream === 'function' ||
typeof object.arrayBuffer === 'function'
) &&
/^(Blob|File)$/.test(object[Symbol.toStringTag])
);
}
const size = ArrayBuffer.isView(part) ? part.byteLength : part.size
if (relativeStart && size <= relativeStart) {
// Skip the beginning and change the relative
// start & end position as we skip the unwanted parts
relativeStart -= size
relativeEnd -= size
} else {
let chunk
if (ArrayBuffer.isView(part)) {
chunk = part.subarray(relativeStart, Math.min(size, relativeEnd))
added += chunk.byteLength
} else {
chunk = part.slice(relativeStart, Math.min(size, relativeEnd))
added += chunk.size
}
relativeEnd -= size
blobParts.push(chunk)
relativeStart = 0 // All next sequential parts should start at 0
}
}
const blob = new Blob([], { type: String(type).toLowerCase() })
blob.#size = span
blob.#parts = blobParts
return blob
}
get [Symbol.toStringTag] () {
return 'Blob'
}
static [Symbol.hasInstance] (object) {
return (
object &&
typeof object === 'object' &&
typeof object.constructor === 'function' &&
(
typeof object.stream === 'function' ||
typeof object.arrayBuffer === 'function'
) &&
/^(Blob|File)$/.test(object[Symbol.toStringTag])
)
}
}
Object.defineProperties(Blob.prototype, {
size: {enumerable: true},
type: {enumerable: true},
slice: {enumerable: true}
});
Object.defineProperties(_Blob.prototype, {
size: { enumerable: true },
type: { enumerable: true },
slice: { enumerable: true }
})
export { Blob };
/** @type {typeof globalThis.Blob} */
export const Blob = _Blob
export default Blob
+15 -37
View File
@@ -1,6 +1,6 @@
{
"name": "fetch-blob",
"version": "3.0.0",
"version": "3.1.5",
"description": "Blob & File implementation in Node.js, originally from node-fetch.",
"main": "index.js",
"type": "module",
@@ -10,13 +10,13 @@
"file.d.ts",
"index.js",
"index.d.ts",
"from.d.ts"
"from.d.ts",
"streams.cjs"
],
"scripts": {
"lint": "xo test.js",
"test": "npm run lint && ava",
"report": "c8 --reporter json --reporter text ava",
"coverage": "c8 --reporter json --reporter text ava && codecov -f coverage/coverage-final.json",
"test": "node --experimental-loader ./test/http-loader.js ./test/test-wpt-in-node.js",
"report": "c8 --reporter json --reporter text npm run test",
"coverage": "npm run report && codecov -f coverage/coverage-final.json",
"prepublishOnly": "tsc --declaration --emitDeclarationOnly --allowJs index.js from.js"
},
"repository": "https://github.com/node-fetch/fetch-blob.git",
@@ -26,7 +26,7 @@
"node-fetch"
],
"engines": {
"node": ">=14.0.0"
"node": "^12.20 || >= 14.13"
},
"author": "Jimmy Wärting <jimmy@warting.se> (https://jimmy.warting.se)",
"license": "MIT",
@@ -34,36 +34,10 @@
"url": "https://github.com/node-fetch/fetch-blob/issues"
},
"homepage": "https://github.com/node-fetch/fetch-blob#readme",
"xo": {
"rules": {
"unicorn/prefer-node-protocol": "off",
"unicorn/numeric-separators-style": "off",
"unicorn/prefer-spread": "off",
"import/extensions": [
"error",
"always",
{
"ignorePackages": true
}
]
},
"overrides": [
{
"files": "test.js",
"rules": {
"node/no-unsupported-features/es-syntax": 0,
"node/no-unsupported-features/node-builtins": 0
}
}
]
},
"devDependencies": {
"ava": "^3.15.0",
"c8": "^7.7.2",
"codecov": "^3.8.2",
"node-fetch": "^3.0.0-beta.9",
"typescript": "^4.3.2",
"xo": "^0.40.1"
"@types/node": "^18.0.2",
"c8": "^7.11.0",
"typescript": "^4.5.4"
},
"funding": [
{
@@ -74,5 +48,9 @@
"type": "paypal",
"url": "https://paypal.me/jimmywarting"
}
]
],
"dependencies": {
"node-domexception": "^1.0.0",
"web-streams-polyfill": "^3.0.3"
}
}
+51
View File
@@ -0,0 +1,51 @@
/* c8 ignore start */
// 64 KiB (same size chrome slice theirs blob into Uint8array's)
const POOL_SIZE = 65536
if (!globalThis.ReadableStream) {
// `node:stream/web` got introduced in v16.5.0 as experimental
// and it's preferred over the polyfilled version. So we also
// suppress the warning that gets emitted by NodeJS for using it.
try {
const process = require('node:process')
const { emitWarning } = process
try {
process.emitWarning = () => {}
Object.assign(globalThis, require('node:stream/web'))
process.emitWarning = emitWarning
} catch (error) {
process.emitWarning = emitWarning
throw error
}
} catch (error) {
// fallback to polyfill implementation
Object.assign(globalThis, require('web-streams-polyfill/dist/ponyfill.es2018.js'))
}
}
try {
// Don't use node: prefix for this, require+node: is not supported until node v14.14
// Only `import()` can use prefix in 12.20 and later
const { Blob } = require('buffer')
if (Blob && !Blob.prototype.stream) {
Blob.prototype.stream = function name (params) {
let position = 0
const blob = this
return new ReadableStream({
type: 'bytes',
async pull (ctrl) {
const chunk = blob.slice(position, Math.min(blob.size, position + POOL_SIZE))
const buffer = await chunk.arrayBuffer()
position += buffer.byteLength
ctrl.enqueue(new Uint8Array(buffer))
if (position === blob.size) {
ctrl.close()
}
}
})
}
}
} catch (error) {}
/* c8 ignore end */
-324
View File
@@ -1,324 +0,0 @@
import fs from 'fs';
import {Readable} from 'stream';
import buffer from 'buffer';
import test from 'ava';
import {Response} from 'node-fetch';
import syncBlob, {blobFromSync, blobFrom, fileFromSync, fileFrom} from './from.js';
import File from './file.js';
import Blob from './index.js';
const license = fs.readFileSync('./LICENSE', 'utf-8');
test('new Blob()', t => {
const blob = new Blob(); // eslint-disable-line no-unused-vars
t.pass();
});
test('new Blob(parts)', t => {
const data = 'a=1';
const blob = new Blob([data]); // eslint-disable-line no-unused-vars
t.pass();
});
test('Blob ctor parts', async t => {
const parts = [
'a',
new Uint8Array([98]),
new Uint16Array([25699]),
new Uint8Array([101]).buffer,
Buffer.from('f'),
new Blob(['g']),
{},
new URLSearchParams('foo')
];
const blob = new Blob(parts);
t.is(await blob.text(), 'abcdefg[object Object]foo=');
});
test('Blob size', t => {
const data = 'a=1';
const blob = new Blob([data]);
t.is(blob.size, data.length);
});
test('Blob type', t => {
const type = 'text/plain';
const blob = new Blob([], {type});
t.is(blob.type, type);
});
test('Blob slice type', t => {
const type = 'text/plain';
const blob = new Blob().slice(0, 0, type);
t.is(blob.type, type);
});
test('invalid Blob type', t => {
const blob = new Blob([], {type: '\u001Ftext/plain'});
t.is(blob.type, '');
});
test('invalid Blob slice type', t => {
const blob = new Blob().slice(0, 0, '\u001Ftext/plain');
t.is(blob.type, '');
});
test('Blob text()', async t => {
const data = 'a=1';
const type = 'text/plain';
const blob = new Blob([data], {type});
t.is(await blob.text(), data);
});
test('Blob arrayBuffer()', async t => {
const data = 'a=1';
const type = 'text/plain';
const blob = new Blob([data], {type});
const decoder = new TextDecoder('utf-8');
const buffer = await blob.arrayBuffer();
t.is(decoder.decode(buffer), data);
});
test('Blob stream()', async t => {
const data = 'a=1';
const type = 'text/plain';
const blob = new Blob([data], {type});
for await (const chunk of blob.stream()) {
t.is(chunk.join(), [97, 61, 49].join());
}
});
test('Blob toString()', t => {
const data = 'a=1';
const type = 'text/plain';
const blob = new Blob([data], {type});
t.is(blob.toString(), '[object Blob]');
});
test('Blob slice()', async t => {
const data = 'abcdefgh';
const blob = new Blob([data]).slice();
t.is(await blob.text(), data);
});
test('Blob slice(0, 1)', async t => {
const data = 'abcdefgh';
const blob = new Blob([data]).slice(0, 1);
t.is(await blob.text(), 'a');
});
test('Blob slice(-1)', async t => {
const data = 'abcdefgh';
const blob = new Blob([data]).slice(-1);
t.is(await blob.text(), 'h');
});
test('Blob slice(0, -1)', async t => {
const data = 'abcdefgh';
const blob = new Blob([data]).slice(0, -1);
t.is(await blob.text(), 'abcdefg');
});
test('Blob(["hello ", "world"]).slice(5)', async t => {
const parts = ['hello ', 'world'];
const blob = new Blob(parts);
t.is(await blob.slice(5).text(), ' world');
});
test('throw away unwanted parts', async t => {
const blob = new Blob(['a', 'b', 'c']).slice(1, 2);
t.is(await blob.text(), 'b');
});
test('Blob works with node-fetch Response.blob()', async t => {
const data = 'a=1';
const type = 'text/plain';
const blob = new Blob([data], {type});
const response = new Response(Readable.from(blob.stream()));
const blob2 = await response.blob();
t.is(await blob2.text(), data);
});
test('Blob works with node-fetch Response.text()', async t => {
const data = 'a=1';
const type = 'text/plain';
const blob = new Blob([data], {type});
const response = new Response(Readable.from(blob.stream()));
const text = await response.text();
t.is(text, data);
});
test('blob part backed up by filesystem', async t => {
const blob = blobFromSync('./LICENSE');
t.is(await blob.slice(0, 3).text(), license.slice(0, 3));
t.is(await blob.slice(4, 11).text(), license.slice(4, 11));
});
test('Reading after modified should fail', async t => {
const blob = blobFromSync('./LICENSE');
await new Promise(resolve => {
setTimeout(resolve, 100);
});
const now = new Date();
// Change modified time
fs.utimesSync('./LICENSE', now, now);
const error = await t.throwsAsync(blob.text());
t.is(error.constructor.name, 'DOMException');
t.is(error instanceof Error, true);
t.is(error.name, 'NotReadableError');
});
test('Reading file after modified should fail', async t => {
const file = fileFromSync('./LICENSE');
await new Promise(resolve => {
setTimeout(resolve, 100);
});
const now = new Date();
// Change modified time
fs.utimesSync('./LICENSE', now, now);
const error = await t.throwsAsync(file.text());
t.is(error.constructor.name, 'DOMException');
t.is(error instanceof Error, true);
t.is(error.name, 'NotReadableError');
});
test('Reading from the stream created by blobFrom', async t => {
const blob = blobFromSync('./LICENSE');
const actual = await blob.text();
t.is(actual, license);
});
test('create a blob from path asynchronous', async t => {
const blob = await blobFrom('./LICENSE');
const actual = await blob.text();
t.is(actual, license);
});
test('Reading empty blobs', async t => {
const blob = blobFromSync('./LICENSE').slice(0, 0);
const actual = await blob.text();
t.is(actual, '');
});
test('Blob-ish class is an instance of Blob', t => {
class File {
stream() {}
get [Symbol.toStringTag]() {
return 'File';
}
}
t.true(new File() instanceof Blob);
});
test('Instanceof check returns false for nullish values', t => {
t.false(null instanceof Blob);
});
/** @see https://github.com/w3c/FileAPI/issues/43 - important to keep boundary value */
test('Dose not lowercase the blob values', t => {
const type = 'multipart/form-data; boundary=----WebKitFormBoundaryTKqdrVt01qOBltBd';
t.is(new Blob([], {type}).type, type);
});
test('Parts are immutable', async t => {
const buf = new Uint8Array([97]);
const blob = new Blob([buf]);
buf[0] = 98;
t.is(await blob.text(), 'a');
});
test('Blobs are immutable', async t => {
const buf = new Uint8Array([97]);
const blob = new Blob([buf]);
const chunk = await blob.stream().next();
t.is(chunk.value[0], 97);
chunk.value[0] = 98;
t.is(await blob.text(), 'a');
});
// This was necessary to avoid large ArrayBuffer clones (slice)
test('Large chunks are divided into smaller chunks', async t => {
const buf = new Uint8Array(65590);
const blob = new Blob([buf]);
let i = 0;
// eslint-disable-next-line no-unused-vars
for await (const chunk of blob.stream()) {
i++;
}
t.is(i === 2, true);
});
test('Can use named import - as well as default', async t => {
// eslint-disable-next-line node/no-unsupported-features/es-syntax
const {Blob, default: def} = await import('./index.js');
t.is(Blob, def);
});
test('default from.js exports blobFromSync', t => {
t.is(blobFromSync, syncBlob);
});
if (buffer.Blob) {
test('Can wrap buffer.Blob to a fetch-blob', async t => {
const blob1 = new buffer.Blob(['blob part']);
const blob2 = new Blob([blob1]);
t.is(await blob2.text(), 'blob part');
});
}
test('File is a instance of blob', t => {
t.true(new File([], '') instanceof Blob);
});
test('fileFrom returns the name', async t => {
t.is((await fileFrom('./LICENSE')).name, 'LICENSE');
});
test('fileFromSync returns the name', t => {
t.is(fileFromSync('./LICENSE').name, 'LICENSE');
});
test('fileFromSync(path, type) sets the type', t => {
t.is(fileFromSync('./LICENSE', 'text/plain').type, 'text/plain');
});
test('blobFromSync(path, type) sets the type', t => {
t.is(blobFromSync('./LICENSE', 'text/plain').type, 'text/plain');
});
test('fileFrom(path, type) sets the type', async t => {
const file = await fileFrom('./LICENSE', 'text/plain');
t.is(file.type, 'text/plain');
});
test('fileFrom(path, type) read/sets the lastModified ', async t => {
const file = await fileFrom('./LICENSE', 'text/plain');
// Earlier test updates the last modified date to now
t.is(typeof file.lastModified, 'number');
// The lastModifiedDate is deprecated and removed from spec
t.false('lastModifiedDate' in file);
t.is(file.lastModified > Date.now() - 60000, true);
});
test('blobFrom(path, type) sets the type', async t => {
const blob = await blobFrom('./LICENSE', 'text/plain');
t.is(blob.type, 'text/plain');
});
test('blobFrom(path) sets empty type', async t => {
const blob = await blobFrom('./LICENSE');
t.is(blob.type, '');
});
test('new File() throws with too few args', t => {
t.throws(() => new File(), {
instanceOf: TypeError,
message: 'Failed to construct \'File\': 2 arguments required, but only 0 present.'
});
});
+65
View File
@@ -0,0 +1,65 @@
import { Buffer } from 'node:buffer'
import fs from 'node:fs'
import { get } from 'node:https'
const fetch = url => new Promise(rs => get(url, rs))
const cache = new URL('./.cache/', import.meta.url)
/**
* @param {string} specifier
* @param {{
* conditions: !Array<string>,
* parentURL: !(string | undefined),
* }} context
* @param {Function} defaultResolve
* @returns {Promise<{ url: string }>}
*/
export async function resolve (specifier, context, defaultResolve) {
const { parentURL = null } = context
// Normally Node.js would error on specifiers starting with 'https://', so
// this hook intercepts them and converts them into absolute URLs to be
// passed along to the later hooks below.
if (specifier.startsWith('https://')) {
return {
url: specifier
}
} else if (parentURL && parentURL.startsWith('https://')) {
return {
url: new URL(specifier, parentURL).href
}
}
// Let Node.js handle all other specifiers.
return defaultResolve(specifier, context, defaultResolve)
}
export async function load (url, context, defaultLoad) {
// For JavaScript to be loaded over the network, we need to fetch and
// return it.
if (url.startsWith('https://')) {
const uuid = Buffer.from(url).toString('hex')
const cachedFile = new URL(uuid, cache)
let data = ''
// cache remote files for 1h
if (fs.existsSync(cachedFile) && fs.statSync(cachedFile).mtimeMs > Date.now() - 1000 * 60 * 60) {
data = fs.readFileSync(cachedFile, 'utf8')
} else {
const res = await fetch(url).catch(err => err)
for await (const chunk of res) data += chunk
fs.mkdirSync(cache, { recursive: true })
fs.writeFileSync(cachedFile, data)
}
return {
// This example assumes all network-provided JavaScript is ES module
// code.
format: 'module',
source: data
}
}
// Let Node.js handle all other URLs.
return defaultLoad(url, context, defaultLoad)
}
+262
View File
@@ -0,0 +1,262 @@
// @ts-nocheck
// @ts-ignore
import fs from 'node:fs'
import buffer from 'node:buffer'
import syncBlob, {
blobFromSync,
blobFrom,
fileFromSync,
fileFrom,
createTemporaryBlob,
createTemporaryFile
} from '../from.js'
const license = fs.readFileSync('./LICENSE')
test_blob(() => new Blob([
'a',
new Uint8Array([98]),
new Uint16Array([25699]),
new Uint8Array([101]).buffer,
Buffer.from('f'),
new Blob(['g']),
{},
new URLSearchParams('foo')
]), {
desc: 'Blob ctor parts',
expected: 'abcdefg[object Object]foo=',
type: '',
length: 26
})
promise_test(async () => {
assert_equals(fileFromSync('./LICENSE').name, 'LICENSE')
assert_equals((await fileFrom('./LICENSE')).name, 'LICENSE')
}, 'file from returns the same name')
// Could not find similar test on WPT
test(() => {
const now = new Date()
assert_equals(new File([], '', { lastModified: now }).lastModified, +now)
assert_equals(new File([], '', { lastModified: +now }).lastModified, +now)
assert_equals(new File([], '', { lastModified: 100 }).lastModified, 100)
assert_equals(new File([], '', { lastModified: '200' }).lastModified, 200)
assert_equals(new File([], '', { lastModified: true }).lastModified, 1)
assert_equals(new File([], '', { lastModified: false }).lastModified, 0)
assert_approx_equals(new File([], '').lastModified, Date.now(), 0.020)
assert_approx_equals(new File([], '', { lastModified: undefined }).lastModified, Date.now(), 0.020)
}, 'File sets current time')
// Could not find similar test on WPT
test(() => {
const values = ['Not a Number', [], {}, null]
// I can't really see anything about this in the spec,
// but this is how browsers handle type casting for this option...
for (const lastModified of values) {
const file = new File([], '', { lastModified })
assert_equals(file.lastModified, 0)
}
}, 'Interpretes NaN value in lastModified option as 0')
test(() => {
assert_equals(blobFromSync, syncBlob)
}, 'default export is named exported blobFromSync')
promise_test(async () => {
const { Blob, default: def } = await import('../index.js')
assert_equals(Blob, def)
}, 'Can use named import - as well as default')
// This was necessary to avoid large ArrayBuffer clones (slice)
promise_test(async t => {
const buf = new Uint8Array(65590)
const blob = new Blob([buf])
let i = 0
// eslint-disable-next-line no-unused-vars
for await (const chunk of blob.stream()) {
i++
}
assert_equals(i, 2)
}, 'Large chunks are divided into smaller chunks')
// Couldn't find a test for this on WPT
promise_test(async () => {
const buf = new Uint8Array([97])
const blob = new Blob([buf])
const chunk = await blob.stream().getReader().read()
assert_equals(chunk.value[0], 97)
chunk.value[0] = 98
assert_equals(await blob.text(), 'a')
}, 'Blobs are immutable')
/**
* Deviation from WPT: it's important to keep boundary value
* so we don't lowercase the type
* @see https://github.com/w3c/FileAPI/issues/43
*/
test(() => {
const type = 'multipart/form-data; boundary=----WebKitFormBoundaryTKqdrVt01qOBltBd'
assert_equals(new Blob([], { type }).type, type)
assert_equals(new File([], '', { type }).type, type)
}, 'Dose not lowercase the type')
test( // Because we have symbol.hasInstance it's best to test it...
() => (assert_false(null instanceof Blob), assert_false(null instanceof File)),
'Instanceof check returns false for nullish values'
)
test( // Because browser normally never tries things taken for granted
() => assert_equals(new Blob().toString(), '[object Blob]'),
'blob.toString() returns [object Blob]'
)
test( // Because browser normally never tries things taken for granted
() => assert_equals(new File([], '').toString(), '[object File]'),
'file.toString() returns [object File]'
)
// fetch-blob uniques is that it supports arbitrary blobs too
test(() => {
class File {
stream () {}
get [Symbol.toStringTag] () { return 'File' }
}
assert_true(new File() instanceof Blob)
}, 'Blob-ish class is an instance of Blob')
// fetch-blob uniques is that it supports arbitrary blobs too
if (buffer.Blob) {
test_blob(() => new Blob([new buffer.Blob(['blob part'])]), {
desc: 'Can wrap buffer.Blob to a fetch-blob',
expected: 'blob part',
type: '',
})
}
/**
* Test if Blob can be constructed with BOM and keep it when casted to string
* Test if blob.text() can correctly remove BOM - `buffer.toString()` is bad
*/
promise_test(async () => {
const text = '{"foo": "bar"}'
const blob = new Blob([`\uFEFF${text}`])
assert_equals(blob.size, 17)
assert_equals(await blob.text(), text)
const ab = await blob.slice(0, 3).arrayBuffer()
assert_equals_typed_array(new Uint8Array(ab), new Uint8Array([0xEF, 0xBB, 0xBF]))
}, 'Can wrap buffer.Blob to a fetch-blob')
// Here to make sure our `toIterator` is working as intended
promise_test(async () => {
const stream = new Blob(['Some content']).stream()
// Cancel the stream before start reading, or this will throw an error
await stream.cancel()
const reader = stream.getReader()
const { done, value: chunk } = await reader.read()
assert_true(done)
assert_equals(chunk, undefined)
}, 'Blob stream() can be cancelled')
/******************************************************************************/
/* */
/* Test Blobs backed up by the filesystem */
/* */
/******************************************************************************/
promise_test(async () => {
assert_equals(fileFromSync('./LICENSE', 'text/plain').type, 'text/plain')
assert_equals(fileFromSync('./LICENSE').type, '')
assert_equals(blobFromSync('./LICENSE', 'text/plain').type, 'text/plain')
assert_equals(blobFromSync('./LICENSE').type, '')
assert_equals((await fileFrom('./LICENSE', 'text/plain')).type, 'text/plain')
assert_equals((await fileFrom('./LICENSE')).type, '')
assert_equals((await blobFrom('./LICENSE', 'text/plain')).type, 'text/plain')
assert_equals((await blobFrom('./LICENSE')).type, '')
}, 'from utilities sets correct type')
promise_test(async () => {
assert_equals(await blobFromSync('./LICENSE').text(), license.toString())
assert_equals(await fileFromSync('./LICENSE').text(), license.toString())
assert_equals(await (await blobFrom('./LICENSE')).text(), license.toString())
assert_equals(await (await fileFrom('./LICENSE')).text(), license.toString())
}, 'blob part backed up by filesystem can be read')
promise_test(async () => {
assert_equals(await blobFromSync('./LICENSE').text(), license.toString())
assert_equals(await fileFromSync('./LICENSE').text(), license.toString())
assert_equals(await (await blobFrom('./LICENSE')).text(), license.toString())
assert_equals(await (await fileFrom('./LICENSE')).text(), license.toString())
}, 'blob part backed up by filesystem slice correctly')
promise_test(async () => {
let blob
// Can construct a temporary blob from a string
blob = await createTemporaryBlob(license.toString())
assert_equals(await blob.text(), license.toString())
// Can construct a temporary blob from a async iterator
blob = await createTemporaryBlob(blob.stream())
assert_equals(await blob.text(), license.toString())
// Can construct a temporary file from a arrayBuffer
blob = await createTemporaryBlob(await blob.arrayBuffer())
assert_equals(await blob.text(), license.toString())
// Can construct a temporary file from a arrayBufferView
blob = await createTemporaryBlob(await blob.arrayBuffer().then(ab => new Uint8Array(ab)))
assert_equals(await blob.text(), license.toString())
// Can specify a mime type
blob = await createTemporaryBlob('abc', { type: 'text/plain' })
assert_equals(blob.type, 'text/plain')
// Can create files too
let file = await createTemporaryFile('abc', 'abc.txt', {
type: 'text/plain',
lastModified: 123
})
assert_equals(file.name, 'abc.txt')
assert_equals(file.size, 3)
assert_equals(file.lastModified, 123)
}, 'creating temporary blob/file backed up by filesystem')
promise_test(async () => {
fs.writeFileSync('temp', '')
await blobFromSync('./temp').text()
fs.unlinkSync('./temp')
}, 'can read empty files')
test(async () => {
const blob = blobFromSync('./LICENSE')
await new Promise(resolve => setTimeout(resolve, 2000))
const now = new Date()
// Change modified time
fs.utimesSync('./LICENSE', now, now)
const error = await blob.text().then(assert_unreached, e => e)
assert_equals(error.constructor.name, 'DOMException')
assert_equals(error instanceof Error, true)
assert_equals(error.name, 'NotReadableError')
const file = fileFromSync('./LICENSE')
// Above test updates the last modified date to now
assert_equals(typeof file.lastModified, 'number')
// The lastModifiedDate is deprecated and removed from spec
assert_false('lastModifiedDate' in file)
assert_approx_equals(file.lastModified, +now, 1000)
}, 'Reading after modified should fail')
promise_test(async () => {
assert_equals(await blobFromSync('./LICENSE').slice(0, 0).text(), '')
assert_equals(await blobFromSync('./LICENSE').slice(0, 3).text(), license.slice(0, 3).toString())
assert_equals(await blobFromSync('./LICENSE').slice(4, 11).text(), license.slice(4, 11).toString())
assert_equals(await blobFromSync('./LICENSE').slice(-11).text(), license.slice(-11).toString())
assert_equals(await blobFromSync('./LICENSE').slice(4, 11).slice(2, 5).text(), license.slice(4, 11).slice(2, 5).toString())
}, 'slicing blobs backed up by filesystem returns correct string')
+143
View File
@@ -0,0 +1,143 @@
// Don't want to use the FileReader, don't want to lowerCase the type either
// import from 'https://wpt.live/resources/testharnessreport.js'
import { File, Blob } from '../from.js'
let hasFailed
globalThis.self = globalThis
await import('https://wpt.live/resources/testharness.js')
setup({
explicit_timeout: true,
explicit_done: true
})
function test_blob (fn, expectations) {
const expected = expectations.expected
const type = expectations.type
const desc = expectations.desc
const length = expectations.length
const t = async_test(desc)
t.step(async function () {
const blob = fn()
assert_true(blob instanceof Blob)
assert_false(blob instanceof File)
assert_equals(blob.type.toLowerCase(), type)
assert_equals(await blob.text(), expected)
t.done()
})
}
function test_blob_binary (fn, expectations) {
const expected = expectations.expected
const type = expectations.type
const desc = expectations.desc
const t = async_test(desc)
t.step(async function () {
const blob = fn()
assert_true(blob instanceof Blob)
assert_false(blob instanceof File)
assert_equals(blob.type.toLowerCase(), type)
const result = await blob.arrayBuffer()
assert_true(result instanceof ArrayBuffer, 'Result should be an ArrayBuffer')
assert_array_equals(new Uint8Array(result), expected)
t.done()
})
}
// Assert that two TypedArray objects have the same byte values
globalThis.assert_equals_typed_array = (array1, array2) => {
const [view1, view2] = [array1, array2].map((array) => {
assert_true(array.buffer instanceof ArrayBuffer,
'Expect input ArrayBuffers to contain field `buffer`')
return new DataView(array.buffer, array.byteOffset, array.byteLength)
})
assert_equals(view1.byteLength, view2.byteLength,
'Expect both arrays to be of the same byte length')
const byteLength = view1.byteLength
for (let i = 0; i < byteLength; ++i) {
assert_equals(view1.getUint8(i), view2.getUint8(i),
`Expect byte at buffer position ${i} to be equal`)
}
}
globalThis.add_result_callback((test, ...args) => {
if ([
'Blob with type "A"',
'Blob with type "TEXT/HTML"',
'Getters and value conversions should happen in order until an exception is thrown.',
'Using type in File constructor: TEXT/PLAIN',
'Using type in File constructor: text/plain;charset=UTF-8'
].includes(test.name)) return
const INDENT_SIZE = 2
const reporter = {}
reporter.startSuite = name => console.log(`\n ${(name)}\n`)
reporter.pass = message => console.log((indent(('√ ') + message, INDENT_SIZE)))
reporter.fail = message => console.log((indent('\u00D7 ' + message, INDENT_SIZE)))
reporter.reportStack = stack => console.log((indent(stack, INDENT_SIZE * 2)))
function indent (string, times) {
const prefix = ' '.repeat(times)
return string.split('\n').map(l => prefix + l).join('\n')
}
if (test.status === 0) {
reporter.pass(test.name)
} else if (test.status === 1) {
reporter.fail(`${test.name}\n`)
reporter.reportStack(`${test.message}\n${test.stack}`)
hasFailed = true
} else if (test.status === 2) {
reporter.fail(`${test.name} (timeout)\n`)
reporter.reportStack(`${test.message}\n${test.stack}`)
hasFailed = true
} else if (test.status === 3) {
reporter.fail(`${test.name} (incomplete)\n`)
reporter.reportStack(`${test.message}\n${test.stack}`)
hasFailed = true
} else if (test.status === 4) {
reporter.fail(`${test.name} (precondition failed)\n`)
reporter.reportStack(`${test.message}\n${test.stack}`)
hasFailed = true
} else {
reporter.fail(`unknown test status: ${test.status}`)
hasFailed = true
}
})
globalThis.File = File
globalThis.Blob = Blob
globalThis.garbageCollect = () => {}
globalThis.test_blob = test_blob
globalThis.test_blob_binary = test_blob_binary
// Cuz WPT don't clean up after itself
globalThis.MessageChannel = class extends MessageChannel {
constructor () {
super()
setTimeout(() => {
this.port1.close()
this.port2.close()
this.port1.onmessage = this.port2.onmessage = null
}, 100)
}
}
import('https://wpt.live/FileAPI/file/File-constructor.any.js')
import('https://wpt.live/FileAPI/blob/Blob-constructor.any.js')
import('https://wpt.live/FileAPI/blob/Blob-array-buffer.any.js')
import('https://wpt.live/FileAPI/blob/Blob-slice-overflow.any.js')
import('https://wpt.live/FileAPI/blob/Blob-slice.any.js')
import('https://wpt.live/FileAPI/blob/Blob-stream.any.js')
import('https://wpt.live/FileAPI/blob/Blob-text.any.js')
import('./own-misc-test.js')
hasFailed && process.exit(1)