tailwindcss/src/lib/content.js
Robin Malfait 2a403267d8
Oxide (#10252)
* temporarily disable workflows

* add oxide

Our Rust related parts

* use oxide

- Setup the codebase to be able to use the Rust parts based on an
  environment variable: `OXIDE=1`.
- Setup some tests that run both the non-Rust and Rust version in the
  same test.
- Sort the candidates in a consistent way, to guarantee the order for
  now (especially in tests).
- Reflect sorting related changes in tests.
- Ensure tests run in both the Rust and non-Rust version. (Some tests
  are explicitly skipped when using the Rust version since we haven't
  implemented those features yet. These include: custom prefix,
  transformers and extractors).
  - `jest`
  -`OXIDE=1 jest`

* remove into_par_iter where it doesn't make sense

* cargo fmt

* wip

* enable tracing based on `DEBUG` env

* improve CI for the Oxide build

* sort test output

This happened because the sorting happens in this branch, but changes
happened on the `master` branch.

* add failing tests

I noticed that some of the tests were failing, and while looking at
them, it happened because the tests were structured like this:

```html
    <div
      class="
        backdrop-filter
        backdrop-filter-none
        backdrop-blur-lg
        backdrop-brightness-50
        backdrop-contrast-0
        backdrop-grayscale
        backdrop-hue-rotate-90
        backdrop-invert
        backdrop-opacity-75
        backdrop-saturate-150
        backdrop-sepia
      "
    ></div>
```

This means that the class names themselves eventually end up like this: `backdrop-filter-none\n`
-> (Notice the `\n`)

/cc @thecrypticace

* fix range to include `\n`

* Include only unique values for tests

Really, what we care about most is that the list contains every expected candidate. Not necessarily how many times it shows up because while many candidates will show up A LOT in a source text we’ll unique them before passing them back to anything that needs them

* Fix failing tests

* Don’t match empty arbitrary values

* skip tests in oxide mode regarding custom separators in arbitrary variants

* re-enable workflows

* use `@tailwindcss/oxide` dependency

* publish `tailwindcss@oxide`

* drop prepublishOnly

I don't think we actually need this anymore (or even want because this
is trying to do things in CI that we don't want to happen. Aka, build
the Oxide Rust code, it is already a dependency).

* WIP

* Defer to existing CLI for Oxide

* Include new compiled typescript stuff when publishing

* Move TS to ./src/oxide

* Update scripts

* Clean up tests for TS

* copy `cli` to `oxide/cli`

* make CLI files TypeScript files

* drop --postcss flag

* setup lightningcss

* Remove autoprefixer and cssnano from oxide CLI

* cleanup Rust code a little bit

- Drop commented out code
- Drop 500 fixture templates

* sort test output

* re-add `prepublishOnly` script

* bump SWC dependencies in package-lock.json

* pin `@swc` dependencies

* ensure to install and build oxide

* update all GitHub Workflows to reflect Oxide required changes

* sort `content-resolution` integration tests

* add `Release Insiders — Oxide`

* setup turbo repo + remote caching

* use `npx` to invoke `turbo`

* setup unique/proper package names for integration tests

* add missing `isomorphic-fetch` dependency

* setup integration tests to use `turborepo`

* scope tailwind tasks to root workspace

* re-enable `node_modules` cache for integration tests

* re-enable `node_modules` cache for main CI workflow

* split cache for `main` and `oxide` node_modules

* fix indent

* split install dependencies so that they can be cached individually

* improve GitHub actions caching

* use correct path for oxide node_modules (crates/node)

* ensure that `cargo install` always succeeds

cargo install X, on CI will fail if it already exists.

* figure out integration tests with turbo

* tmp: use `npm` instead of `turbo`

* disable `fail-fast`

This will allow us to run integration tests so that it still caches the
succesful ones.

* YAML OH YAML, Y U WHITESPACE SENSITIVE

* copy the oxide-ci workflow to release-oxide

* make `oxide-ci` a normal CI workflow

Without publishing

* try to cache cargo and node_modules for the oxide build

* configure turbo to run scripts in the root

* explicitly skip failing test for the Oxide version

* run oxide tests in CI

* only use build script for root package

* sync package-lock.json

* do not cache node_modules for each individual integration

* look for hoisted `.bin`

* use turbo for caching build tailwind css in integration tests

* Robin...

* try to use the local binary first

* skip installing integration test dependencies

Should already be installed due to workspace usage

* Robin...

* drop `output.clean`

* explicitly add `mini-css-extract-plugin`

* drop oxide-ci, this is tested by proxy

* ensure oxide build is used in integration tests

This will ensure the `@tailwindcss/oxide` dependency is available
(whether we use it or not).

* setup Oxide shim in insiders release

* add browserslist dependency

* use `install:all` script name

Just using `install` as a script name will be called when running
`npm install`.
Now that we marked the repo as a `workspace`, `npm install` will run
install in all workspaces which is... not ideal.

* tmp: enable insiders release in PRs

Just to check if everything works before merging. Can be removed once
tested.

* don't cache node_modules?

I feel there is some catch 22 going on here.
We require `npm install` to build the `oxide/crates/node` version.
But we also require `oxide/crates/node` for the `npm install` becaus of
the dependency: `"@tailwindcss/oxide": "file:oxide/creates/node"`

* try to use `oxide/crates/node` as part of the workspace

* let's think about this

Let's try and cache the `node_modules` and share as much as possible.
However, some scripts still need to be installed specific to the OS.

Running `npm install` locally doesn't throw away your `node_modules`,
so if we just cache `node_modules` but also run `npm install` that
should keep as much as possible and still improve install times since
`node_modules` is already there.

I think.

* ensure generated `index.js` and `index.d.ts` files are considered outputs

* use `npx napi` instead of `napi` directly

* include all `package-lock.json` files

* normalize caching further in all workflows

* drop nested `package-lock.json` files

* `npm uninstall mini-css-extract-plugin && npm install mini-css-extract-plugin --save-dev`

* bump webpack-5 integration tests dependencies

* only release insiders on `master` branch

* tmp: let's figure out release insiders oxide

* fix little typo

* use Node 18 for Oxide Insiders

* syncup package-lock.json

* let's try node 16

Node 18 currently fails on `Build x86_64-unknown-linux-gnu (OXIDE)`
Workflow.

Install Node.JS output:

```
Environment details
Warning: /__t/node/18.13.0/x64/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__t/node/18.13.0/x64/bin/node)
/__t/node/18.13.0/x64/bin/node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /__t/node/18.13.0/x64/bin/node)
/__t/node/18.13.0/x64/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /__t/node/18.13.0/x64/bin/node)
/__t/node/18.13.0/x64/bin/node: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /__t/node/18.13.0/x64/bin/node)
/__t/node/18.13.0/x64/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /__t/node/18.13.0/x64/bin/node)
/__t/node/18.13.0/x64/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /__t/node/18.13.0/x64/bin/node)

Warning: node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by node)
node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by node)
node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by node)
node: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by node)
node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by node)
node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by node)
```

* bump some Node versions

* only release oxide insiders on `master` branch

* don't cache `npm`

* bump napi-rs

Co-authored-by: Jordan Pittman <jordan@cryptica.me>
Co-authored-by: Adam Wathan <4323180+adamwathan@users.noreply.github.com>
2023-01-13 12:22:00 +01:00

208 lines
6 KiB
JavaScript

// @ts-check
import fs from 'fs'
import path from 'path'
import isGlob from 'is-glob'
import fastGlob from 'fast-glob'
import normalizePath from 'normalize-path'
import { parseGlob } from '../util/parseGlob'
import { env } from './sharedState'
/** @typedef {import('../../types/config.js').RawFile} RawFile */
/** @typedef {import('../../types/config.js').FilePath} FilePath */
/**
* @typedef {object} ContentPath
* @property {string} original
* @property {string} base
* @property {string | null} glob
* @property {boolean} ignore
* @property {string} pattern
*/
/**
* Turn a list of content paths (absolute or not; glob or not) into a list of
* absolute file paths that exist on the filesystem
*
* If there are symlinks in the path then multiple paths will be returned
* one for the symlink and one for the actual file
*
* @param {*} context
* @param {import('tailwindcss').Config} tailwindConfig
* @returns {ContentPath[]}
*/
export function parseCandidateFiles(context, tailwindConfig) {
let files = tailwindConfig.content.files
// Normalize the file globs
files = files.filter((filePath) => typeof filePath === 'string')
files = files.map(normalizePath)
// Split into included and excluded globs
let tasks = fastGlob.generateTasks(files)
/** @type {ContentPath[]} */
let included = []
/** @type {ContentPath[]} */
let excluded = []
for (const task of tasks) {
included.push(...task.positive.map((filePath) => parseFilePath(filePath, false)))
excluded.push(...task.negative.map((filePath) => parseFilePath(filePath, true)))
}
let paths = [...included, ...excluded]
// Resolve paths relative to the config file or cwd
paths = resolveRelativePaths(context, paths)
// Resolve symlinks if possible
paths = paths.flatMap(resolvePathSymlinks)
// Update cached patterns
paths = paths.map(resolveGlobPattern)
return paths
}
/**
*
* @param {string} filePath
* @param {boolean} ignore
* @returns {ContentPath}
*/
function parseFilePath(filePath, ignore) {
let contentPath = {
original: filePath,
base: filePath,
ignore,
pattern: filePath,
glob: null,
}
if (isGlob(filePath)) {
Object.assign(contentPath, parseGlob(filePath))
}
return contentPath
}
/**
*
* @param {ContentPath} contentPath
* @returns {ContentPath}
*/
function resolveGlobPattern(contentPath) {
// This is required for Windows support to properly pick up Glob paths.
// Afaik, this technically shouldn't be needed but there's probably
// some internal, direct path matching with a normalized path in
// a package which can't handle mixed directory separators
let base = normalizePath(contentPath.base)
// If the user's file path contains any special characters (like parens) for instance fast-glob
// is like "OOOH SHINY" and treats them as such. So we have to escape the base path to fix this
base = fastGlob.escapePath(base)
contentPath.pattern = contentPath.glob ? `${base}/${contentPath.glob}` : base
contentPath.pattern = contentPath.ignore ? `!${contentPath.pattern}` : contentPath.pattern
return contentPath
}
/**
* Resolve each path relative to the config file (when possible) if the experimental flag is enabled
* Otherwise, resolve relative to the current working directory
*
* @param {any} context
* @param {ContentPath[]} contentPaths
* @returns {ContentPath[]}
*/
function resolveRelativePaths(context, contentPaths) {
let resolveFrom = []
// Resolve base paths relative to the config file (when possible) if the experimental flag is enabled
if (context.userConfigPath && context.tailwindConfig.content.relative) {
resolveFrom = [path.dirname(context.userConfigPath)]
}
return contentPaths.map((contentPath) => {
contentPath.base = path.resolve(...resolveFrom, contentPath.base)
return contentPath
})
}
/**
* Resolve the symlink for the base directory / file in each path
* These are added as additional dependencies to watch for changes because
* some tools (like webpack) will only watch the actual file or directory
* but not the symlink itself even in projects that use monorepos.
*
* @param {ContentPath} contentPath
* @returns {ContentPath[]}
*/
function resolvePathSymlinks(contentPath) {
let paths = [contentPath]
try {
let resolvedPath = fs.realpathSync(contentPath.base)
if (resolvedPath !== contentPath.base) {
paths.push({
...contentPath,
base: resolvedPath,
})
}
} catch {
// TODO: log this?
}
return paths
}
/**
* @param {any} context
* @param {ContentPath[]} candidateFiles
* @param {Map<string, number>} fileModifiedMap
* @returns {[{ content: string, extension: string }[], Map<string, number>]}
*/
export function resolvedChangedContent(context, candidateFiles, fileModifiedMap) {
let changedContent = context.tailwindConfig.content.files
.filter((item) => typeof item.raw === 'string')
.map(({ raw, extension = 'html' }) => ({ content: raw, extension }))
let [changedFiles, mTimesToCommit] = resolveChangedFiles(candidateFiles, fileModifiedMap)
for (let changedFile of changedFiles) {
let extension = path.extname(changedFile).slice(1)
changedContent.push({ file: changedFile, extension })
}
return [changedContent, mTimesToCommit]
}
/**
*
* @param {ContentPath[]} candidateFiles
* @param {Map<string, number>} fileModifiedMap
* @returns {[Set<string>, Map<string, number>]}
*/
function resolveChangedFiles(candidateFiles, fileModifiedMap) {
let paths = candidateFiles.map((contentPath) => contentPath.pattern)
let mTimesToCommit = new Map()
let changedFiles = new Set()
env.DEBUG && console.time('Finding changed files')
let files = fastGlob.sync(paths, { absolute: true })
for (let file of files) {
let prevModified = fileModifiedMap.get(file) || -Infinity
let modified = fs.statSync(file).mtimeMs
if (modified > prevModified) {
changedFiles.add(file)
mTimesToCommit.set(file, modified)
}
}
env.DEBUG && console.timeEnd('Finding changed files')
return [changedFiles, mTimesToCommit]
}