Improve performance in bigger projects (#19632)
This PR improves the performance of Oxide when scanning large codebases.
The `Oxide` API, looks something like this:
```ts
let scanner = new Scanner({ sources })
let candidates = scanner.scan() // Found candidates
let files = scanner.files // Scanned files
let globs = scanner.globs // Scanned globs
```
The `files` and `globs` are used to tell PostCSS, Vite, webpack etc
which files to watch for changes.
The `.scan()` operation extracts the candidates from the source files.
You can think of these as potential Tailwind CSS classes.
In all these scenarios we have to walk the file system and find files
that match the `sources`.
### 1. Prevent multiple file system walks
The first big win came from the fact that accessing `.files` after a
`.scan()` also does an entire walk of the file system (for the given
`sources`), which is unnecessary because we just walked the file system.
This is something that's not really an issue in smaller codebases
because we have `mtime` tracking. We don't re-scan a file if its `mtime`
hasn't changed since the last scan. However, in large codebases with
thousands of files, even walking the file system to check `mtime`s can
be expensive.
### 2. Use parallel file system walking
Another big win is to use a parallel file system walker instead of a
synchronous one. The big problem here is that the parallel build has
20ms-50ms of overhead which is noticeable on small codebases. We don't
really know if you have a small or big codebase ahead of time, so maybe
some kind of hint in the future would be useful.
So the solution I settled on right now is to use a synchronous walker
for the initial scan, and then switch to a parallel walker for
subsequent scans (think dev mode). This gives us the best of both
worlds: fast initial scan on small codebases, and fast re-scans on large
codebases.
Caveat: if you use the `@tailwindcss/cli` we know exactly which files
changed so we can just re-scan those files directly without walking the
file system at all. But in `@tailwindcss/postcss` we don't know which
files changed, so we have to walk the file system to check `mtime`s.
While this improvement is nice, it resulted in an annoying issue related
to `mtime` tracking. Since the parallel walker processes files in
parallel, the `mtime` was typed as `Arc<Mutex<FxHashMap<PathBuf,
SystemTime>>>` so to avoid locking, I decided to only walk the files
here and collect their paths. Then later we check the `mtime` to know
whether to re-scan them or not.
Initially I just removed the `mtime` tracking altogether. But it did
have an impact when actually extracting candidates from those files, so
I added it back later.
### 3. Delaying work
I was still a bit annoyed by the fact that we had to track `mtime`
values for every file. This seems like annoying overhead, especially
when doing a single build (no dev mode).
So the trick I applied here is to only start tracking `mtime` values
after the initial scan.
This means that, in dev mode, we would do this:
1. Walk entire file system to track files.
2. On a subsequent scan, walk entire file system (again) and start
tracking `mtime` values. This time, we use the parallel walker instead
of the synchronous one.
3. On further scans, only re-scan files whose `mtime` has changed
The trade-off here is that on the second scan we always re-scan all
files, even if they haven't changed. Since this typically only happens
in dev mode, I think this is an acceptable trade-off especially if the
initial build is therefor faster this way.
### 3. Small wins
There are also a few small wins in here that I would like to mention but
that are less significant:
1. Pre-computed normalized `source` patterns instead of in every walker
filter call.
2. Tried to avoid some allocations in various places. For example the
`pre_process_input` always called `content.to_vec()` which allocates.
Instead we now accept an owned `Vec<u8>` so we don't have to call
`.to_vec()` in the default case (in my testing, this is ~92% of the time
in the codebases I checked).
3. Made the `Cursor` struct smaller, which is used a lot during
candidate extraction.
### Benchmarks
Now for the fun stuff, the benchmarks!
<details>
<summary>The code for the benchmarks</summary>
```ts
import path from 'node:path'
import { bench, boxplot, do_not_optimize, run, summary } from 'mitata'
import { Scanner as ScannerPr } from '/path/to/repo/with/pr/branch/tailwindcss/crates/node'
import { Scanner as ScannerMain } from '/path/to/repo/with/main/branch/tailwindcss/crates/node'
let base = '/path/to/some/codebase'
let sources = [{ base, pattern: '**/*', negated: false }]
// Verify the results are the same before benchmarking
let scannerPr = new ScannerPr({ sources })
let scannerMain = new ScannerMain({ sources })
{
let aCandidates = scannerPr.scan()
let bCandidates = scannerMain.scan()
if (aCandidates.length !== bCandidates.length) {
throw new Error(`Mismatch in candidate count: ${aCandidates.length} vs ${bCandidates.length}`)
}
for (let i = 0; i < aCandidates.length; i++) {
if (aCandidates[i] !== bCandidates[i]) {
throw new Error(`Mismatch in candidate at index ${i}: ${aCandidates[i]} vs ${bCandidates[i]}`)
}
}
let aFiles = scannerPr.files
let bFiles = scannerMain.files
if (aFiles.length !== bFiles.length) {
throw new Error(`Mismatch in file count: ${aFiles.length} vs ${bFiles.length}`)
}
for (let i = 0; i < aFiles.length; i++) {
if (aFiles[i] !== bFiles[i]) {
throw new Error(`Mismatch in file at index ${i}: ${aFiles[i]} vs ${bFiles[i]}`)
}
}
console.log('Scanned', aFiles.length, 'files')
console.log('Extracted', aCandidates.length, 'candidates')
console.log('Base =', base)
console.log()
}
summary(() => {
boxplot(() => {
bench('PR (build, .scan()))', function* () {
yield {
[0]() {
return new ScannerPr({ sources })
},
bench(scanner: ScannerPr) {
do_not_optimize(scanner.scan())
},
}
})
bench('main (build, .scan()))', function* () {
yield {
[0]() {
return new ScannerMain({ sources })
},
bench(scanner: ScannerMain) {
do_not_optimize(scanner.scan())
},
}
})
})
})
summary(() => {
boxplot(() => {
bench('PR (build, .scan() + .files)', function* () {
yield {
[0]() {
return new ScannerPr({ sources })
},
bench(scanner: ScannerPr) {
do_not_optimize(scanner.scan())
do_not_optimize(scanner.files)
},
}
})
bench('main (build, .scan() + .files)', function* () {
yield {
[0]() {
return new ScannerMain({ sources })
},
bench(scanner: ScannerMain) {
do_not_optimize(scanner.scan())
do_not_optimize(scanner.files)
},
}
})
})
})
summary(() => {
boxplot(() => {
bench('PR (watch, .scan()))', function* () {
yield {
bench() {
do_not_optimize(scannerPr.scan())
},
}
})
bench('main (watch, .scan()))', function* () {
yield {
bench() {
do_not_optimize(scannerMain.scan())
},
}
})
})
})
summary(() => {
boxplot(() => {
bench('PR (watch, .scan() + .files)', function* () {
yield {
bench() {
do_not_optimize(scannerPr.scan())
do_not_optimize(scannerPr.files)
},
}
})
bench('main (watch, .scan() + .files)', function* () {
yield {
bench() {
do_not_optimize(scannerMain.scan())
do_not_optimize(scannerMain.files)
},
}
})
})
})
await run()
```
</details>
#### tailwindcss.com codebase
```
Scanned 462 files
Extracted 13200 candidates
Base = /Users/robin/github.com/tailwindlabs/tailwindcss.com
clk: ~3.09 GHz
cpu: Apple M1 Max
runtime: bun 1.3.3 (arm64-darwin)
```
In these benchmarks the `PR` one is consistently faster than `main`.
It's not by a lot but that's mainly because the codebase itself isn't
that big. It is a codebase with _a lot_ of candidates though, but not
that many files.
The candidate extraction was already pretty fast, so the wins here
mainly come from avoiding re-walking the file system when accessing
`.files`, and from delaying `mtime` tracking until after the initial
scan.
**Single initial build**:
It's not a lot, but it's a bit faster. This is due to avoiding tracking
the `mtime` values initially and making some small optimizations related
to the struct size and allocations.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (build, .scan())) 22.87 ms/iter 23.28 ms █
(21.49 ms … 25.68 ms) 23.98 ms ▂ ▂ ▂ █ ▂▂
(832.00 kb … 2.69 mb) 1.41 mb ▆▆▆▆█▆▆█▁▆▆█▁▁█▁▆██▁▆
main (build, .scan())) 25.67 ms/iter 26.12 ms █ █ █
(24.54 ms … 27.74 ms) 27.06 ms █ █ █ ███
(432.00 kb … 2.78 mb) 996.00 kb ██▁████▁█▁████▁█▁▁█▁█
┌ ┐
╷ ┌─────┬──┐ ╷
PR (build, .scan())) ├────┤ │ ├─────┤
╵ └─────┴──┘ ╵
╷ ┌─────┬──┐ ╷
main (build, .scan())) ├──┤ │ ├───────┤
╵ └─────┴──┘ ╵
└ ┘
21.49 ms 24.28 ms 27.06 ms
summary
PR (build, .scan()))
1.12x faster than main (build, .scan()))
```
**Single initial build + accessing `.files`**:
We don't have to re-walk the entire file system even if we're just
dealing with ~462 scanned files.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (build, .scan() + .files) 22.54 ms/iter 22.99 ms █ ▂
(21.41 ms … 25.86 ms) 24.26 ms █ ▅ ▅▅█▅ ▅▅
(368.00 kb … 2.05 mb) 853.00 kb █▇█▇▇████▇▇██▁▁▇▁▁▇▁▇
main (build, .scan() + .files) 32.15 ms/iter 32.17 ms █ ▂
(30.78 ms … 36.22 ms) 35.75 ms ▅█ ▅█ ▅
(400.00 kb … 2.45 mb) 952.00 kb ██▁██▇▇█▇▁▁▁▁▁▁▁▁▁▁▁▇
┌ ┐
╷┌──┬┐ ╷
PR (build, .scan() + .files) ├┤ │├───┤
╵└──┴┘ ╵
╷┌───┬ ╷
main (build, .scan() + .files) ├┤ │──────────┤
╵└───┴ ╵
└ ┘
21.41 ms 28.58 ms 35.75 ms
summary
PR (build, .scan() + .files)
1.43x faster than main (build, .scan() + .files)
```
**Watch/dev mode, only scanning**:
This now switches to the parallel walker, but since it's not a super big
codebase we don't see a huge win here yet.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (watch, .scan())) 6.85 ms/iter 7.22 ms █▄
(6.34 ms … 7.94 ms) 7.91 ms ▄██▃
( 64.00 kb … 688.00 kb) 452.82 kb ▃████▆▂▂▁▂▁▂▁▁▁▅█▆▃▅▃
main (watch, .scan())) 7.92 ms/iter 8.08 ms █ █ ▃ █▃▃
(7.41 ms … 8.71 ms) 8.68 ms █▆█▆▃█████
( 0.00 b … 64.00 kb) 19.20 kb ▆▄██████████▆▁▆▆█▄▄▄▆
┌ ┐
╷ ┌──────┬──────┐ ╷
PR (watch, .scan())) ├──┤ │ ├────────────┤
╵ └──────┴──────┘ ╵
╷ ┌───┬──┐ ╷
main (watch, .scan())) ├────┤ │ ├───────────┤
╵ └───┴──┘ ╵
└ ┘
6.34 ms 7.51 ms 8.68 ms
summary
PR (watch, .scan()))
1.16x faster than main (watch, .scan()))
```
**Watch/dev mode, scanning + accessing `.files`**:
Again we avoid re-walking the entire file system when accessing
`.files`.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (watch, .scan() + .files) 12.10 ms/iter 12.74 ms █ █ █ █ ▃▃▃
(10.69 ms … 13.89 ms) 13.81 ms █ █▂▂▂ ▇█▂█▂███▇
(128.00 kb … 10.73 mb) 5.23 mb █▆████▁█████████▆▆▆▆▆
main (watch, .scan() + .files) 14.44 ms/iter 14.74 ms █
(13.93 ms … 15.33 ms) 15.18 ms ███▅ █ ▅ ▅
( 16.00 kb … 80.00 kb) 39.51 kb █▅████▁███▅▁█████▅▁▅▅
┌ ┐
╷ ┌──────┬──────┐ ╷
PR (watch, .scan() + .files) ├──────┤ │ ├─────────┤
╵ └──────┴──────┘ ╵
╷ ┌───┬──┐ ╷
main (watch, .scan() + .files) ├─┤ │ ├───┤
╵ └───┴──┘ ╵
└ ┘
10.69 ms 12.93 ms 15.18 ms
summary
PR (watch, .scan() + .files)
1.19x faster than main (watch, .scan() + .files)
```
#### Synthetic 5000 files codebase
Based on the instructions from #19616 I created a codebase with 5000
files. Each file contains a `flex` class and a unique class like
`content-['/path/to/file']` to ensure we have a decent amount of unique
candidates.
You can test the script yourself by running this:
```
mkdir -p fixtures/app-5000/src/components/{auth,dashboard,settings,profile,notifications,messages,search,navigation,footer,sidebar}/sub{001..500} && for dir in fixtures/app-5000/src/components/*/sub*; do echo "export const Component = () => <div className=\"flex content-['$dir']\">test</div>" > "$dir/index.tsx"; done && find fixtures/app-5000/src/components -type f | wc -lc
```
```
Scanned 5000 files
Extracted 5005 candidates
Base = /Users/robin/github.com/RobinMalfait/playground/scanner-benchmarks/fixtures/app-5000
clk: ~3.08 GHz
cpu: Apple M1 Max
runtime: bun 1.3.3 (arm64-darwin)
```
**Single initial build**:
As expected not a super big win here because it's a single build. But
there is a noticeable improvement.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (build, .scan())) 217.27 ms/iter 211.97 ms █
(205.99 ms … 289.53 ms) 214.33 ms ▅ ▅ ▅▅▅▅█▅▅ ▅
( 3.34 mb … 4.25 mb) 3.72 mb █▁▁▁▁▁▁█▁███████▁▁▁▁█
main (build, .scan())) 249.26 ms/iter 239.88 ms █
(231.51 ms … 381.66 ms) 241.01 ms ▅ ▅ ▅▅ ▅ █▅ ▅▅▅
( 4.22 mb … 4.78 mb) 4.49 mb █▁▁▁▁█▁██▁▁█▁▁██▁▁███
┌ ┐
╷ ┌─────╷──┬
PR (build, .scan())) ├────┤ ┤ │
╵ └─────╵──┴
╷ ┌───────╷
main (build, .scan())) ├───┤ ┤
╵ └───────╵
└ ┘
205.99 ms 223.50 ms 241.01 ms
summary
PR (build, .scan()))
1.15x faster than main (build, .scan()))
```
**Single initial build + accessing `.files`**:
Now things are getting interesting. Almost a 2x speedup by avoiding
re-walking the file system when accessing `.files`.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (build, .scan() + .files) 216.35 ms/iter 214.53 ms █ █ █
(211.00 ms … 242.64 ms) 221.45 ms █ █▅█ ▅▅ ▅ ▅
( 2.97 mb … 4.47 mb) 3.97 mb █▁███▁██▁▁▁▁▁▁█▁▁▁▁▁█
main (build, .scan() + .files) 414.79 ms/iter 406.05 ms ██
(396.72 ms … 542.30 ms) 413.69 ms ▅ ██▅ ▅▅ ▅ ▅ ▅
( 5.19 mb … 6.03 mb) 5.63 mb █▁▁▁███▁██▁█▁█▁▁▁▁▁▁█
┌ ┐
┌┬╷
PR (build, .scan() + .files) ││┤
└┴╵
╷┌──╷
main (build, .scan() + .files) ├┤ ┤
╵└──╵
└ ┘
211.00 ms 312.34 ms 413.69 ms
summary
PR (build, .scan() + .files)
1.92x faster than main (build, .scan() + .files)
```
**Watch/dev mode, only scanning**:
This is where we see bigger wins because now we're using the parallel
walker.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (watch, .scan())) 76.26 ms/iter 77.41 ms █
(73.56 ms … 79.02 ms) 77.81 ms ▅ ▅ ▅ ▅▅ ▅▅▅ ▅ █
( 2.53 mb … 5.52 mb) 3.06 mb █▁▁▁█▁▁█▁██▁███▁▁▁█▁█
main (watch, .scan())) 166.71 ms/iter 165.14 ms █ █
(161.49 ms … 198.26 ms) 168.99 ms █ ▅█ ▅▅▅ ▅ ▅ ▅
( 1.08 mb … 2.72 mb) 1.24 mb █▁██▁███▁▁█▁▁█▁▁▁▁▁▁█
┌ ┐
╷┬┐
PR (watch, .scan())) ├││
╵┴┘
╷┌─┬╷
main (watch, .scan())) ├┤ │┤
╵└─┴╵
└ ┘
73.56 ms 121.28 ms 168.99 ms
summary
PR (watch, .scan()))
2.19x faster than main (watch, .scan()))
```
**Watch/dev mode, scanning + accessing `.files`**:
This is the biggest win of them all because we have all the benefits
combined:
1. Avoiding re-walking the file system when accessing `.files`
2. Using the parallel walker for faster file system walking
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (watch, .scan() + .files) 84.04 ms/iter 84.84 ms █
(80.96 ms … 87.53 ms) 87.27 ms ▅▅ ▅▅ ▅▅ █▅ ▅ ▅
( 15.42 mb … 31.34 mb) 22.16 mb ██▁▁▁██▁██▁██▁█▁▁▁▁▁█
main (watch, .scan() + .files) 338.59 ms/iter 353.89 ms █
(321.87 ms … 378.43 ms) 358.70 ms █ █
( 2.39 mb … 2.45 mb) 2.42 mb ███▁▁██▁▁▁▁▁▁▁▁▁▁██▁█
┌ ┐
┬┐
PR (watch, .scan() + .files) ││
┴┘
┌──┬─┐╷
main (watch, .scan() + .files) │ │ ├┤
└──┴─┘╵
└ ┘
80.96 ms 219.83 ms 358.70 ms
summary
PR (watch, .scan() + .files)
4.03x faster than main (watch, .scan() + .files)
```
## Test plan
1. All existing tests still pass
2. All public APIs remain the same
3. In the benchmarks I'm sharing, I first verify that the candidates
returned and the files returned are the same before and after the
change.
4. Benchmarked against real codebases, and against a synthetic large
codebase (5000 files).
Fixes: #19616
This commit is contained in:
Robin Malfait
GitHub
parent
f212b0fc22
commit
095ff96ba3
28 changed files with 579 additions and 479 deletions
|
|
@ -28,6 +28,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
- Fix infinite loop when using `@variant` inside `@custom-variant` ([#19633](https://github.com/tailwindlabs/tailwindcss/pull/19633))
|
||||
- Allow multiples of `.25` in `aspect-*` fractions ([#19688](https://github.com/tailwindlabs/tailwindcss/pull/19688))
|
||||
- Ensure changes to external files listed via `@source` trigger a full page reload when using `@tailwindcss/vite` ([#19670](https://github.com/tailwindlabs/tailwindcss/pull/19670))
|
||||
- Improve performance Oxide scanner in bigger projects ([#19632](https://github.com/tailwindlabs/tailwindcss/pull/19632))
|
||||
|
||||
### Deprecated
|
||||
|
||||
|
|
|
|||
|
|
@ -1,44 +1,49 @@
|
|||
use std::{ascii::escape_default, fmt::Display};
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct Cursor<'a> {
|
||||
// The input we're scanning
|
||||
pub input: &'a [u8],
|
||||
|
||||
// The location of the cursor in the input
|
||||
pub pos: usize,
|
||||
|
||||
/// Is the cursor at the start of the input
|
||||
pub at_start: bool,
|
||||
|
||||
/// Is the cursor at the end of the input
|
||||
pub at_end: bool,
|
||||
|
||||
/// The previously consumed character
|
||||
/// If `at_start` is true, this will be NUL
|
||||
pub prev: u8,
|
||||
|
||||
/// The current character
|
||||
pub curr: u8,
|
||||
|
||||
/// The upcoming character (if any)
|
||||
/// If `at_end` is true, this will be NUL
|
||||
pub next: u8,
|
||||
}
|
||||
|
||||
impl<'a> Cursor<'a> {
|
||||
#[inline(always)]
|
||||
pub fn new(input: &'a [u8]) -> Self {
|
||||
let mut cursor = Self {
|
||||
input,
|
||||
pos: 0,
|
||||
at_start: true,
|
||||
at_end: false,
|
||||
prev: 0x00,
|
||||
curr: 0x00,
|
||||
next: 0x00,
|
||||
};
|
||||
cursor.move_to(0);
|
||||
cursor
|
||||
Self { input, pos: 0 }
|
||||
}
|
||||
|
||||
/// The current byte at `pos`, or 0x00 if past the end.
|
||||
#[inline(always)]
|
||||
pub fn curr(&self) -> u8 {
|
||||
if self.pos < self.input.len() {
|
||||
unsafe { *self.input.get_unchecked(self.pos) }
|
||||
} else {
|
||||
0x00
|
||||
}
|
||||
}
|
||||
|
||||
/// The next byte at `pos + 1`, or 0x00 if past the end.
|
||||
#[inline(always)]
|
||||
pub fn next(&self) -> u8 {
|
||||
let next_pos = self.pos + 1;
|
||||
if next_pos < self.input.len() {
|
||||
unsafe { *self.input.get_unchecked(next_pos) }
|
||||
} else {
|
||||
0x00
|
||||
}
|
||||
}
|
||||
|
||||
/// The previous byte at `pos - 1`, or 0x00 if at the start.
|
||||
#[inline(always)]
|
||||
pub fn prev(&self) -> u8 {
|
||||
if self.pos > 0 {
|
||||
unsafe { *self.input.get_unchecked(self.pos - 1) }
|
||||
} else {
|
||||
0x00
|
||||
}
|
||||
}
|
||||
|
||||
pub fn advance_by(&mut self, amount: usize) {
|
||||
|
|
@ -48,38 +53,15 @@ impl<'a> Cursor<'a> {
|
|||
#[inline(always)]
|
||||
pub fn advance(&mut self) {
|
||||
self.pos += 1;
|
||||
|
||||
self.prev = self.curr;
|
||||
self.curr = self.next;
|
||||
self.next = *self
|
||||
.input
|
||||
.get(self.pos.saturating_add(1))
|
||||
.unwrap_or(&0x00u8);
|
||||
}
|
||||
|
||||
#[inline(always)]
|
||||
pub fn advance_twice(&mut self) {
|
||||
self.pos += 2;
|
||||
|
||||
self.prev = self.next;
|
||||
self.curr = *self.input.get(self.pos).unwrap_or(&0x00u8);
|
||||
self.next = *self
|
||||
.input
|
||||
.get(self.pos.saturating_add(1))
|
||||
.unwrap_or(&0x00u8);
|
||||
}
|
||||
|
||||
pub fn move_to(&mut self, pos: usize) {
|
||||
let len = self.input.len();
|
||||
let pos = pos.clamp(0, len);
|
||||
|
||||
self.pos = pos;
|
||||
self.at_start = pos == 0;
|
||||
self.at_end = pos + 1 >= len;
|
||||
|
||||
self.prev = *self.input.get(pos.wrapping_sub(1)).unwrap_or(&0x00u8);
|
||||
self.curr = *self.input.get(pos).unwrap_or(&0x00u8);
|
||||
self.next = *self.input.get(pos.saturating_add(1)).unwrap_or(&0x00u8);
|
||||
self.pos = pos.min(self.input.len());
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -90,9 +72,9 @@ impl Display for Cursor<'_> {
|
|||
let pos = format!("{: >len_count$}", self.pos, len_count = len.len());
|
||||
write!(f, "{}/{} ", pos, len)?;
|
||||
|
||||
if self.at_start {
|
||||
if self.pos == 0 {
|
||||
write!(f, "S ")?;
|
||||
} else if self.at_end {
|
||||
} else if self.pos + 1 >= self.input.len() {
|
||||
write!(f, "E ")?;
|
||||
} else {
|
||||
write!(f, "M ")?;
|
||||
|
|
@ -109,9 +91,9 @@ impl Display for Cursor<'_> {
|
|||
write!(
|
||||
f,
|
||||
"[{} {} {}]",
|
||||
to_str(self.prev),
|
||||
to_str(self.curr),
|
||||
to_str(self.next)
|
||||
to_str(self.prev()),
|
||||
to_str(self.curr()),
|
||||
to_str(self.next())
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -125,36 +107,28 @@ mod test {
|
|||
fn test_cursor() {
|
||||
let mut cursor = Cursor::new(b"hello world");
|
||||
assert_eq!(cursor.pos, 0);
|
||||
assert!(cursor.at_start);
|
||||
assert!(!cursor.at_end);
|
||||
assert_eq!(cursor.prev, 0x00);
|
||||
assert_eq!(cursor.curr, b'h');
|
||||
assert_eq!(cursor.next, b'e');
|
||||
assert_eq!(cursor.prev(), 0x00);
|
||||
assert_eq!(cursor.curr(), b'h');
|
||||
assert_eq!(cursor.next(), b'e');
|
||||
|
||||
cursor.advance_by(1);
|
||||
assert_eq!(cursor.pos, 1);
|
||||
assert!(!cursor.at_start);
|
||||
assert!(!cursor.at_end);
|
||||
assert_eq!(cursor.prev, b'h');
|
||||
assert_eq!(cursor.curr, b'e');
|
||||
assert_eq!(cursor.next, b'l');
|
||||
assert_eq!(cursor.prev(), b'h');
|
||||
assert_eq!(cursor.curr(), b'e');
|
||||
assert_eq!(cursor.next(), b'l');
|
||||
|
||||
// Advancing too far should stop at the end
|
||||
cursor.advance_by(10);
|
||||
assert_eq!(cursor.pos, 11);
|
||||
assert!(!cursor.at_start);
|
||||
assert!(cursor.at_end);
|
||||
assert_eq!(cursor.prev, b'd');
|
||||
assert_eq!(cursor.curr, 0x00);
|
||||
assert_eq!(cursor.next, 0x00);
|
||||
assert_eq!(cursor.prev(), b'd');
|
||||
assert_eq!(cursor.curr(), 0x00);
|
||||
assert_eq!(cursor.next(), 0x00);
|
||||
|
||||
// Can't advance past the end
|
||||
cursor.advance_by(1);
|
||||
assert_eq!(cursor.pos, 11);
|
||||
assert!(!cursor.at_start);
|
||||
assert!(cursor.at_end);
|
||||
assert_eq!(cursor.prev, b'd');
|
||||
assert_eq!(cursor.curr, 0x00);
|
||||
assert_eq!(cursor.next, 0x00);
|
||||
assert_eq!(cursor.prev(), b'd');
|
||||
assert_eq!(cursor.curr(), 0x00);
|
||||
assert_eq!(cursor.next(), 0x00);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -74,7 +74,7 @@ impl Machine for ArbitraryPropertyMachine<IdleState> {
|
|||
|
||||
#[inline]
|
||||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match cursor.curr.into() {
|
||||
match cursor.curr().into() {
|
||||
// Start of an arbitrary property
|
||||
Class::OpenBracket => {
|
||||
self.start_pos = cursor.pos;
|
||||
|
|
@ -97,8 +97,8 @@ impl Machine for ArbitraryPropertyMachine<ParsingPropertyState> {
|
|||
let len = cursor.input.len();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr.into() {
|
||||
Class::Dash => match cursor.next.into() {
|
||||
match cursor.curr().into() {
|
||||
Class::Dash => match cursor.next().into() {
|
||||
// Start of a CSS variable
|
||||
//
|
||||
// E.g.: `[--my-color:red]`
|
||||
|
|
@ -137,7 +137,7 @@ impl ArbitraryPropertyMachine<ParsingPropertyState> {
|
|||
fn parse_property_variable(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match self.css_variable_machine.next(cursor) {
|
||||
MachineState::Idle => self.restart(),
|
||||
MachineState::Done(_) => match cursor.next.into() {
|
||||
MachineState::Done(_) => match cursor.next().into() {
|
||||
// End of the CSS variable, must be followed by a `:`
|
||||
//
|
||||
// E.g.: `[--my-color:red]`
|
||||
|
|
@ -165,8 +165,8 @@ impl Machine for ArbitraryPropertyMachine<ParsingValueState> {
|
|||
let len = cursor.input.len();
|
||||
let start_of_value_pos = cursor.pos;
|
||||
while cursor.pos < len {
|
||||
match cursor.curr.into() {
|
||||
Class::Escape => match cursor.next.into() {
|
||||
match cursor.curr().into() {
|
||||
Class::Escape => match cursor.next().into() {
|
||||
// An escaped whitespace character is not allowed
|
||||
//
|
||||
// E.g.: `[color:var(--my-\ color)]`
|
||||
|
|
@ -181,7 +181,7 @@ impl Machine for ArbitraryPropertyMachine<ParsingValueState> {
|
|||
},
|
||||
|
||||
Class::OpenParen | Class::OpenBracket | Class::OpenCurly => {
|
||||
if !self.bracket_stack.push(cursor.curr) {
|
||||
if !self.bracket_stack.push(cursor.curr()) {
|
||||
return self.restart();
|
||||
}
|
||||
cursor.advance();
|
||||
|
|
@ -190,7 +190,7 @@ impl Machine for ArbitraryPropertyMachine<ParsingValueState> {
|
|||
Class::CloseParen | Class::CloseBracket | Class::CloseCurly
|
||||
if !self.bracket_stack.is_empty() =>
|
||||
{
|
||||
if !self.bracket_stack.pop(cursor.curr) {
|
||||
if !self.bracket_stack.pop(cursor.curr()) {
|
||||
return self.restart();
|
||||
}
|
||||
cursor.advance();
|
||||
|
|
@ -227,7 +227,7 @@ impl Machine for ArbitraryPropertyMachine<ParsingValueState> {
|
|||
Class::Slash if start_of_value_pos == cursor.pos => return self.restart(),
|
||||
|
||||
// String interpolation-like syntax is not allowed. E.g.: `[${x}]`
|
||||
Class::Dollar if matches!(cursor.next.into(), Class::OpenCurly) => {
|
||||
Class::Dollar if matches!(cursor.next().into(), Class::OpenCurly) => {
|
||||
return self.restart()
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ impl Machine for ArbitraryValueMachine {
|
|||
#[inline]
|
||||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
// An arbitrary value must start with an open bracket
|
||||
if Class::OpenBracket != cursor.curr.into() {
|
||||
if Class::OpenBracket != cursor.curr().into() {
|
||||
return MachineState::Idle;
|
||||
}
|
||||
|
||||
|
|
@ -42,8 +42,8 @@ impl Machine for ArbitraryValueMachine {
|
|||
let len = cursor.input.len();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr.into() {
|
||||
Class::Escape => match cursor.next.into() {
|
||||
match cursor.curr().into() {
|
||||
Class::Escape => match cursor.next().into() {
|
||||
// An escaped whitespace character is not allowed
|
||||
//
|
||||
// E.g.: `[color:var(--my-\ color)]`
|
||||
|
|
@ -61,7 +61,7 @@ impl Machine for ArbitraryValueMachine {
|
|||
},
|
||||
|
||||
Class::OpenParen | Class::OpenBracket | Class::OpenCurly => {
|
||||
if !self.bracket_stack.push(cursor.curr) {
|
||||
if !self.bracket_stack.push(cursor.curr()) {
|
||||
return self.restart();
|
||||
}
|
||||
cursor.advance();
|
||||
|
|
@ -70,7 +70,7 @@ impl Machine for ArbitraryValueMachine {
|
|||
Class::CloseParen | Class::CloseBracket | Class::CloseCurly
|
||||
if !self.bracket_stack.is_empty() =>
|
||||
{
|
||||
if !self.bracket_stack.pop(cursor.curr) {
|
||||
if !self.bracket_stack.pop(cursor.curr()) {
|
||||
return self.restart();
|
||||
}
|
||||
cursor.advance();
|
||||
|
|
@ -96,7 +96,7 @@ impl Machine for ArbitraryValueMachine {
|
|||
Class::Whitespace => return self.restart(),
|
||||
|
||||
// String interpolation-like syntax is not allowed. E.g.: `[${x}]`
|
||||
Class::Dollar if matches!(cursor.next.into(), Class::OpenCurly) => {
|
||||
Class::Dollar if matches!(cursor.next().into(), Class::OpenCurly) => {
|
||||
return self.restart()
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -80,13 +80,13 @@ impl Machine for ArbitraryVariableMachine<IdleState> {
|
|||
|
||||
#[inline(always)]
|
||||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match cursor.curr.into() {
|
||||
match cursor.curr().into() {
|
||||
// Arbitrary variables start with `(` followed by a CSS variable
|
||||
//
|
||||
// E.g.: `(--my-variable)`
|
||||
// ^^
|
||||
//
|
||||
Class::OpenParen => match cursor.next.into() {
|
||||
Class::OpenParen => match cursor.next().into() {
|
||||
Class::Dash => {
|
||||
self.start_pos = cursor.pos;
|
||||
cursor.advance();
|
||||
|
|
@ -117,7 +117,7 @@ impl Machine for ArbitraryVariableMachine<ParsingState> {
|
|||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match self.css_variable_machine.next(cursor) {
|
||||
MachineState::Idle => self.restart(),
|
||||
MachineState::Done(_) => match cursor.next.into() {
|
||||
MachineState::Done(_) => match cursor.next().into() {
|
||||
// A CSS variable followed by a `,` means that there is a fallback
|
||||
//
|
||||
// E.g.: `(--my-color,red)`
|
||||
|
|
@ -134,7 +134,7 @@ impl Machine for ArbitraryVariableMachine<ParsingState> {
|
|||
_ => {
|
||||
cursor.advance();
|
||||
|
||||
match cursor.curr.into() {
|
||||
match cursor.curr().into() {
|
||||
// End of an arbitrary variable, must be followed by `)`
|
||||
Class::CloseParen => self.done(self.start_pos, cursor),
|
||||
|
||||
|
|
@ -156,7 +156,7 @@ impl Machine for ArbitraryVariableMachine<ParsingDataTypeState> {
|
|||
let len = cursor.input.len();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr.into() {
|
||||
match cursor.curr().into() {
|
||||
// Valid data type characters
|
||||
//
|
||||
// E.g.: `(length:--my-length)`
|
||||
|
|
@ -169,7 +169,7 @@ impl Machine for ArbitraryVariableMachine<ParsingDataTypeState> {
|
|||
//
|
||||
// E.g.: `(length:--my-length)`
|
||||
// ^
|
||||
Class::Colon => match cursor.next.into() {
|
||||
Class::Colon => match cursor.next().into() {
|
||||
Class::Dash => {
|
||||
cursor.advance();
|
||||
return self.transition::<ParsingState>().next(cursor);
|
||||
|
|
@ -196,8 +196,8 @@ impl Machine for ArbitraryVariableMachine<ParsingFallbackState> {
|
|||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
let len = cursor.input.len();
|
||||
while cursor.pos < len {
|
||||
match cursor.curr.into() {
|
||||
Class::Escape => match cursor.next.into() {
|
||||
match cursor.curr().into() {
|
||||
Class::Escape => match cursor.next().into() {
|
||||
// An escaped whitespace character is not allowed
|
||||
//
|
||||
// E.g.: `(--my-\ color)`
|
||||
|
|
@ -212,7 +212,7 @@ impl Machine for ArbitraryVariableMachine<ParsingFallbackState> {
|
|||
},
|
||||
|
||||
Class::OpenParen | Class::OpenBracket | Class::OpenCurly => {
|
||||
if !self.bracket_stack.push(cursor.curr) {
|
||||
if !self.bracket_stack.push(cursor.curr()) {
|
||||
return self.restart();
|
||||
}
|
||||
cursor.advance();
|
||||
|
|
@ -221,7 +221,7 @@ impl Machine for ArbitraryVariableMachine<ParsingFallbackState> {
|
|||
Class::CloseParen | Class::CloseBracket | Class::CloseCurly
|
||||
if !self.bracket_stack.is_empty() =>
|
||||
{
|
||||
if !self.bracket_stack.pop(cursor.curr) {
|
||||
if !self.bracket_stack.pop(cursor.curr()) {
|
||||
return self.restart();
|
||||
}
|
||||
cursor.advance();
|
||||
|
|
@ -253,7 +253,7 @@ impl Machine for ArbitraryVariableMachine<ParsingFallbackState> {
|
|||
Class::Whitespace => return self.restart(),
|
||||
|
||||
// String interpolation-like syntax is not allowed. E.g.: `[${x}]`
|
||||
Class::Dollar if matches!(cursor.next.into(), Class::OpenCurly) => {
|
||||
Class::Dollar if matches!(cursor.next().into(), Class::OpenCurly) => {
|
||||
return self.restart()
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -32,14 +32,14 @@ impl Machine for CandidateMachine {
|
|||
while cursor.pos < len {
|
||||
// Skip ahead for known characters that will never be part of a candidate. No need to
|
||||
// run any sub-machines.
|
||||
if cursor.curr.is_ascii_whitespace() {
|
||||
if cursor.curr().is_ascii_whitespace() {
|
||||
self.reset();
|
||||
cursor.advance();
|
||||
continue;
|
||||
}
|
||||
|
||||
// Candidates don't start with these characters, so we can skip ahead.
|
||||
if matches!(cursor.curr, b':' | b'"' | b'\'' | b'`') {
|
||||
if matches!(cursor.curr(), b':' | b'"' | b'\'' | b'`') {
|
||||
self.reset();
|
||||
cursor.advance();
|
||||
continue;
|
||||
|
|
@ -56,7 +56,7 @@ impl Machine for CandidateMachine {
|
|||
// E.g.: `Some Class`
|
||||
// ^ ^ Invalid, we can jump ahead to the next boundary
|
||||
//
|
||||
if matches!(cursor.curr, b'<' | b'A'..=b'Z') {
|
||||
if matches!(cursor.curr(), b'<' | b'A'..=b'Z') {
|
||||
if let Some(offset) = cursor.input[cursor.pos..]
|
||||
.iter()
|
||||
.position(|&c| is_valid_before_boundary(&c))
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ impl Machine for CssVariableMachine {
|
|||
#[inline]
|
||||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
// CSS Variables must start with `--`
|
||||
if Class::Dash != cursor.curr.into() || Class::Dash != cursor.next.into() {
|
||||
if Class::Dash != cursor.curr().into() || Class::Dash != cursor.next().into() {
|
||||
return MachineState::Idle;
|
||||
}
|
||||
|
||||
|
|
@ -30,11 +30,11 @@ impl Machine for CssVariableMachine {
|
|||
cursor.advance_twice();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr.into() {
|
||||
match cursor.curr().into() {
|
||||
// https://drafts.csswg.org/css-syntax-3/#ident-token-diagram
|
||||
//
|
||||
Class::AllowedCharacter | Class::Dash => {
|
||||
match cursor.next.into() {
|
||||
match cursor.next().into() {
|
||||
// Valid character followed by a valid character or an escape character
|
||||
//
|
||||
// E.g.: `--my-variable`
|
||||
|
|
@ -58,7 +58,7 @@ impl Machine for CssVariableMachine {
|
|||
}
|
||||
}
|
||||
|
||||
Class::Escape => match cursor.next.into() {
|
||||
Class::Escape => match cursor.next().into() {
|
||||
// An escaped whitespace character is not allowed
|
||||
//
|
||||
// In CSS it is allowed, but in the context of a class it's not because then we
|
||||
|
|
|
|||
|
|
@ -86,7 +86,7 @@ impl<'a> Extractor<'a> {
|
|||
{
|
||||
let cursor = &mut self.cursor.clone();
|
||||
while cursor.pos < len {
|
||||
if cursor.curr.is_ascii_whitespace() {
|
||||
if cursor.curr().is_ascii_whitespace() {
|
||||
cursor.advance();
|
||||
continue;
|
||||
}
|
||||
|
|
@ -104,7 +104,7 @@ impl<'a> Extractor<'a> {
|
|||
let cursor = &mut self.cursor.clone();
|
||||
|
||||
while cursor.pos < len {
|
||||
if cursor.curr.is_ascii_whitespace() {
|
||||
if cursor.curr().is_ascii_whitespace() {
|
||||
cursor.advance();
|
||||
continue;
|
||||
}
|
||||
|
|
@ -147,7 +147,7 @@ impl<'a> Extractor<'a> {
|
|||
|
||||
let cursor = &mut self.cursor.clone();
|
||||
while cursor.pos < len {
|
||||
if cursor.curr.is_ascii_whitespace() {
|
||||
if cursor.curr().is_ascii_whitespace() {
|
||||
cursor.advance();
|
||||
continue;
|
||||
}
|
||||
|
|
@ -238,7 +238,7 @@ mod tests {
|
|||
use std::hint::black_box;
|
||||
|
||||
fn pre_process_input(input: &str, extension: &str) -> String {
|
||||
let input = crate::scanner::pre_process_input(input.as_bytes(), extension);
|
||||
let input = crate::scanner::pre_process_input(input.as_bytes().to_vec(), extension);
|
||||
String::from_utf8(input).unwrap()
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -31,14 +31,14 @@ impl Machine for ModifierMachine {
|
|||
#[inline]
|
||||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
// A modifier must start with a `/`, everything else is not a valid start of a modifier
|
||||
if Class::Slash != cursor.curr.into() {
|
||||
if Class::Slash != cursor.curr().into() {
|
||||
return MachineState::Idle;
|
||||
}
|
||||
|
||||
let start_pos = cursor.pos;
|
||||
cursor.advance();
|
||||
|
||||
match cursor.curr.into() {
|
||||
match cursor.curr().into() {
|
||||
// Start of an arbitrary value:
|
||||
//
|
||||
// ```
|
||||
|
|
@ -70,9 +70,9 @@ impl Machine for ModifierMachine {
|
|||
Class::ValidStart => {
|
||||
let len = cursor.input.len();
|
||||
while cursor.pos < len {
|
||||
match cursor.curr.into() {
|
||||
match cursor.curr().into() {
|
||||
Class::ValidStart | Class::ValidInside => {
|
||||
match cursor.next.into() {
|
||||
match cursor.next().into() {
|
||||
// Only valid characters are allowed, if followed by another valid character
|
||||
Class::ValidStart | Class::ValidInside => cursor.advance(),
|
||||
|
||||
|
|
|
|||
|
|
@ -52,8 +52,8 @@ impl Machine for NamedUtilityMachine<IdleState> {
|
|||
|
||||
#[inline]
|
||||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match cursor.curr.into() {
|
||||
Class::AlphaLower => match cursor.next.into() {
|
||||
match cursor.curr().into() {
|
||||
Class::AlphaLower => match cursor.next().into() {
|
||||
// Valid single character utility in between quotes
|
||||
//
|
||||
// E.g.: `<div class="a"></div>`
|
||||
|
|
@ -90,7 +90,7 @@ impl Machine for NamedUtilityMachine<IdleState> {
|
|||
//
|
||||
// E.g.: `-mx-2.5`
|
||||
// ^^
|
||||
Class::Dash => match cursor.next.into() {
|
||||
Class::Dash => match cursor.next().into() {
|
||||
Class::AlphaLower => {
|
||||
self.start_pos = cursor.pos;
|
||||
cursor.advance();
|
||||
|
|
@ -118,7 +118,7 @@ impl Machine for NamedUtilityMachine<ParsingState> {
|
|||
let len = cursor.input.len();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr.into() {
|
||||
match cursor.curr().into() {
|
||||
// Followed by a boundary character, we are at the end of the utility.
|
||||
//
|
||||
// E.g.: `'flex'`
|
||||
|
|
@ -132,14 +132,14 @@ impl Machine for NamedUtilityMachine<ParsingState> {
|
|||
// E.g.: `:div="{ flex: true }"` (JavaScript object syntax)
|
||||
// ^
|
||||
Class::AlphaLower | Class::AlphaUpper => {
|
||||
if is_valid_after_boundary(&cursor.next) || {
|
||||
if is_valid_after_boundary(&cursor.next()) || {
|
||||
// Or any of these characters
|
||||
//
|
||||
// - `:`, because of JS object keys
|
||||
// - `/`, because of modifiers
|
||||
// - `!`, because of important
|
||||
matches!(
|
||||
cursor.next.into(),
|
||||
cursor.next().into(),
|
||||
Class::Colon | Class::Slash | Class::Exclamation
|
||||
)
|
||||
} {
|
||||
|
|
@ -150,7 +150,7 @@ impl Machine for NamedUtilityMachine<ParsingState> {
|
|||
cursor.advance()
|
||||
}
|
||||
|
||||
Class::Dash => match cursor.next.into() {
|
||||
Class::Dash => match cursor.next().into() {
|
||||
// Start of an arbitrary value
|
||||
//
|
||||
// E.g.: `bg-[#0088cc]`
|
||||
|
|
@ -196,7 +196,7 @@ impl Machine for NamedUtilityMachine<ParsingState> {
|
|||
_ => return self.restart(),
|
||||
},
|
||||
|
||||
Class::Underscore => match cursor.next.into() {
|
||||
Class::Underscore => match cursor.next().into() {
|
||||
// Valid characters _if_ followed by another valid character. These characters are
|
||||
// only valid inside of the utility but not at the end of the utility.
|
||||
//
|
||||
|
|
@ -225,14 +225,14 @@ impl Machine for NamedUtilityMachine<ParsingState> {
|
|||
// ^
|
||||
// E.g.: `:div="{ flex: true }"` (JavaScript object syntax)
|
||||
// ^
|
||||
_ if is_valid_after_boundary(&cursor.next) || {
|
||||
_ if is_valid_after_boundary(&cursor.next()) || {
|
||||
// Or any of these characters
|
||||
//
|
||||
// - `:`, because of JS object keys
|
||||
// - `/`, because of modifiers
|
||||
// - `!`, because of important
|
||||
matches!(
|
||||
cursor.next.into(),
|
||||
cursor.next().into(),
|
||||
Class::Colon | Class::Slash | Class::Exclamation
|
||||
)
|
||||
} =>
|
||||
|
|
@ -249,11 +249,11 @@ impl Machine for NamedUtilityMachine<ParsingState> {
|
|||
// E.g.: `px-2.5`
|
||||
// ^^^
|
||||
Class::Dot => {
|
||||
if !matches!(cursor.prev.into(), Class::Number) {
|
||||
if !matches!(cursor.prev().into(), Class::Number) {
|
||||
return self.restart();
|
||||
}
|
||||
|
||||
if !matches!(cursor.next.into(), Class::Number) {
|
||||
if !matches!(cursor.next().into(), Class::Number) {
|
||||
return self.restart();
|
||||
}
|
||||
|
||||
|
|
@ -278,7 +278,7 @@ impl Machine for NamedUtilityMachine<ParsingState> {
|
|||
//
|
||||
Class::Number => {
|
||||
if !matches!(
|
||||
cursor.prev.into(),
|
||||
cursor.prev().into(),
|
||||
Class::Dash
|
||||
| Class::Underscore
|
||||
| Class::Dot
|
||||
|
|
@ -290,7 +290,7 @@ impl Machine for NamedUtilityMachine<ParsingState> {
|
|||
}
|
||||
|
||||
if !matches!(
|
||||
cursor.next.into(),
|
||||
cursor.next().into(),
|
||||
Class::Dot
|
||||
| Class::Number
|
||||
| Class::AlphaLower
|
||||
|
|
@ -314,7 +314,7 @@ impl Machine for NamedUtilityMachine<ParsingState> {
|
|||
// ^^
|
||||
// ```
|
||||
Class::Percent => {
|
||||
if !matches!(cursor.prev.into(), Class::Number) {
|
||||
if !matches!(cursor.prev().into(), Class::Number) {
|
||||
return self.restart();
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -81,8 +81,8 @@ impl Machine for NamedVariantMachine<IdleState> {
|
|||
|
||||
#[inline(always)]
|
||||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match cursor.curr.into() {
|
||||
Class::AlphaLower | Class::Star => match cursor.next.into() {
|
||||
match cursor.curr().into() {
|
||||
Class::AlphaLower | Class::Star => match cursor.next().into() {
|
||||
// Valid single character variant, must be followed by a `:`
|
||||
//
|
||||
// E.g.: `<div class="x:flex"></div>`
|
||||
|
|
@ -134,8 +134,8 @@ impl Machine for NamedVariantMachine<ParsingState> {
|
|||
let len = cursor.input.len();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr.into() {
|
||||
Class::Dash => match cursor.next.into() {
|
||||
match cursor.curr().into() {
|
||||
Class::Dash => match cursor.next().into() {
|
||||
// Start of an arbitrary value
|
||||
//
|
||||
// E.g.: `data-[state=pending]:`.
|
||||
|
|
@ -192,7 +192,7 @@ impl Machine for NamedVariantMachine<ParsingState> {
|
|||
};
|
||||
}
|
||||
|
||||
Class::Underscore => match cursor.next.into() {
|
||||
Class::Underscore => match cursor.next().into() {
|
||||
// Valid characters _if_ followed by another valid character. These characters are
|
||||
// only valid inside of the variant but not at the end of the variant.
|
||||
//
|
||||
|
|
@ -240,11 +240,11 @@ impl Machine for NamedVariantMachine<ParsingState> {
|
|||
// E.g.: `2.5xl:flex`
|
||||
// ^^^
|
||||
Class::Dot => {
|
||||
if !matches!(cursor.prev.into(), Class::Number) {
|
||||
if !matches!(cursor.prev().into(), Class::Number) {
|
||||
return self.restart();
|
||||
}
|
||||
|
||||
if !matches!(cursor.next.into(), Class::Number) {
|
||||
if !matches!(cursor.next().into(), Class::Number) {
|
||||
return self.restart();
|
||||
}
|
||||
|
||||
|
|
@ -263,7 +263,7 @@ impl Machine for NamedVariantMachine<ParsingState> {
|
|||
impl NamedVariantMachine<ParsingState> {
|
||||
#[inline(always)]
|
||||
fn parse_arbitrary_end(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match cursor.next.into() {
|
||||
match cursor.next().into() {
|
||||
Class::Slash => {
|
||||
cursor.advance();
|
||||
self.transition::<ParsingModifierState>().next(cursor)
|
||||
|
|
@ -285,7 +285,7 @@ impl Machine for NamedVariantMachine<ParsingModifierState> {
|
|||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match self.modifier_machine.next(cursor) {
|
||||
MachineState::Idle => self.restart(),
|
||||
MachineState::Done(_) => match cursor.next.into() {
|
||||
MachineState::Done(_) => match cursor.next().into() {
|
||||
// Modifier must be followed by a `:`
|
||||
//
|
||||
// E.g.: `group-hover/name:`
|
||||
|
|
@ -308,7 +308,7 @@ impl Machine for NamedVariantMachine<ParsingEndState> {
|
|||
|
||||
#[inline(always)]
|
||||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match cursor.curr.into() {
|
||||
match cursor.curr().into() {
|
||||
// The end of a variant must be the `:`
|
||||
//
|
||||
// E.g.: `hover:`
|
||||
|
|
|
|||
|
|
@ -32,14 +32,14 @@ impl PreProcessor for Clojure {
|
|||
let mut cursor = cursor::Cursor::new(&content);
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Consume strings as-is
|
||||
b'"' => {
|
||||
result[cursor.pos] = b' ';
|
||||
cursor.advance();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Escaped character, skip ahead to the next character
|
||||
b'\\' => cursor.advance_twice(),
|
||||
|
||||
|
|
@ -57,8 +57,8 @@ impl PreProcessor for Clojure {
|
|||
|
||||
// Discard line comments until the end of the line.
|
||||
// Comments start with `;;`
|
||||
b';' if matches!(cursor.next, b';') => {
|
||||
while cursor.pos < len && cursor.curr != b'\n' {
|
||||
b';' if matches!(cursor.next(), b';') => {
|
||||
while cursor.pos < len && cursor.curr() != b'\n' {
|
||||
result[cursor.pos] = b' ';
|
||||
cursor.advance();
|
||||
}
|
||||
|
|
@ -70,7 +70,7 @@ impl PreProcessor for Clojure {
|
|||
cursor.advance();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// A `.` surrounded by digits is a decimal number, so we don't want to replace it.
|
||||
//
|
||||
// E.g.:
|
||||
|
|
@ -78,8 +78,8 @@ impl PreProcessor for Clojure {
|
|||
// gap-1.5
|
||||
// ^
|
||||
// ```
|
||||
b'.' if cursor.prev.is_ascii_digit()
|
||||
&& cursor.next.is_ascii_digit() =>
|
||||
b'.' if cursor.prev().is_ascii_digit()
|
||||
&& cursor.next().is_ascii_digit() =>
|
||||
{
|
||||
// Keep the `.` as-is
|
||||
}
|
||||
|
|
@ -95,7 +95,7 @@ impl PreProcessor for Clojure {
|
|||
result[cursor.pos] = b' ';
|
||||
}
|
||||
// End of keyword.
|
||||
_ if !is_keyword_character(cursor.curr) => {
|
||||
_ if !is_keyword_character(cursor.curr()) => {
|
||||
result[cursor.pos] = b' ';
|
||||
break;
|
||||
}
|
||||
|
|
@ -110,11 +110,11 @@ impl PreProcessor for Clojure {
|
|||
|
||||
// Handle quote with a list, e.g.: `'(…)`
|
||||
// and with a vector, e.g.: `'[…]`
|
||||
b'\'' if matches!(cursor.next, b'[' | b'(') => {
|
||||
b'\'' if matches!(cursor.next(), b'[' | b'(') => {
|
||||
result[cursor.pos] = b' ';
|
||||
cursor.advance();
|
||||
result[cursor.pos] = b' ';
|
||||
let end = match cursor.curr {
|
||||
let end = match cursor.curr() {
|
||||
b'[' => b']',
|
||||
b'(' => b')',
|
||||
_ => unreachable!(),
|
||||
|
|
@ -122,7 +122,7 @@ impl PreProcessor for Clojure {
|
|||
|
||||
// Consume until the closing `]`
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
x if x == end => {
|
||||
result[cursor.pos] = b' ';
|
||||
break;
|
||||
|
|
@ -134,7 +134,7 @@ impl PreProcessor for Clojure {
|
|||
cursor.advance();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Escaped character, skip ahead to the next character
|
||||
b'\\' => cursor.advance_twice(),
|
||||
|
||||
|
|
@ -157,14 +157,14 @@ impl PreProcessor for Clojure {
|
|||
}
|
||||
|
||||
// Handle quote with a keyword, e.g.: `'bg-white`
|
||||
b'\'' if !cursor.next.is_ascii_whitespace() => {
|
||||
b'\'' if !cursor.next().is_ascii_whitespace() => {
|
||||
result[cursor.pos] = b' ';
|
||||
cursor.advance();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// End of keyword.
|
||||
_ if !is_keyword_character(cursor.curr) => {
|
||||
_ if !is_keyword_character(cursor.curr()) => {
|
||||
result[cursor.pos] = b' ';
|
||||
break;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,13 +13,13 @@ impl PreProcessor for Elixir {
|
|||
|
||||
while cursor.pos < content.len() {
|
||||
// Look for a sigil marker
|
||||
if cursor.curr != b'~' {
|
||||
if cursor.curr() != b'~' {
|
||||
cursor.advance();
|
||||
continue;
|
||||
}
|
||||
|
||||
// Scan charlists, strings, and wordlists
|
||||
if !matches!(cursor.next, b'c' | b'C' | b's' | b'S' | b'w' | b'W') {
|
||||
if !matches!(cursor.next(), b'c' | b'C' | b's' | b'S' | b'w' | b'W') {
|
||||
cursor.advance();
|
||||
continue;
|
||||
}
|
||||
|
|
@ -27,7 +27,7 @@ impl PreProcessor for Elixir {
|
|||
cursor.advance_twice();
|
||||
|
||||
// Match the opening for a sigil
|
||||
if !matches!(cursor.curr, b'(' | b'[' | b'{') {
|
||||
if !matches!(cursor.curr(), b'(' | b'[' | b'{') {
|
||||
continue;
|
||||
}
|
||||
|
||||
|
|
@ -35,19 +35,19 @@ impl PreProcessor for Elixir {
|
|||
result[cursor.pos] = b' ';
|
||||
|
||||
// Scan until we find a balanced closing one and replace it too
|
||||
bracket_stack.push(cursor.curr);
|
||||
bracket_stack.push(cursor.curr());
|
||||
|
||||
while cursor.pos < content.len() {
|
||||
cursor.advance();
|
||||
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Escaped character, skip ahead to the next character
|
||||
b'\\' => cursor.advance_twice(),
|
||||
b'(' | b'[' | b'{' => {
|
||||
bracket_stack.push(cursor.curr);
|
||||
bracket_stack.push(cursor.curr());
|
||||
}
|
||||
b')' | b']' | b'}' if !bracket_stack.is_empty() => {
|
||||
bracket_stack.pop(cursor.curr);
|
||||
bracket_stack.pop(cursor.curr());
|
||||
|
||||
if bracket_stack.is_empty() {
|
||||
// Replace the closing bracket with a space
|
||||
|
|
|
|||
|
|
@ -101,7 +101,7 @@ impl PreProcessor for Haml {
|
|||
let mut last_newline_position = 0;
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Escape the next character
|
||||
b'\\' => {
|
||||
cursor.advance_twice();
|
||||
|
|
@ -119,7 +119,7 @@ impl PreProcessor for Haml {
|
|||
b'-' if cursor.input[last_newline_position..cursor.pos]
|
||||
.iter()
|
||||
.all(u8::is_ascii_whitespace)
|
||||
&& matches!(cursor.next, b'#') =>
|
||||
&& matches!(cursor.next(), b'#') =>
|
||||
{
|
||||
// Just consume the comment
|
||||
let updated_last_newline_position =
|
||||
|
|
@ -159,7 +159,7 @@ impl PreProcessor for Haml {
|
|||
// Override the last known newline position
|
||||
last_newline_position = end;
|
||||
|
||||
let replaced = pre_process_input(ruby_code, "rb");
|
||||
let replaced = pre_process_input(ruby_code.to_vec(), "rb");
|
||||
result.replace_range(start..end, replaced);
|
||||
}
|
||||
|
||||
|
|
@ -190,7 +190,7 @@ impl PreProcessor for Haml {
|
|||
// digit.
|
||||
// E.g.: `bg-red-500.2xl:flex`
|
||||
// ^^^
|
||||
if cursor.prev.is_ascii_digit() && cursor.next.is_ascii_digit() {
|
||||
if cursor.prev().is_ascii_digit() && cursor.next().is_ascii_digit() {
|
||||
let mut next_cursor = cursor.clone();
|
||||
next_cursor.advance();
|
||||
|
||||
|
|
@ -213,11 +213,11 @@ impl PreProcessor for Haml {
|
|||
if bracket_stack.is_empty() {
|
||||
result[cursor.pos] = b' ';
|
||||
}
|
||||
bracket_stack.push(cursor.curr);
|
||||
bracket_stack.push(cursor.curr());
|
||||
}
|
||||
|
||||
b')' | b']' | b'}' if !bracket_stack.is_empty() => {
|
||||
bracket_stack.pop(cursor.curr);
|
||||
bracket_stack.pop(cursor.curr());
|
||||
|
||||
// Replace closing bracket with a space
|
||||
if bracket_stack.is_empty() {
|
||||
|
|
@ -257,7 +257,7 @@ impl Haml {
|
|||
// :url => { :action => "add", :id => product.id },
|
||||
// :update => { :success => "cart", :failure => "error" }
|
||||
// ```
|
||||
let evaluation_type = cursor.curr;
|
||||
let evaluation_type = cursor.curr();
|
||||
|
||||
let block_indentation_level = cursor
|
||||
.pos
|
||||
|
|
@ -267,17 +267,17 @@ impl Haml {
|
|||
let mut last_newline_position = last_known_newline_position;
|
||||
|
||||
// Consume until the end of the line first
|
||||
while cursor.pos < len && cursor.curr != b'\n' {
|
||||
while cursor.pos < len && cursor.curr() != b'\n' {
|
||||
cursor.advance();
|
||||
}
|
||||
|
||||
// Block is already done, aka just a line
|
||||
if evaluation_type == b'=' && cursor.prev != b',' {
|
||||
if evaluation_type == b'=' && cursor.prev() != b',' {
|
||||
return cursor.pos;
|
||||
}
|
||||
|
||||
'outer: while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Escape the next character
|
||||
b'\\' => {
|
||||
cursor.advance_twice();
|
||||
|
|
@ -289,7 +289,7 @@ impl Haml {
|
|||
last_newline_position = cursor.pos;
|
||||
|
||||
// We are done with this block
|
||||
if evaluation_type == b'=' && cursor.prev != b',' {
|
||||
if evaluation_type == b'=' && cursor.prev() != b',' {
|
||||
break;
|
||||
}
|
||||
|
||||
|
|
@ -300,11 +300,11 @@ impl Haml {
|
|||
// Skip whitespace and compute the indentation level
|
||||
x if x.is_ascii_whitespace() => {
|
||||
// Find first non-whitespace character
|
||||
while cursor.pos < len && cursor.curr.is_ascii_whitespace() {
|
||||
if cursor.curr == b'\n' {
|
||||
while cursor.pos < len && cursor.curr().is_ascii_whitespace() {
|
||||
if cursor.curr() == b'\n' {
|
||||
last_newline_position = cursor.pos;
|
||||
|
||||
if evaluation_type == b'=' && cursor.prev != b',' {
|
||||
if evaluation_type == b'=' && cursor.prev() != b',' {
|
||||
// We are done with this block
|
||||
break 'outer;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,13 +11,13 @@ impl PreProcessor for Json {
|
|||
let mut cursor = cursor::Cursor::new(content);
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Consume strings as-is
|
||||
b'"' => {
|
||||
cursor.advance();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Escaped character, skip ahead to the next character
|
||||
b'\\' => cursor.advance_twice(),
|
||||
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ impl PreProcessor for Markdown {
|
|||
let mut in_directive = false;
|
||||
|
||||
while cursor.pos < len {
|
||||
match (in_directive, cursor.curr) {
|
||||
match (in_directive, cursor.curr()) {
|
||||
(false, b'{') => {
|
||||
result[cursor.pos] = b' ';
|
||||
in_directive = true;
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ impl PreProcessor for Pug {
|
|||
let mut bracket_stack = BracketStack::default();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Only replace `.` with a space if it's not surrounded by numbers. E.g.:
|
||||
//
|
||||
// ```diff
|
||||
|
|
@ -43,7 +43,7 @@ impl PreProcessor for Pug {
|
|||
// digit.
|
||||
// E.g.: `bg-red-500.2xl:flex`
|
||||
// ^^^
|
||||
if cursor.prev.is_ascii_digit() && cursor.next.is_ascii_digit() {
|
||||
if cursor.prev().is_ascii_digit() && cursor.next().is_ascii_digit() {
|
||||
let mut next_cursor = cursor.clone();
|
||||
next_cursor.advance();
|
||||
|
||||
|
|
@ -68,17 +68,17 @@ impl PreProcessor for Pug {
|
|||
//
|
||||
// However, we also need to make sure that we keep the parens that are part of the
|
||||
// utility class. E.g.: `bg-(--my-color)`.
|
||||
b'(' if bracket_stack.is_empty() && !matches!(cursor.prev, b'-' | b'/') => {
|
||||
b'(' if bracket_stack.is_empty() && !matches!(cursor.prev(), b'-' | b'/') => {
|
||||
result[cursor.pos] = b' ';
|
||||
bracket_stack.push(cursor.curr);
|
||||
bracket_stack.push(cursor.curr());
|
||||
}
|
||||
|
||||
b'(' | b'[' | b'{' => {
|
||||
bracket_stack.push(cursor.curr);
|
||||
bracket_stack.push(cursor.curr());
|
||||
}
|
||||
|
||||
b')' | b']' | b'}' if !bracket_stack.is_empty() => {
|
||||
bracket_stack.pop(cursor.curr);
|
||||
bracket_stack.pop(cursor.curr());
|
||||
}
|
||||
|
||||
// Consume everything else
|
||||
|
|
|
|||
|
|
@ -68,7 +68,8 @@ impl PreProcessor for Ruby {
|
|||
}
|
||||
|
||||
let body = &content_as_str[body_start..body_end];
|
||||
let replaced = pre_process_input(body.as_bytes(), &lang.to_ascii_lowercase());
|
||||
let replaced =
|
||||
pre_process_input(body.as_bytes().to_vec(), &lang.to_ascii_lowercase());
|
||||
|
||||
result.replace_range(body_start..body_end, replaced);
|
||||
break;
|
||||
|
|
@ -77,12 +78,12 @@ impl PreProcessor for Ruby {
|
|||
|
||||
// Ruby extraction
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
b'"' => {
|
||||
cursor.advance();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Escaped character, skip ahead to the next character
|
||||
b'\\' => cursor.advance_twice(),
|
||||
|
||||
|
|
@ -102,7 +103,7 @@ impl PreProcessor for Ruby {
|
|||
cursor.advance();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Escaped character, skip ahead to the next character
|
||||
b'\\' => cursor.advance_twice(),
|
||||
|
||||
|
|
@ -123,12 +124,12 @@ impl PreProcessor for Ruby {
|
|||
// Except for strict locals, these are defined in a `<%# locals: … %>`. Checking if
|
||||
// the comment is preceded by a `%` should be enough without having to perform more
|
||||
// parsing logic. Worst case we _do_ scan a few comments.
|
||||
b'#' if !matches!(cursor.prev, b'%') => {
|
||||
b'#' if !matches!(cursor.prev(), b'%') => {
|
||||
result[cursor.pos] = b' ';
|
||||
cursor.advance();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// End of the comment
|
||||
b'\n' => break,
|
||||
|
||||
|
|
@ -148,7 +149,7 @@ impl PreProcessor for Ruby {
|
|||
}
|
||||
|
||||
// Looking for `%w`, `%W`, or `%p`
|
||||
if cursor.curr != b'%' || !matches!(cursor.next, b'w' | b'W' | b'p') {
|
||||
if cursor.curr() != b'%' || !matches!(cursor.next(), b'w' | b'W' | b'p') {
|
||||
cursor.advance();
|
||||
continue;
|
||||
}
|
||||
|
|
@ -156,7 +157,7 @@ impl PreProcessor for Ruby {
|
|||
cursor.advance_twice();
|
||||
|
||||
// Boundary character
|
||||
let boundary = match cursor.curr {
|
||||
let boundary = match cursor.curr() {
|
||||
b'[' => b']',
|
||||
b'(' => b')',
|
||||
b'{' => b'}',
|
||||
|
|
@ -177,11 +178,11 @@ impl PreProcessor for Ruby {
|
|||
cursor.advance();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Skip escaped characters
|
||||
b'\\' => {
|
||||
// Use backslash to embed spaces in the strings.
|
||||
if cursor.next == b' ' {
|
||||
if cursor.next() == b' ' {
|
||||
result[cursor.pos] = b' ';
|
||||
}
|
||||
|
||||
|
|
@ -190,19 +191,19 @@ impl PreProcessor for Ruby {
|
|||
|
||||
// Start of a nested bracket
|
||||
b'[' | b'(' | b'{' => {
|
||||
bracket_stack.push(cursor.curr);
|
||||
bracket_stack.push(cursor.curr());
|
||||
}
|
||||
|
||||
// End of a nested bracket
|
||||
b']' | b')' | b'}' if !bracket_stack.is_empty() => {
|
||||
if !bracket_stack.pop(cursor.curr) {
|
||||
if !bracket_stack.pop(cursor.curr()) {
|
||||
// Unbalanced
|
||||
cursor.advance();
|
||||
}
|
||||
}
|
||||
|
||||
// End of the pattern, replace the boundary character with a space
|
||||
_ if cursor.curr == boundary => {
|
||||
_ if cursor.curr() == boundary => {
|
||||
if boundary != b'\n' {
|
||||
result[cursor.pos] = b' ';
|
||||
}
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ impl Rust {
|
|||
let mut bracket_stack = bracket_stack::BracketStack::default();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Escaped character, skip ahead to the next character
|
||||
b'\\' => {
|
||||
cursor.advance_twice();
|
||||
|
|
@ -46,7 +46,7 @@ impl Rust {
|
|||
cursor.advance();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Escaped character, skip ahead to the next character
|
||||
b'\\' => cursor.advance_twice(),
|
||||
|
||||
|
|
@ -89,7 +89,7 @@ impl Rust {
|
|||
// digit.
|
||||
// E.g.: `bg-red-500.2xl:flex`
|
||||
// ^^^
|
||||
if cursor.prev.is_ascii_digit() && cursor.next.is_ascii_digit() {
|
||||
if cursor.prev().is_ascii_digit() && cursor.next().is_ascii_digit() {
|
||||
let mut next_cursor = cursor.clone();
|
||||
next_cursor.advance();
|
||||
|
||||
|
|
@ -103,11 +103,11 @@ impl Rust {
|
|||
}
|
||||
|
||||
b'[' => {
|
||||
bracket_stack.push(cursor.curr);
|
||||
bracket_stack.push(cursor.curr());
|
||||
}
|
||||
|
||||
b']' if !bracket_stack.is_empty() => {
|
||||
bracket_stack.pop(cursor.curr);
|
||||
bracket_stack.pop(cursor.curr());
|
||||
}
|
||||
|
||||
// Consume everything else
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ impl PreProcessor for Slim {
|
|||
let mut bracket_stack = BracketStack::default();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr {
|
||||
match cursor.curr() {
|
||||
// Only replace `.` with a space if it's not surrounded by numbers. E.g.:
|
||||
//
|
||||
// ```diff
|
||||
|
|
@ -43,7 +43,7 @@ impl PreProcessor for Slim {
|
|||
// digit.
|
||||
// E.g.: `bg-red-500.2xl:flex`
|
||||
// ^^^
|
||||
if cursor.prev.is_ascii_digit() && cursor.next.is_ascii_digit() {
|
||||
if cursor.prev().is_ascii_digit() && cursor.next().is_ascii_digit() {
|
||||
let mut next_cursor = cursor.clone();
|
||||
next_cursor.advance();
|
||||
|
||||
|
|
@ -65,7 +65,7 @@ impl PreProcessor for Slim {
|
|||
// class=%w[bg-blue-500 w-10 h-10]
|
||||
// ]
|
||||
// ```
|
||||
b'%' if matches!(cursor.next, b'w' | b'W')
|
||||
b'%' if matches!(cursor.next(), b'w' | b'W')
|
||||
&& matches!(cursor.input.get(cursor.pos + 2), Some(b'[' | b'(' | b'{')) =>
|
||||
{
|
||||
result[cursor.pos] = b' '; // Replace `%`
|
||||
|
|
@ -73,7 +73,7 @@ impl PreProcessor for Slim {
|
|||
result[cursor.pos] = b' '; // Replace `w`
|
||||
cursor.advance();
|
||||
result[cursor.pos] = b' '; // Replace `[` or `(` or `{`
|
||||
bracket_stack.push(cursor.curr);
|
||||
bracket_stack.push(cursor.curr());
|
||||
cursor.advance(); // Move past the bracket
|
||||
continue;
|
||||
}
|
||||
|
|
@ -96,10 +96,10 @@ impl PreProcessor for Slim {
|
|||
// Instead of listing all boundary characters, let's list the characters we know
|
||||
// will be invalid instead.
|
||||
b'[' if bracket_stack.is_empty()
|
||||
&& matches!(cursor.prev, b'a'..=b'z' | b'A'..=b'Z' | b'0'..=b'9') =>
|
||||
&& matches!(cursor.prev(), b'a'..=b'z' | b'A'..=b'Z' | b'0'..=b'9') =>
|
||||
{
|
||||
result[cursor.pos] = b' ';
|
||||
bracket_stack.push(cursor.curr);
|
||||
bracket_stack.push(cursor.curr());
|
||||
}
|
||||
|
||||
// In Slim the class name shorthand can be followed by a parenthesis. E.g.:
|
||||
|
|
@ -114,17 +114,17 @@ impl PreProcessor for Slim {
|
|||
//
|
||||
// However, we also need to make sure that we keep the parens that are part of the
|
||||
// utility class. E.g.: `bg-(--my-color)`.
|
||||
b'(' if bracket_stack.is_empty() && !matches!(cursor.prev, b'-' | b'/') => {
|
||||
b'(' if bracket_stack.is_empty() && !matches!(cursor.prev(), b'-' | b'/') => {
|
||||
result[cursor.pos] = b' ';
|
||||
bracket_stack.push(cursor.curr);
|
||||
bracket_stack.push(cursor.curr());
|
||||
}
|
||||
|
||||
b'(' | b'[' | b'{' => {
|
||||
bracket_stack.push(cursor.curr);
|
||||
bracket_stack.push(cursor.curr());
|
||||
}
|
||||
|
||||
b')' | b']' | b'}' if !bracket_stack.is_empty() => {
|
||||
bracket_stack.pop(cursor.curr);
|
||||
bracket_stack.pop(cursor.curr());
|
||||
}
|
||||
|
||||
// Consume everything else
|
||||
|
|
|
|||
|
|
@ -14,13 +14,12 @@ pub struct Vue;
|
|||
impl PreProcessor for Vue {
|
||||
fn process(&self, content: &[u8]) -> Vec<u8> {
|
||||
let mut result = content.to_vec();
|
||||
|
||||
let content_as_str = std::str::from_utf8(content).unwrap();
|
||||
for (_, [lang, body]) in TEMPLATE_REGEX
|
||||
.captures_iter(content_as_str)
|
||||
.map(|c| c.extract())
|
||||
{
|
||||
let replaced = pre_process_input(body.as_bytes(), lang);
|
||||
let replaced = pre_process_input(body.as_bytes().to_vec(), lang);
|
||||
result = result.replace(body, replaced);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -30,20 +30,20 @@ impl Machine for StringMachine {
|
|||
|
||||
#[inline]
|
||||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
if Class::Quote != cursor.curr.into() {
|
||||
if Class::Quote != cursor.curr().into() {
|
||||
return MachineState::Idle;
|
||||
}
|
||||
|
||||
// Start of a string
|
||||
let len = cursor.input.len();
|
||||
let start_pos = cursor.pos;
|
||||
let end_char = cursor.curr;
|
||||
let end_char = cursor.curr();
|
||||
|
||||
cursor.advance();
|
||||
|
||||
while cursor.pos < len {
|
||||
match cursor.curr.into() {
|
||||
Class::Escape => match cursor.next.into() {
|
||||
match cursor.curr().into() {
|
||||
Class::Escape => match cursor.next().into() {
|
||||
// An escaped whitespace character is not allowed
|
||||
Class::Whitespace => return MachineState::Idle,
|
||||
|
||||
|
|
@ -52,7 +52,7 @@ impl Machine for StringMachine {
|
|||
},
|
||||
|
||||
// End of the string
|
||||
Class::Quote if cursor.curr == end_char => return self.done(start_pos, cursor),
|
||||
Class::Quote if cursor.curr() == end_char => return self.done(start_pos, cursor),
|
||||
|
||||
// Any kind of whitespace is not allowed
|
||||
Class::Whitespace => return MachineState::Idle,
|
||||
|
|
|
|||
|
|
@ -27,12 +27,12 @@ impl Machine for UtilityMachine {
|
|||
|
||||
#[inline]
|
||||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match cursor.curr.into() {
|
||||
match cursor.curr().into() {
|
||||
// LEGACY: Important marker
|
||||
Class::Exclamation => {
|
||||
self.legacy_important = true;
|
||||
|
||||
match cursor.next.into() {
|
||||
match cursor.next().into() {
|
||||
// Start of an arbitrary property
|
||||
//
|
||||
// E.g.: `![color:red]`
|
||||
|
|
@ -78,7 +78,7 @@ impl UtilityMachine {
|
|||
fn parse_arbitrary_property(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match self.arbitrary_property_machine.next(cursor) {
|
||||
MachineState::Idle => self.restart(),
|
||||
MachineState::Done(_) => match cursor.next.into() {
|
||||
MachineState::Done(_) => match cursor.next().into() {
|
||||
// End of arbitrary property, but there is a potential modifier.
|
||||
//
|
||||
// E.g.: `[color:#0088cc]/`
|
||||
|
|
@ -109,7 +109,7 @@ impl UtilityMachine {
|
|||
fn parse_named_utility(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match self.named_utility_machine.next(cursor) {
|
||||
MachineState::Idle => self.restart(),
|
||||
MachineState::Done(_) => match cursor.next.into() {
|
||||
MachineState::Done(_) => match cursor.next().into() {
|
||||
// End of a named utility, but there is a potential modifier.
|
||||
//
|
||||
// E.g.: `bg-red-500/`
|
||||
|
|
@ -140,7 +140,7 @@ impl UtilityMachine {
|
|||
fn parse_modifier(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match self.modifier_machine.next(cursor) {
|
||||
MachineState::Idle => self.restart(),
|
||||
MachineState::Done(_) => match cursor.next.into() {
|
||||
MachineState::Done(_) => match cursor.next().into() {
|
||||
// A modifier followed by a modifier is invalid
|
||||
Class::Slash => self.restart(),
|
||||
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ impl Machine for VariantMachine {
|
|||
|
||||
#[inline]
|
||||
fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState {
|
||||
match cursor.curr.into() {
|
||||
match cursor.curr().into() {
|
||||
// Start of an arbitrary variant
|
||||
//
|
||||
// E.g.: `[&:hover]:`
|
||||
|
|
@ -48,7 +48,7 @@ impl VariantMachine {
|
|||
start_pos: usize,
|
||||
cursor: &mut cursor::Cursor<'_>,
|
||||
) -> MachineState {
|
||||
match cursor.next.into() {
|
||||
match cursor.next().into() {
|
||||
// End of an arbitrary value, must be followed by a `:`
|
||||
//
|
||||
// E.g.: `[&:hover]:`
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ pub fn fast_skip(cursor: &Cursor) -> Option<usize> {
|
|||
return None;
|
||||
}
|
||||
|
||||
if !cursor.curr.is_ascii_whitespace() {
|
||||
if !cursor.curr().is_ascii_whitespace() {
|
||||
return None;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -30,11 +30,9 @@ fn sort_by_dir_and_name(a: &DirEntry, z: &DirEntry) -> Ordering {
|
|||
|
||||
pub fn resolve_globs(
|
||||
base: PathBuf,
|
||||
dirs: &[PathBuf],
|
||||
dirs: &FxHashSet<PathBuf>,
|
||||
extensions: &FxHashSet<String>,
|
||||
) -> Vec<GlobEntry> {
|
||||
let allowed_paths: FxHashSet<PathBuf> = FxHashSet::from_iter(dirs.iter().cloned());
|
||||
|
||||
// A list of known extensions + a list of extensions we found in the project.
|
||||
let mut found_extensions: FxHashSet<String> =
|
||||
FxHashSet::from_iter(KNOWN_EXTENSIONS.iter().map(|x| x.to_string()));
|
||||
|
|
@ -72,7 +70,7 @@ pub fn resolve_globs(
|
|||
continue;
|
||||
}
|
||||
|
||||
if !allowed_paths.contains(path) {
|
||||
if !dirs.contains(path) {
|
||||
let mut path = path;
|
||||
while let Some(parent) = path.parent() {
|
||||
if parent == base {
|
||||
|
|
@ -113,7 +111,7 @@ pub fn resolve_globs(
|
|||
continue;
|
||||
}
|
||||
|
||||
if !allowed_paths.contains(path) {
|
||||
if !dirs.contains(path) {
|
||||
continue;
|
||||
}
|
||||
|
||||
|
|
|
|||
70
crates/oxide/src/scanner/init_tracing.rs
Normal file
70
crates/oxide/src/scanner/init_tracing.rs
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
use std::fs::OpenOptions;
|
||||
use std::io::{self, Write};
|
||||
use std::path::Path;
|
||||
use std::sync::{self, Arc, Mutex};
|
||||
use tracing_subscriber::fmt::writer::BoxMakeWriter;
|
||||
|
||||
pub static SHOULD_TRACE: sync::LazyLock<bool> = sync::LazyLock::new(
|
||||
|| matches!(std::env::var("DEBUG"), Ok(value) if value.eq("*") || (value.contains("tailwindcss:oxide") && !value.contains("-tailwindcss:oxide"))),
|
||||
);
|
||||
|
||||
fn dim(input: &str) -> String {
|
||||
format!("\u{001b}[2m{input}\u{001b}[22m")
|
||||
}
|
||||
|
||||
fn blue(input: &str) -> String {
|
||||
format!("\u{001b}[34m{input}\u{001b}[39m")
|
||||
}
|
||||
|
||||
fn highlight(input: &str) -> String {
|
||||
format!("{}{}{}", dim(&blue("`")), blue(input), dim(&blue("`")))
|
||||
}
|
||||
|
||||
struct MutexWriter(Arc<Mutex<std::fs::File>>);
|
||||
|
||||
impl Write for MutexWriter {
|
||||
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
|
||||
self.0.lock().unwrap().write(buf)
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> io::Result<()> {
|
||||
self.0.lock().unwrap().flush()
|
||||
}
|
||||
}
|
||||
|
||||
pub fn init_tracing() {
|
||||
if !*SHOULD_TRACE {
|
||||
return;
|
||||
}
|
||||
|
||||
let file_path = format!("tailwindcss-{}.log", std::process::id());
|
||||
let file = OpenOptions::new()
|
||||
.create(true)
|
||||
.append(true)
|
||||
.open(&file_path)
|
||||
.unwrap_or_else(|_| panic!("Failed to open {file_path}"));
|
||||
|
||||
let file_path = Path::new(&file_path);
|
||||
let absolute_file_path = dunce::canonicalize(file_path)
|
||||
.unwrap_or_else(|_| panic!("Failed to canonicalize {file_path:?}"));
|
||||
eprintln!(
|
||||
"{} Writing debug info to: {}\n",
|
||||
dim("[DEBUG]"),
|
||||
highlight(absolute_file_path.as_path().to_str().unwrap())
|
||||
);
|
||||
|
||||
let file = Arc::new(Mutex::new(file));
|
||||
|
||||
let writer: BoxMakeWriter = BoxMakeWriter::new({
|
||||
let file = file.clone();
|
||||
move || Box::new(MutexWriter(file.clone())) as Box<dyn Write + Send>
|
||||
});
|
||||
|
||||
_ = tracing_subscriber::fmt()
|
||||
.with_max_level(tracing::Level::INFO)
|
||||
.with_span_events(tracing_subscriber::fmt::format::FmtSpan::ACTIVE)
|
||||
.with_writer(writer)
|
||||
.with_ansi(false)
|
||||
.compact()
|
||||
.try_init();
|
||||
}
|
||||
|
|
@ -1,5 +1,6 @@
|
|||
pub mod auto_source_detection;
|
||||
pub mod detect_sources;
|
||||
pub mod init_tracing;
|
||||
pub mod sources;
|
||||
|
||||
use crate::extractor::{Extracted, Extractor};
|
||||
|
|
@ -14,15 +15,13 @@ use bstr::ByteSlice;
|
|||
use fast_glob::glob_match;
|
||||
use fxhash::{FxHashMap, FxHashSet};
|
||||
use ignore::{gitignore::GitignoreBuilder, WalkBuilder};
|
||||
use init_tracing::{init_tracing, SHOULD_TRACE};
|
||||
use rayon::prelude::*;
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
use std::fs::OpenOptions;
|
||||
use std::io::{self, Write};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::{self, Arc, Mutex};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::SystemTime;
|
||||
use tracing::event;
|
||||
use tracing_subscriber::fmt::writer::BoxMakeWriter;
|
||||
|
||||
// @source "some/folder"; // This is auto source detection
|
||||
// @source "some/folder/**/*"; // This is auto source detection
|
||||
|
|
@ -34,70 +33,6 @@ use tracing_subscriber::fmt::writer::BoxMakeWriter;
|
|||
//
|
||||
// @source "do-include-me.bin"; // `.bin` is typically ignored, but now it's explicit so should be included
|
||||
// @source "git-ignored.html"; // A git ignored file that is listed explicitly, should be scanned
|
||||
static SHOULD_TRACE: sync::LazyLock<bool> = sync::LazyLock::new(
|
||||
|| matches!(std::env::var("DEBUG"), Ok(value) if value.eq("*") || (value.contains("tailwindcss:oxide") && !value.contains("-tailwindcss:oxide"))),
|
||||
);
|
||||
|
||||
fn dim(input: &str) -> String {
|
||||
format!("\u{001b}[2m{input}\u{001b}[22m")
|
||||
}
|
||||
|
||||
fn blue(input: &str) -> String {
|
||||
format!("\u{001b}[34m{input}\u{001b}[39m")
|
||||
}
|
||||
|
||||
fn highlight(input: &str) -> String {
|
||||
format!("{}{}{}", dim(&blue("`")), blue(input), dim(&blue("`")))
|
||||
}
|
||||
|
||||
fn init_tracing() {
|
||||
if !*SHOULD_TRACE {
|
||||
return;
|
||||
}
|
||||
|
||||
let file_path = format!("tailwindcss-{}.log", std::process::id());
|
||||
let file = OpenOptions::new()
|
||||
.create(true)
|
||||
.append(true)
|
||||
.open(&file_path)
|
||||
.unwrap_or_else(|_| panic!("Failed to open {file_path}"));
|
||||
|
||||
let file_path = Path::new(&file_path);
|
||||
let absolute_file_path = dunce::canonicalize(file_path)
|
||||
.unwrap_or_else(|_| panic!("Failed to canonicalize {file_path:?}"));
|
||||
eprintln!(
|
||||
"{} Writing debug info to: {}\n",
|
||||
dim("[DEBUG]"),
|
||||
highlight(absolute_file_path.as_path().to_str().unwrap())
|
||||
);
|
||||
|
||||
let file = Arc::new(Mutex::new(file));
|
||||
|
||||
let writer: BoxMakeWriter = BoxMakeWriter::new({
|
||||
let file = file.clone();
|
||||
move || Box::new(MutexWriter(file.clone())) as Box<dyn Write + Send>
|
||||
});
|
||||
|
||||
_ = tracing_subscriber::fmt()
|
||||
.with_max_level(tracing::Level::INFO)
|
||||
.with_span_events(tracing_subscriber::fmt::format::FmtSpan::ACTIVE)
|
||||
.with_writer(writer)
|
||||
.with_ansi(false)
|
||||
.compact()
|
||||
.try_init();
|
||||
}
|
||||
|
||||
struct MutexWriter(Arc<Mutex<std::fs::File>>);
|
||||
|
||||
impl Write for MutexWriter {
|
||||
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
|
||||
self.0.lock().unwrap().write(buf)
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> io::Result<()> {
|
||||
self.0.lock().unwrap().flush()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum ChangedContent {
|
||||
|
|
@ -129,26 +64,32 @@ pub struct Scanner {
|
|||
/// The walker to detect all files that we have to scan
|
||||
walker: Option<WalkBuilder>,
|
||||
|
||||
/// All changed content that we have to parse
|
||||
changed_content: Vec<ChangedContent>,
|
||||
|
||||
/// All found extensions
|
||||
extensions: FxHashSet<String>,
|
||||
|
||||
/// All CSS files we want to scan for CSS variable usage
|
||||
css_files: Vec<PathBuf>,
|
||||
|
||||
/// All files that we have to scan
|
||||
files: Vec<PathBuf>,
|
||||
files: FxHashSet<PathBuf>,
|
||||
|
||||
/// All directories, sub-directories, etc… we saw during source detection
|
||||
dirs: Vec<PathBuf>,
|
||||
dirs: FxHashSet<PathBuf>,
|
||||
|
||||
/// All generated globs, used for setting up watchers
|
||||
globs: Option<Vec<GlobEntry>>,
|
||||
|
||||
/// Track unique set of candidates
|
||||
candidates: FxHashSet<String>,
|
||||
|
||||
/// Track mtimes for files so re-scans can skip unchanged files.
|
||||
/// Only populated after the first scan completes (to avoid unnecessary
|
||||
/// metadata calls on initial build).
|
||||
mtimes: FxHashMap<PathBuf, SystemTime>,
|
||||
|
||||
/// Whether we've completed at least one full scan. When false, we skip
|
||||
/// mtime tracking entirely so the initial build stays fast.
|
||||
has_scanned_once: bool,
|
||||
|
||||
/// Whether sources have been scanned since the last `scan()` call
|
||||
sources_scanned: bool,
|
||||
}
|
||||
|
||||
impl Scanner {
|
||||
|
|
@ -163,7 +104,6 @@ impl Scanner {
|
|||
}
|
||||
|
||||
let sources = Sources::new(public_source_entries_to_private_source_entries(sources));
|
||||
|
||||
if *SHOULD_TRACE {
|
||||
event!(tracing::Level::INFO, "Optimized sources:");
|
||||
for source in sources.iter() {
|
||||
|
|
@ -171,29 +111,26 @@ impl Scanner {
|
|||
}
|
||||
}
|
||||
|
||||
let walker = create_walker(&sources);
|
||||
|
||||
Self {
|
||||
sources: sources.clone(),
|
||||
walker: create_walker(sources),
|
||||
sources,
|
||||
walker,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
pub fn scan(&mut self) -> Vec<String> {
|
||||
self.scan_sources();
|
||||
self.sources_scanned = false;
|
||||
|
||||
// TODO: performance improvement, bail early if we don't have any changed content
|
||||
// if self.changed_content.is_empty() {
|
||||
// return vec![];
|
||||
// }
|
||||
let (scanned_blobs, css_files) = self.discover_sources();
|
||||
|
||||
let _new_candidates = self.extract_candidates();
|
||||
self.extract_candidates(scanned_blobs, css_files);
|
||||
|
||||
// Make sure we have a sorted list of candidates
|
||||
let mut candidates = self.candidates.iter().cloned().collect::<Vec<_>>();
|
||||
candidates.par_sort_unstable();
|
||||
|
||||
// Return all candidates instead of only the new ones
|
||||
candidates
|
||||
// Return all candidates sorted
|
||||
let mut result = self.candidates.iter().cloned().collect::<Vec<_>>();
|
||||
result.par_sort_unstable();
|
||||
result
|
||||
}
|
||||
|
||||
#[tracing::instrument(skip_all)]
|
||||
|
|
@ -208,7 +145,7 @@ impl Scanner {
|
|||
|
||||
// Raw content can be parsed directly, no need to verify if the file exists and is allowed
|
||||
// to be scanned.
|
||||
self.changed_content.extend(changed_contents);
|
||||
let mut content_to_scan: Vec<ChangedContent> = changed_contents;
|
||||
|
||||
// Fully resolve all files
|
||||
let changed_files = changed_files
|
||||
|
|
@ -232,7 +169,7 @@ impl Scanner {
|
|||
});
|
||||
|
||||
// All known files are allowed to be scanned
|
||||
self.changed_content.extend(known_files);
|
||||
content_to_scan.extend(known_files);
|
||||
|
||||
// Figure out if the new unknown files are allowed to be scanned
|
||||
if !new_unknown_files.is_empty() {
|
||||
|
|
@ -252,8 +189,8 @@ impl Scanner {
|
|||
// When the file is found on disk it means that all the rules pass. We can
|
||||
// extract the current file and remove it from the list of passed in files.
|
||||
if file == path {
|
||||
self.files.push(path.to_path_buf()); // Track for future use
|
||||
self.changed_content.push(changed_file.clone()); // Track for parsing
|
||||
self.files.insert(path.to_path_buf()); // Track for future use
|
||||
content_to_scan.push(changed_file.clone()); // Track for parsing
|
||||
drop_file_indexes.push(idx);
|
||||
}
|
||||
}
|
||||
|
|
@ -274,18 +211,17 @@ impl Scanner {
|
|||
}
|
||||
}
|
||||
|
||||
self.extract_candidates()
|
||||
// Read all content into blobs for extraction
|
||||
let blobs = read_all_files(content_to_scan);
|
||||
self.extract_candidates(blobs, vec![])
|
||||
}
|
||||
|
||||
#[tracing::instrument(skip_all)]
|
||||
fn extract_candidates(&mut self) -> Vec<String> {
|
||||
let changed_content = self.changed_content.drain(..).collect::<Vec<_>>();
|
||||
|
||||
// Extract all candidates from the changed content
|
||||
let mut new_candidates = parse_all_blobs(read_all_files(changed_content));
|
||||
fn extract_candidates(&mut self, blobs: Vec<Vec<u8>>, css_files: Vec<PathBuf>) -> Vec<String> {
|
||||
// Extract all candidates from the pre-read blobs
|
||||
let mut new_candidates = parse_all_blobs(blobs);
|
||||
|
||||
// Extract all CSS variables from the CSS files
|
||||
let css_files = self.css_files.drain(..).collect::<Vec<_>>();
|
||||
if !css_files.is_empty() {
|
||||
let css_variables = extract_css_variables(read_all_files(
|
||||
css_files
|
||||
|
|
@ -297,63 +233,23 @@ impl Scanner {
|
|||
new_candidates.extend(css_variables);
|
||||
}
|
||||
|
||||
// Only compute the new candidates and ignore the ones we already have. This is for
|
||||
// subsequent calls to prevent serializing the entire set of candidates every time.
|
||||
let mut new_candidates = new_candidates
|
||||
.into_par_iter()
|
||||
.filter(|candidate| !self.candidates.contains(candidate))
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
new_candidates.par_sort_unstable();
|
||||
// Only keep candidates we haven't seen before
|
||||
for existing in self.candidates.iter() {
|
||||
new_candidates.remove(existing);
|
||||
}
|
||||
|
||||
// Track new candidates for subsequent calls
|
||||
self.candidates.par_extend(new_candidates.clone());
|
||||
self.candidates.extend(new_candidates.iter().cloned());
|
||||
|
||||
new_candidates
|
||||
}
|
||||
let mut result: Vec<String> = new_candidates.into_iter().collect();
|
||||
result.par_sort_unstable();
|
||||
|
||||
#[tracing::instrument(skip_all)]
|
||||
fn scan_sources(&mut self) {
|
||||
let Some(walker) = &mut self.walker else {
|
||||
return;
|
||||
};
|
||||
|
||||
for entry in walker.build().filter_map(Result::ok) {
|
||||
let path = entry.into_path();
|
||||
let Ok(metadata) = path.metadata() else {
|
||||
continue;
|
||||
};
|
||||
if metadata.is_dir() {
|
||||
self.dirs.push(path);
|
||||
} else if metadata.is_file() {
|
||||
let extension = path
|
||||
.extension()
|
||||
.and_then(|x| x.to_str())
|
||||
.unwrap_or_default(); // In case the file has no extension
|
||||
|
||||
match extension {
|
||||
// Special handing for CSS files, we don't want to extract candidates from
|
||||
// these files, but we do want to extract used CSS variables.
|
||||
"css" => {
|
||||
self.css_files.push(path.clone());
|
||||
}
|
||||
_ => {
|
||||
self.changed_content.push(ChangedContent::File(
|
||||
path.to_path_buf(),
|
||||
extension.to_owned(),
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
self.extensions.insert(extension.to_owned());
|
||||
self.files.push(path);
|
||||
}
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
#[tracing::instrument(skip_all)]
|
||||
pub fn get_files(&mut self) -> Vec<String> {
|
||||
self.scan_sources();
|
||||
let _ = self.discover_sources();
|
||||
|
||||
self.files
|
||||
.par_iter()
|
||||
|
|
@ -367,7 +263,7 @@ impl Scanner {
|
|||
return globs.clone();
|
||||
}
|
||||
|
||||
self.scan_sources();
|
||||
let _ = self.discover_sources();
|
||||
|
||||
let mut globs = vec![];
|
||||
for source in self.sources.iter() {
|
||||
|
|
@ -456,6 +352,99 @@ impl Scanner {
|
|||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[tracing::instrument(skip_all)]
|
||||
fn discover_sources(&mut self) -> (Vec<Vec<u8>>, Vec<PathBuf>) {
|
||||
if self.sources_scanned {
|
||||
return (vec![], vec![]);
|
||||
}
|
||||
self.sources_scanned = true;
|
||||
|
||||
let Some(walker) = &mut self.walker else {
|
||||
return (vec![], vec![]);
|
||||
};
|
||||
|
||||
// Use synchronous walk for the initial build (lower overhead) and parallel
|
||||
// walk for subsequent calls (watch mode) where the overhead is amortised.
|
||||
let all_entries = if self.has_scanned_once {
|
||||
walk_parallel(walker)
|
||||
} else {
|
||||
walk_synchronous(walker)
|
||||
};
|
||||
|
||||
let mut css_files: Vec<PathBuf> = vec![];
|
||||
let mut content_paths: Vec<(PathBuf, String)> = Vec::new();
|
||||
let mut seen_files: FxHashSet<PathBuf> = FxHashSet::default();
|
||||
|
||||
for (path, is_dir, extension) in all_entries {
|
||||
if is_dir {
|
||||
self.dirs.insert(path);
|
||||
} else {
|
||||
// Deduplicate: parallel walk can visit the same file from multiple threads
|
||||
if !seen_files.insert(path.clone()) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// On re-scans, check mtime to skip unchanged files.
|
||||
// On the first scan we skip this entirely to avoid extra
|
||||
// metadata syscalls.
|
||||
let changed = if self.has_scanned_once {
|
||||
let current_mtime = std::fs::metadata(&path)
|
||||
.ok()
|
||||
.and_then(|m| m.modified().ok());
|
||||
|
||||
match current_mtime {
|
||||
Some(mtime) => {
|
||||
let prev = self.mtimes.insert(path.clone(), mtime);
|
||||
prev.is_none_or(|prev| prev != mtime)
|
||||
}
|
||||
None => true,
|
||||
}
|
||||
} else {
|
||||
true
|
||||
};
|
||||
|
||||
match extension.as_str() {
|
||||
// Special handing for CSS files, we don't want to extract candidates from
|
||||
// these files, but we do want to extract used CSS variables.
|
||||
"css" => {
|
||||
if changed {
|
||||
css_files.push(path.clone());
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
if changed {
|
||||
content_paths.push((path.clone(), extension.clone()));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
self.extensions.insert(extension);
|
||||
self.files.insert(path);
|
||||
}
|
||||
}
|
||||
|
||||
// Read + preprocess all discovered files in parallel
|
||||
let scanned_blobs: Vec<Vec<u8>> = content_paths
|
||||
.into_par_iter()
|
||||
.filter_map(|(path, ext)| {
|
||||
let content = std::fs::read(&path).ok()?;
|
||||
event!(tracing::Level::INFO, "Reading {:?}", path);
|
||||
let processed = pre_process_input(content, &ext);
|
||||
if processed.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(processed)
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
|
||||
if !self.has_scanned_once {
|
||||
self.has_scanned_once = true;
|
||||
}
|
||||
|
||||
(scanned_blobs, css_files)
|
||||
}
|
||||
}
|
||||
|
||||
fn read_changed_content(c: ChangedContent) -> Option<Vec<u8>> {
|
||||
|
|
@ -474,26 +463,26 @@ fn read_changed_content(c: ChangedContent) -> Option<Vec<u8>> {
|
|||
ChangedContent::Content(contents, extension) => (contents.into_bytes(), extension),
|
||||
};
|
||||
|
||||
Some(pre_process_input(&content, &extension))
|
||||
Some(pre_process_input(content, &extension))
|
||||
}
|
||||
|
||||
pub fn pre_process_input(content: &[u8], extension: &str) -> Vec<u8> {
|
||||
pub fn pre_process_input(content: Vec<u8>, extension: &str) -> Vec<u8> {
|
||||
use crate::extractor::pre_processors::*;
|
||||
|
||||
match extension {
|
||||
"clj" | "cljs" | "cljc" => Clojure.process(content),
|
||||
"heex" | "eex" | "ex" | "exs" => Elixir.process(content),
|
||||
"cshtml" | "razor" => Razor.process(content),
|
||||
"haml" => Haml.process(content),
|
||||
"json" => Json.process(content),
|
||||
"md" | "mdx" => Markdown.process(content),
|
||||
"pug" => Pug.process(content),
|
||||
"rb" | "erb" => Ruby.process(content),
|
||||
"slim" | "slang" => Slim.process(content),
|
||||
"svelte" => Svelte.process(content),
|
||||
"rs" => Rust.process(content),
|
||||
"vue" => Vue.process(content),
|
||||
_ => content.to_vec(),
|
||||
"clj" | "cljs" | "cljc" => Clojure.process(&content),
|
||||
"heex" | "eex" | "ex" | "exs" => Elixir.process(&content),
|
||||
"cshtml" | "razor" => Razor.process(&content),
|
||||
"haml" => Haml.process(&content),
|
||||
"json" => Json.process(&content),
|
||||
"md" | "mdx" => Markdown.process(&content),
|
||||
"pug" => Pug.process(&content),
|
||||
"rb" | "erb" => Ruby.process(&content),
|
||||
"slim" | "slang" => Slim.process(&content),
|
||||
"svelte" => Svelte.process(&content),
|
||||
"rs" => Rust.process(&content),
|
||||
"vue" => Vue.process(&content),
|
||||
_ => content,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -512,23 +501,23 @@ fn read_all_files(changed_content: Vec<ChangedContent>) -> Vec<Vec<u8>> {
|
|||
}
|
||||
|
||||
#[tracing::instrument(skip_all)]
|
||||
fn extract_css_variables(blobs: Vec<Vec<u8>>) -> Vec<String> {
|
||||
fn extract_css_variables(blobs: Vec<Vec<u8>>) -> FxHashSet<String> {
|
||||
extract(blobs, |mut extractor| {
|
||||
extractor.extract_variables_from_css()
|
||||
})
|
||||
}
|
||||
|
||||
#[tracing::instrument(skip_all)]
|
||||
fn parse_all_blobs(blobs: Vec<Vec<u8>>) -> Vec<String> {
|
||||
fn parse_all_blobs(blobs: Vec<Vec<u8>>) -> FxHashSet<String> {
|
||||
extract(blobs, |mut extractor| extractor.extract())
|
||||
}
|
||||
|
||||
#[tracing::instrument(skip_all)]
|
||||
fn extract<H>(blobs: Vec<Vec<u8>>, handle: H) -> Vec<String>
|
||||
fn extract<H>(blobs: Vec<Vec<u8>>, handle: H) -> FxHashSet<String>
|
||||
where
|
||||
H: Fn(Extractor) -> Vec<Extracted> + std::marker::Sync,
|
||||
{
|
||||
let mut result: Vec<_> = blobs
|
||||
blobs
|
||||
.par_iter()
|
||||
.flat_map(|blob| blob.par_split(|x| *x == b'\n'))
|
||||
.filter_map(|blob| {
|
||||
|
|
@ -554,22 +543,97 @@ where
|
|||
})
|
||||
.into_iter()
|
||||
.map(|s| unsafe { String::from_utf8_unchecked(s.to_vec()) })
|
||||
.collect();
|
||||
|
||||
// SAFETY: Unstable sort is faster and in this scenario it's also safe because we are
|
||||
// guaranteed to have unique candidates.
|
||||
result.par_sort_unstable();
|
||||
|
||||
result
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Create a walker for the given sources to detect all the files that we have to scan.
|
||||
type WalkEntry = (PathBuf, bool, String);
|
||||
|
||||
/// Walk the file system synchronously. Used for the initial build where the overhead of spawning
|
||||
/// parallel walker threads is not worth it.
|
||||
#[tracing::instrument(skip_all)]
|
||||
fn walk_synchronous(walker: &mut WalkBuilder) -> Vec<WalkEntry> {
|
||||
let mut entries = Vec::new();
|
||||
|
||||
for entry in walker.build().filter_map(Result::ok) {
|
||||
let is_dir = entry.file_type().map(|ft| ft.is_dir()).unwrap_or(false);
|
||||
let path = entry.into_path();
|
||||
|
||||
if is_dir {
|
||||
entries.push((path, true, String::new()));
|
||||
} else {
|
||||
let ext = path
|
||||
.extension()
|
||||
.and_then(|x| x.to_str())
|
||||
.unwrap_or_default()
|
||||
.to_owned();
|
||||
entries.push((path, false, ext));
|
||||
}
|
||||
}
|
||||
|
||||
entries
|
||||
}
|
||||
|
||||
/// Walk the file system in parallel. Used in watch mode where the parallel walker overhead is
|
||||
/// amortised across many rebuilds and subsequent calls are much faster.
|
||||
#[tracing::instrument(skip_all)]
|
||||
fn walk_parallel(walker: &mut WalkBuilder) -> Vec<WalkEntry> {
|
||||
struct FlushOnDrop {
|
||||
local: Vec<WalkEntry>,
|
||||
shared: Arc<Mutex<Vec<WalkEntry>>>,
|
||||
}
|
||||
|
||||
impl Drop for FlushOnDrop {
|
||||
fn drop(&mut self) {
|
||||
if !self.local.is_empty() {
|
||||
self.shared.lock().unwrap().append(&mut self.local);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let collected: Arc<Mutex<Vec<WalkEntry>>> = Arc::new(Mutex::new(Vec::new()));
|
||||
|
||||
walker.build_parallel().run(|| {
|
||||
let mut buf = FlushOnDrop {
|
||||
local: Vec::with_capacity(256),
|
||||
shared: collected.clone(),
|
||||
};
|
||||
|
||||
Box::new(move |entry| {
|
||||
let Ok(entry) = entry else {
|
||||
return ignore::WalkState::Continue;
|
||||
};
|
||||
|
||||
let is_dir = entry.file_type().map(|ft| ft.is_dir()).unwrap_or(false);
|
||||
let path = entry.into_path();
|
||||
|
||||
if is_dir {
|
||||
buf.local.push((path, true, String::new()));
|
||||
} else {
|
||||
let ext = path
|
||||
.extension()
|
||||
.and_then(|x| x.to_str())
|
||||
.unwrap_or_default()
|
||||
.to_owned();
|
||||
buf.local.push((path, false, ext));
|
||||
}
|
||||
|
||||
if buf.local.len() >= 256 {
|
||||
buf.shared.lock().unwrap().append(&mut buf.local);
|
||||
}
|
||||
|
||||
ignore::WalkState::Continue
|
||||
})
|
||||
});
|
||||
|
||||
// All threads have finished and flushed their buffers via FlushOnDrop::drop
|
||||
Arc::try_unwrap(collected).unwrap().into_inner().unwrap()
|
||||
}
|
||||
|
||||
/// Sets up a WalkBuilder with all source roots, gitignore rules, and source pattern matching.
|
||||
///
|
||||
/// The `mtimes` map is used to keep track of the last modified time of each file. This is used to
|
||||
/// determine if a file or folder has changed since the last scan and we can skip folders that
|
||||
/// haven't changed.
|
||||
fn create_walker(sources: Sources) -> Option<WalkBuilder> {
|
||||
let mtimes: Arc<Mutex<FxHashMap<PathBuf, SystemTime>>> = Default::default();
|
||||
/// This is the common setup shared between the full walker (with mtime tracking for re-scans)
|
||||
/// and the parallel walker (without mtime tracking for the initial scan).
|
||||
fn create_walker(sources: &Sources) -> Option<WalkBuilder> {
|
||||
let mut other_roots: FxHashSet<&PathBuf> = FxHashSet::default();
|
||||
let mut first_root: Option<&PathBuf> = None;
|
||||
let mut ignores: BTreeMap<&PathBuf, BTreeSet<String>> = Default::default();
|
||||
|
|
@ -717,75 +781,68 @@ fn create_walker(sources: Sources) -> Option<WalkBuilder> {
|
|||
builder.add_gitignore(ignore);
|
||||
}
|
||||
|
||||
builder.filter_entry({
|
||||
move |entry| {
|
||||
let path = entry.path();
|
||||
// Pre-compute source matching data to avoid allocations in the hot filter_entry path
|
||||
let auto_bases: Vec<PathBuf> = sources
|
||||
.iter()
|
||||
.filter_map(|source| match source {
|
||||
SourceEntry::Auto { base } | SourceEntry::External { base } => Some(base.clone()),
|
||||
_ => None,
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Ensure the entries are matching any of the provided source patterns (this is
|
||||
// necessary for manual-patterns that can filter the file extension)
|
||||
if path.is_file() {
|
||||
let mut matches = false;
|
||||
for source in sources.iter() {
|
||||
match source {
|
||||
SourceEntry::Auto { base } | SourceEntry::External { base } => {
|
||||
if path.starts_with(base) {
|
||||
matches = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
SourceEntry::Pattern { base, pattern } => {
|
||||
let mut pattern = pattern.to_string();
|
||||
// Ensure that the pattern is pinned to the base path.
|
||||
if !pattern.starts_with("/") {
|
||||
pattern = format!("/{pattern}");
|
||||
}
|
||||
let pattern_sources: Vec<(PathBuf, String)> = sources
|
||||
.iter()
|
||||
.filter_map(|source| match source {
|
||||
SourceEntry::Pattern { base, pattern } => {
|
||||
let normalized = if pattern.starts_with("/") {
|
||||
pattern.to_string()
|
||||
} else {
|
||||
format!("/{pattern}")
|
||||
};
|
||||
Some((base.clone(), normalized))
|
||||
}
|
||||
_ => None,
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Check if path starts with base, if so, remove the prefix and check the remainder against the pattern
|
||||
let remainder = path.strip_prefix(base);
|
||||
if remainder.is_ok_and(|remainder| {
|
||||
let mut path_str = remainder.to_string_lossy().to_string();
|
||||
if !path_str.starts_with("/") {
|
||||
path_str = format!("/{path_str}");
|
||||
}
|
||||
glob_match(pattern, path_str.as_bytes())
|
||||
}) {
|
||||
matches = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
// Source pattern matching filter (lock-free, safe for parallel walking)
|
||||
builder.filter_entry(move |entry| {
|
||||
let path = entry.path();
|
||||
|
||||
if !matches {
|
||||
return false;
|
||||
// Ensure the entries are matching any of the provided source patterns (this is
|
||||
// necessary for manual-patterns that can filter the file extension)
|
||||
if path.is_file() {
|
||||
let mut matches = false;
|
||||
|
||||
for base in &auto_bases {
|
||||
if path.starts_with(base) {
|
||||
matches = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
let mut mtimes = mtimes.lock().unwrap();
|
||||
let current_time = match entry.metadata() {
|
||||
Ok(metadata) if metadata.is_file() => {
|
||||
if let Ok(time) = metadata.modified() {
|
||||
Some(time)
|
||||
} else {
|
||||
None
|
||||
if !matches {
|
||||
for (base, pattern) in &pattern_sources {
|
||||
let remainder = path.strip_prefix(base);
|
||||
if remainder.is_ok_and(|remainder| {
|
||||
let mut path_str = remainder.to_string_lossy().to_string();
|
||||
if !path_str.starts_with("/") {
|
||||
path_str = format!("/{path_str}");
|
||||
}
|
||||
glob_match(pattern, path_str.as_bytes())
|
||||
}) {
|
||||
matches = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
_ => None,
|
||||
};
|
||||
}
|
||||
|
||||
let previous_time =
|
||||
current_time.and_then(|time| mtimes.insert(entry.clone().into_path(), time));
|
||||
|
||||
match (current_time, previous_time) {
|
||||
(Some(current), Some(prev)) if prev == current => false,
|
||||
_ => {
|
||||
event!(tracing::Level::INFO, "Discovering {:?}", path);
|
||||
|
||||
true
|
||||
}
|
||||
if !matches {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
true
|
||||
});
|
||||
|
||||
Some(builder)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue