From 095ff96ba35de0824313fd150b6e70695d300dd1 Mon Sep 17 00:00:00 2001 From: Robin Malfait Date: Tue, 17 Feb 2026 14:15:20 -0500 Subject: [PATCH] Improve performance in bigger projects (#19632) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR improves the performance of Oxide when scanning large codebases. The `Oxide` API, looks something like this: ```ts let scanner = new Scanner({ sources }) let candidates = scanner.scan() // Found candidates let files = scanner.files // Scanned files let globs = scanner.globs // Scanned globs ``` The `files` and `globs` are used to tell PostCSS, Vite, webpack etc which files to watch for changes. The `.scan()` operation extracts the candidates from the source files. You can think of these as potential Tailwind CSS classes. In all these scenarios we have to walk the file system and find files that match the `sources`. ### 1. Prevent multiple file system walks The first big win came from the fact that accessing `.files` after a `.scan()` also does an entire walk of the file system (for the given `sources`), which is unnecessary because we just walked the file system. This is something that's not really an issue in smaller codebases because we have `mtime` tracking. We don't re-scan a file if its `mtime` hasn't changed since the last scan. However, in large codebases with thousands of files, even walking the file system to check `mtime`s can be expensive. ### 2. Use parallel file system walking Another big win is to use a parallel file system walker instead of a synchronous one. The big problem here is that the parallel build has 20ms-50ms of overhead which is noticeable on small codebases. We don't really know if you have a small or big codebase ahead of time, so maybe some kind of hint in the future would be useful. So the solution I settled on right now is to use a synchronous walker for the initial scan, and then switch to a parallel walker for subsequent scans (think dev mode). This gives us the best of both worlds: fast initial scan on small codebases, and fast re-scans on large codebases. Caveat: if you use the `@tailwindcss/cli` we know exactly which files changed so we can just re-scan those files directly without walking the file system at all. But in `@tailwindcss/postcss` we don't know which files changed, so we have to walk the file system to check `mtime`s. While this improvement is nice, it resulted in an annoying issue related to `mtime` tracking. Since the parallel walker processes files in parallel, the `mtime` was typed as `Arc>>` so to avoid locking, I decided to only walk the files here and collect their paths. Then later we check the `mtime` to know whether to re-scan them or not. Initially I just removed the `mtime` tracking altogether. But it did have an impact when actually extracting candidates from those files, so I added it back later. ### 3. Delaying work I was still a bit annoyed by the fact that we had to track `mtime` values for every file. This seems like annoying overhead, especially when doing a single build (no dev mode). So the trick I applied here is to only start tracking `mtime` values after the initial scan. This means that, in dev mode, we would do this: 1. Walk entire file system to track files. 2. On a subsequent scan, walk entire file system (again) and start tracking `mtime` values. This time, we use the parallel walker instead of the synchronous one. 3. On further scans, only re-scan files whose `mtime` has changed The trade-off here is that on the second scan we always re-scan all files, even if they haven't changed. Since this typically only happens in dev mode, I think this is an acceptable trade-off especially if the initial build is therefor faster this way. ### 3. Small wins There are also a few small wins in here that I would like to mention but that are less significant: 1. Pre-computed normalized `source` patterns instead of in every walker filter call. 2. Tried to avoid some allocations in various places. For example the `pre_process_input` always called `content.to_vec()` which allocates. Instead we now accept an owned `Vec` so we don't have to call `.to_vec()` in the default case (in my testing, this is ~92% of the time in the codebases I checked). 3. Made the `Cursor` struct smaller, which is used a lot during candidate extraction. ### Benchmarks Now for the fun stuff, the benchmarks!
The code for the benchmarks ```ts import path from 'node:path' import { bench, boxplot, do_not_optimize, run, summary } from 'mitata' import { Scanner as ScannerPr } from '/path/to/repo/with/pr/branch/tailwindcss/crates/node' import { Scanner as ScannerMain } from '/path/to/repo/with/main/branch/tailwindcss/crates/node' let base = '/path/to/some/codebase' let sources = [{ base, pattern: '**/*', negated: false }] // Verify the results are the same before benchmarking let scannerPr = new ScannerPr({ sources }) let scannerMain = new ScannerMain({ sources }) { let aCandidates = scannerPr.scan() let bCandidates = scannerMain.scan() if (aCandidates.length !== bCandidates.length) { throw new Error(`Mismatch in candidate count: ${aCandidates.length} vs ${bCandidates.length}`) } for (let i = 0; i < aCandidates.length; i++) { if (aCandidates[i] !== bCandidates[i]) { throw new Error(`Mismatch in candidate at index ${i}: ${aCandidates[i]} vs ${bCandidates[i]}`) } } let aFiles = scannerPr.files let bFiles = scannerMain.files if (aFiles.length !== bFiles.length) { throw new Error(`Mismatch in file count: ${aFiles.length} vs ${bFiles.length}`) } for (let i = 0; i < aFiles.length; i++) { if (aFiles[i] !== bFiles[i]) { throw new Error(`Mismatch in file at index ${i}: ${aFiles[i]} vs ${bFiles[i]}`) } } console.log('Scanned', aFiles.length, 'files') console.log('Extracted', aCandidates.length, 'candidates') console.log('Base =', base) console.log() } summary(() => { boxplot(() => { bench('PR (build, .scan()))', function* () { yield { [0]() { return new ScannerPr({ sources }) }, bench(scanner: ScannerPr) { do_not_optimize(scanner.scan()) }, } }) bench('main (build, .scan()))', function* () { yield { [0]() { return new ScannerMain({ sources }) }, bench(scanner: ScannerMain) { do_not_optimize(scanner.scan()) }, } }) }) }) summary(() => { boxplot(() => { bench('PR (build, .scan() + .files)', function* () { yield { [0]() { return new ScannerPr({ sources }) }, bench(scanner: ScannerPr) { do_not_optimize(scanner.scan()) do_not_optimize(scanner.files) }, } }) bench('main (build, .scan() + .files)', function* () { yield { [0]() { return new ScannerMain({ sources }) }, bench(scanner: ScannerMain) { do_not_optimize(scanner.scan()) do_not_optimize(scanner.files) }, } }) }) }) summary(() => { boxplot(() => { bench('PR (watch, .scan()))', function* () { yield { bench() { do_not_optimize(scannerPr.scan()) }, } }) bench('main (watch, .scan()))', function* () { yield { bench() { do_not_optimize(scannerMain.scan()) }, } }) }) }) summary(() => { boxplot(() => { bench('PR (watch, .scan() + .files)', function* () { yield { bench() { do_not_optimize(scannerPr.scan()) do_not_optimize(scannerPr.files) }, } }) bench('main (watch, .scan() + .files)', function* () { yield { bench() { do_not_optimize(scannerMain.scan()) do_not_optimize(scannerMain.files) }, } }) }) }) await run() ```
#### tailwindcss.com codebase ``` Scanned 462 files Extracted 13200 candidates Base = /Users/robin/github.com/tailwindlabs/tailwindcss.com clk: ~3.09 GHz cpu: Apple M1 Max runtime: bun 1.3.3 (arm64-darwin) ``` In these benchmarks the `PR` one is consistently faster than `main`. It's not by a lot but that's mainly because the codebase itself isn't that big. It is a codebase with _a lot_ of candidates though, but not that many files. The candidate extraction was already pretty fast, so the wins here mainly come from avoiding re-walking the file system when accessing `.files`, and from delaying `mtime` tracking until after the initial scan. **Single initial build**: It's not a lot, but it's a bit faster. This is due to avoiding tracking the `mtime` values initially and making some small optimizations related to the struct size and allocations. ``` benchmark avg (min … max) p75 / p99 (min … top 1%) --------------------------------------------- ------------------------------- PR (build, .scan())) 22.87 ms/iter 23.28 ms █ (21.49 ms … 25.68 ms) 23.98 ms ▂ ▂ ▂ █ ▂▂ (832.00 kb … 2.69 mb) 1.41 mb ▆▆▆▆█▆▆█▁▆▆█▁▁█▁▆██▁▆ main (build, .scan())) 25.67 ms/iter 26.12 ms █ █ █ (24.54 ms … 27.74 ms) 27.06 ms █ █ █ ███ (432.00 kb … 2.78 mb) 996.00 kb ██▁████▁█▁████▁█▁▁█▁█ ┌ ┐ ╷ ┌─────┬──┐ ╷ PR (build, .scan())) ├────┤ │ ├─────┤ ╵ └─────┴──┘ ╵ ╷ ┌─────┬──┐ ╷ main (build, .scan())) ├──┤ │ ├───────┤ ╵ └─────┴──┘ ╵ └ ┘ 21.49 ms 24.28 ms 27.06 ms summary PR (build, .scan())) 1.12x faster than main (build, .scan())) ``` **Single initial build + accessing `.files`**: We don't have to re-walk the entire file system even if we're just dealing with ~462 scanned files. ``` benchmark avg (min … max) p75 / p99 (min … top 1%) --------------------------------------------- ------------------------------- PR (build, .scan() + .files) 22.54 ms/iter 22.99 ms █ ▂ (21.41 ms … 25.86 ms) 24.26 ms █ ▅ ▅▅█▅ ▅▅ (368.00 kb … 2.05 mb) 853.00 kb █▇█▇▇████▇▇██▁▁▇▁▁▇▁▇ main (build, .scan() + .files) 32.15 ms/iter 32.17 ms █ ▂ (30.78 ms … 36.22 ms) 35.75 ms ▅█ ▅█ ▅ (400.00 kb … 2.45 mb) 952.00 kb ██▁██▇▇█▇▁▁▁▁▁▁▁▁▁▁▁▇ ┌ ┐ ╷┌──┬┐ ╷ PR (build, .scan() + .files) ├┤ │├───┤ ╵└──┴┘ ╵ ╷┌───┬ ╷ main (build, .scan() + .files) ├┤ │──────────┤ ╵└───┴ ╵ └ ┘ 21.41 ms 28.58 ms 35.75 ms summary PR (build, .scan() + .files) 1.43x faster than main (build, .scan() + .files) ``` **Watch/dev mode, only scanning**: This now switches to the parallel walker, but since it's not a super big codebase we don't see a huge win here yet. ``` benchmark avg (min … max) p75 / p99 (min … top 1%) --------------------------------------------- ------------------------------- PR (watch, .scan())) 6.85 ms/iter 7.22 ms █▄ (6.34 ms … 7.94 ms) 7.91 ms ▄██▃ ( 64.00 kb … 688.00 kb) 452.82 kb ▃████▆▂▂▁▂▁▂▁▁▁▅█▆▃▅▃ main (watch, .scan())) 7.92 ms/iter 8.08 ms █ █ ▃ █▃▃ (7.41 ms … 8.71 ms) 8.68 ms █▆█▆▃█████ ( 0.00 b … 64.00 kb) 19.20 kb ▆▄██████████▆▁▆▆█▄▄▄▆ ┌ ┐ ╷ ┌──────┬──────┐ ╷ PR (watch, .scan())) ├──┤ │ ├────────────┤ ╵ └──────┴──────┘ ╵ ╷ ┌───┬──┐ ╷ main (watch, .scan())) ├────┤ │ ├───────────┤ ╵ └───┴──┘ ╵ └ ┘ 6.34 ms 7.51 ms 8.68 ms summary PR (watch, .scan())) 1.16x faster than main (watch, .scan())) ``` **Watch/dev mode, scanning + accessing `.files`**: Again we avoid re-walking the entire file system when accessing `.files`. ``` benchmark avg (min … max) p75 / p99 (min … top 1%) --------------------------------------------- ------------------------------- PR (watch, .scan() + .files) 12.10 ms/iter 12.74 ms █ █ █ █ ▃▃▃ (10.69 ms … 13.89 ms) 13.81 ms █ █▂▂▂ ▇█▂█▂███▇ (128.00 kb … 10.73 mb) 5.23 mb █▆████▁█████████▆▆▆▆▆ main (watch, .scan() + .files) 14.44 ms/iter 14.74 ms █ (13.93 ms … 15.33 ms) 15.18 ms ███▅ █ ▅ ▅ ( 16.00 kb … 80.00 kb) 39.51 kb █▅████▁███▅▁█████▅▁▅▅ ┌ ┐ ╷ ┌──────┬──────┐ ╷ PR (watch, .scan() + .files) ├──────┤ │ ├─────────┤ ╵ └──────┴──────┘ ╵ ╷ ┌───┬──┐ ╷ main (watch, .scan() + .files) ├─┤ │ ├───┤ ╵ └───┴──┘ ╵ └ ┘ 10.69 ms 12.93 ms 15.18 ms summary PR (watch, .scan() + .files) 1.19x faster than main (watch, .scan() + .files) ``` #### Synthetic 5000 files codebase Based on the instructions from #19616 I created a codebase with 5000 files. Each file contains a `flex` class and a unique class like `content-['/path/to/file']` to ensure we have a decent amount of unique candidates. You can test the script yourself by running this: ``` mkdir -p fixtures/app-5000/src/components/{auth,dashboard,settings,profile,notifications,messages,search,navigation,footer,sidebar}/sub{001..500} && for dir in fixtures/app-5000/src/components/*/sub*; do echo "export const Component = () =>
test
" > "$dir/index.tsx"; done && find fixtures/app-5000/src/components -type f | wc -lc ``` ``` Scanned 5000 files Extracted 5005 candidates Base = /Users/robin/github.com/RobinMalfait/playground/scanner-benchmarks/fixtures/app-5000 clk: ~3.08 GHz cpu: Apple M1 Max runtime: bun 1.3.3 (arm64-darwin) ``` **Single initial build**: As expected not a super big win here because it's a single build. But there is a noticeable improvement. ``` benchmark avg (min … max) p75 / p99 (min … top 1%) --------------------------------------------- ------------------------------- PR (build, .scan())) 217.27 ms/iter 211.97 ms █ (205.99 ms … 289.53 ms) 214.33 ms ▅ ▅ ▅▅▅▅█▅▅ ▅ ( 3.34 mb … 4.25 mb) 3.72 mb █▁▁▁▁▁▁█▁███████▁▁▁▁█ main (build, .scan())) 249.26 ms/iter 239.88 ms █ (231.51 ms … 381.66 ms) 241.01 ms ▅ ▅ ▅▅ ▅ █▅ ▅▅▅ ( 4.22 mb … 4.78 mb) 4.49 mb █▁▁▁▁█▁██▁▁█▁▁██▁▁███ ┌ ┐ ╷ ┌─────╷──┬ PR (build, .scan())) ├────┤ ┤ │ ╵ └─────╵──┴ ╷ ┌───────╷ main (build, .scan())) ├───┤ ┤ ╵ └───────╵ └ ┘ 205.99 ms 223.50 ms 241.01 ms summary PR (build, .scan())) 1.15x faster than main (build, .scan())) ``` **Single initial build + accessing `.files`**: Now things are getting interesting. Almost a 2x speedup by avoiding re-walking the file system when accessing `.files`. ``` benchmark avg (min … max) p75 / p99 (min … top 1%) --------------------------------------------- ------------------------------- PR (build, .scan() + .files) 216.35 ms/iter 214.53 ms █ █ █ (211.00 ms … 242.64 ms) 221.45 ms █ █▅█ ▅▅ ▅ ▅ ( 2.97 mb … 4.47 mb) 3.97 mb █▁███▁██▁▁▁▁▁▁█▁▁▁▁▁█ main (build, .scan() + .files) 414.79 ms/iter 406.05 ms ██ (396.72 ms … 542.30 ms) 413.69 ms ▅ ██▅ ▅▅ ▅ ▅ ▅ ( 5.19 mb … 6.03 mb) 5.63 mb █▁▁▁███▁██▁█▁█▁▁▁▁▁▁█ ┌ ┐ ┌┬╷ PR (build, .scan() + .files) ││┤ └┴╵ ╷┌──╷ main (build, .scan() + .files) ├┤ ┤ ╵└──╵ └ ┘ 211.00 ms 312.34 ms 413.69 ms summary PR (build, .scan() + .files) 1.92x faster than main (build, .scan() + .files) ``` **Watch/dev mode, only scanning**: This is where we see bigger wins because now we're using the parallel walker. ``` benchmark avg (min … max) p75 / p99 (min … top 1%) --------------------------------------------- ------------------------------- PR (watch, .scan())) 76.26 ms/iter 77.41 ms █ (73.56 ms … 79.02 ms) 77.81 ms ▅ ▅ ▅ ▅▅ ▅▅▅ ▅ █ ( 2.53 mb … 5.52 mb) 3.06 mb █▁▁▁█▁▁█▁██▁███▁▁▁█▁█ main (watch, .scan())) 166.71 ms/iter 165.14 ms █ █ (161.49 ms … 198.26 ms) 168.99 ms █ ▅█ ▅▅▅ ▅ ▅ ▅ ( 1.08 mb … 2.72 mb) 1.24 mb █▁██▁███▁▁█▁▁█▁▁▁▁▁▁█ ┌ ┐ ╷┬┐ PR (watch, .scan())) ├││ ╵┴┘ ╷┌─┬╷ main (watch, .scan())) ├┤ │┤ ╵└─┴╵ └ ┘ 73.56 ms 121.28 ms 168.99 ms summary PR (watch, .scan())) 2.19x faster than main (watch, .scan())) ``` **Watch/dev mode, scanning + accessing `.files`**: This is the biggest win of them all because we have all the benefits combined: 1. Avoiding re-walking the file system when accessing `.files` 2. Using the parallel walker for faster file system walking ``` benchmark avg (min … max) p75 / p99 (min … top 1%) --------------------------------------------- ------------------------------- PR (watch, .scan() + .files) 84.04 ms/iter 84.84 ms █ (80.96 ms … 87.53 ms) 87.27 ms ▅▅ ▅▅ ▅▅ █▅ ▅ ▅ ( 15.42 mb … 31.34 mb) 22.16 mb ██▁▁▁██▁██▁██▁█▁▁▁▁▁█ main (watch, .scan() + .files) 338.59 ms/iter 353.89 ms █ (321.87 ms … 378.43 ms) 358.70 ms █ █ ( 2.39 mb … 2.45 mb) 2.42 mb ███▁▁██▁▁▁▁▁▁▁▁▁▁██▁█ ┌ ┐ ┬┐ PR (watch, .scan() + .files) ││ ┴┘ ┌──┬─┐╷ main (watch, .scan() + .files) │ │ ├┤ └──┴─┘╵ └ ┘ 80.96 ms 219.83 ms 358.70 ms summary PR (watch, .scan() + .files) 4.03x faster than main (watch, .scan() + .files) ``` ## Test plan 1. All existing tests still pass 2. All public APIs remain the same 3. In the benchmarks I'm sharing, I first verify that the candidates returned and the files returned are the same before and after the change. 4. Benchmarked against real codebases, and against a synthetic large codebase (5000 files). Fixes: #19616 --- CHANGELOG.md | 1 + crates/oxide/src/cursor.rs | 130 ++--- .../extractor/arbitrary_property_machine.rs | 18 +- .../src/extractor/arbitrary_value_machine.rs | 12 +- .../extractor/arbitrary_variable_machine.rs | 22 +- .../oxide/src/extractor/candidate_machine.rs | 6 +- .../src/extractor/css_variable_machine.rs | 8 +- crates/oxide/src/extractor/mod.rs | 8 +- .../oxide/src/extractor/modifier_machine.rs | 8 +- .../src/extractor/named_utility_machine.rs | 30 +- .../src/extractor/named_variant_machine.rs | 20 +- .../src/extractor/pre_processors/clojure.rs | 30 +- .../src/extractor/pre_processors/elixir.rs | 14 +- .../src/extractor/pre_processors/haml.rs | 28 +- .../src/extractor/pre_processors/json.rs | 4 +- .../src/extractor/pre_processors/markdown.rs | 2 +- .../oxide/src/extractor/pre_processors/pug.rs | 12 +- .../src/extractor/pre_processors/ruby.rs | 27 +- .../src/extractor/pre_processors/rust.rs | 10 +- .../src/extractor/pre_processors/slim.rs | 20 +- .../oxide/src/extractor/pre_processors/vue.rs | 3 +- crates/oxide/src/extractor/string_machine.rs | 10 +- crates/oxide/src/extractor/utility_machine.rs | 10 +- crates/oxide/src/extractor/variant_machine.rs | 4 +- crates/oxide/src/fast_skip.rs | 2 +- crates/oxide/src/scanner/detect_sources.rs | 8 +- crates/oxide/src/scanner/init_tracing.rs | 70 +++ crates/oxide/src/scanner/mod.rs | 541 ++++++++++-------- 28 files changed, 579 insertions(+), 479 deletions(-) create mode 100644 crates/oxide/src/scanner/init_tracing.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index a549a1232..065700829 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Fix infinite loop when using `@variant` inside `@custom-variant` ([#19633](https://github.com/tailwindlabs/tailwindcss/pull/19633)) - Allow multiples of `.25` in `aspect-*` fractions ([#19688](https://github.com/tailwindlabs/tailwindcss/pull/19688)) - Ensure changes to external files listed via `@source` trigger a full page reload when using `@tailwindcss/vite` ([#19670](https://github.com/tailwindlabs/tailwindcss/pull/19670)) +- Improve performance Oxide scanner in bigger projects ([#19632](https://github.com/tailwindlabs/tailwindcss/pull/19632)) ### Deprecated diff --git a/crates/oxide/src/cursor.rs b/crates/oxide/src/cursor.rs index 818a0ef1b..71cdccdf6 100644 --- a/crates/oxide/src/cursor.rs +++ b/crates/oxide/src/cursor.rs @@ -1,44 +1,49 @@ use std::{ascii::escape_default, fmt::Display}; -#[derive(Debug, Clone)] +#[derive(Debug, Clone, Copy)] pub struct Cursor<'a> { // The input we're scanning pub input: &'a [u8], // The location of the cursor in the input pub pos: usize, - - /// Is the cursor at the start of the input - pub at_start: bool, - - /// Is the cursor at the end of the input - pub at_end: bool, - - /// The previously consumed character - /// If `at_start` is true, this will be NUL - pub prev: u8, - - /// The current character - pub curr: u8, - - /// The upcoming character (if any) - /// If `at_end` is true, this will be NUL - pub next: u8, } impl<'a> Cursor<'a> { + #[inline(always)] pub fn new(input: &'a [u8]) -> Self { - let mut cursor = Self { - input, - pos: 0, - at_start: true, - at_end: false, - prev: 0x00, - curr: 0x00, - next: 0x00, - }; - cursor.move_to(0); - cursor + Self { input, pos: 0 } + } + + /// The current byte at `pos`, or 0x00 if past the end. + #[inline(always)] + pub fn curr(&self) -> u8 { + if self.pos < self.input.len() { + unsafe { *self.input.get_unchecked(self.pos) } + } else { + 0x00 + } + } + + /// The next byte at `pos + 1`, or 0x00 if past the end. + #[inline(always)] + pub fn next(&self) -> u8 { + let next_pos = self.pos + 1; + if next_pos < self.input.len() { + unsafe { *self.input.get_unchecked(next_pos) } + } else { + 0x00 + } + } + + /// The previous byte at `pos - 1`, or 0x00 if at the start. + #[inline(always)] + pub fn prev(&self) -> u8 { + if self.pos > 0 { + unsafe { *self.input.get_unchecked(self.pos - 1) } + } else { + 0x00 + } } pub fn advance_by(&mut self, amount: usize) { @@ -48,38 +53,15 @@ impl<'a> Cursor<'a> { #[inline(always)] pub fn advance(&mut self) { self.pos += 1; - - self.prev = self.curr; - self.curr = self.next; - self.next = *self - .input - .get(self.pos.saturating_add(1)) - .unwrap_or(&0x00u8); } #[inline(always)] pub fn advance_twice(&mut self) { self.pos += 2; - - self.prev = self.next; - self.curr = *self.input.get(self.pos).unwrap_or(&0x00u8); - self.next = *self - .input - .get(self.pos.saturating_add(1)) - .unwrap_or(&0x00u8); } pub fn move_to(&mut self, pos: usize) { - let len = self.input.len(); - let pos = pos.clamp(0, len); - - self.pos = pos; - self.at_start = pos == 0; - self.at_end = pos + 1 >= len; - - self.prev = *self.input.get(pos.wrapping_sub(1)).unwrap_or(&0x00u8); - self.curr = *self.input.get(pos).unwrap_or(&0x00u8); - self.next = *self.input.get(pos.saturating_add(1)).unwrap_or(&0x00u8); + self.pos = pos.min(self.input.len()); } } @@ -90,9 +72,9 @@ impl Display for Cursor<'_> { let pos = format!("{: >len_count$}", self.pos, len_count = len.len()); write!(f, "{}/{} ", pos, len)?; - if self.at_start { + if self.pos == 0 { write!(f, "S ")?; - } else if self.at_end { + } else if self.pos + 1 >= self.input.len() { write!(f, "E ")?; } else { write!(f, "M ")?; @@ -109,9 +91,9 @@ impl Display for Cursor<'_> { write!( f, "[{} {} {}]", - to_str(self.prev), - to_str(self.curr), - to_str(self.next) + to_str(self.prev()), + to_str(self.curr()), + to_str(self.next()) ) } } @@ -125,36 +107,28 @@ mod test { fn test_cursor() { let mut cursor = Cursor::new(b"hello world"); assert_eq!(cursor.pos, 0); - assert!(cursor.at_start); - assert!(!cursor.at_end); - assert_eq!(cursor.prev, 0x00); - assert_eq!(cursor.curr, b'h'); - assert_eq!(cursor.next, b'e'); + assert_eq!(cursor.prev(), 0x00); + assert_eq!(cursor.curr(), b'h'); + assert_eq!(cursor.next(), b'e'); cursor.advance_by(1); assert_eq!(cursor.pos, 1); - assert!(!cursor.at_start); - assert!(!cursor.at_end); - assert_eq!(cursor.prev, b'h'); - assert_eq!(cursor.curr, b'e'); - assert_eq!(cursor.next, b'l'); + assert_eq!(cursor.prev(), b'h'); + assert_eq!(cursor.curr(), b'e'); + assert_eq!(cursor.next(), b'l'); // Advancing too far should stop at the end cursor.advance_by(10); assert_eq!(cursor.pos, 11); - assert!(!cursor.at_start); - assert!(cursor.at_end); - assert_eq!(cursor.prev, b'd'); - assert_eq!(cursor.curr, 0x00); - assert_eq!(cursor.next, 0x00); + assert_eq!(cursor.prev(), b'd'); + assert_eq!(cursor.curr(), 0x00); + assert_eq!(cursor.next(), 0x00); // Can't advance past the end cursor.advance_by(1); assert_eq!(cursor.pos, 11); - assert!(!cursor.at_start); - assert!(cursor.at_end); - assert_eq!(cursor.prev, b'd'); - assert_eq!(cursor.curr, 0x00); - assert_eq!(cursor.next, 0x00); + assert_eq!(cursor.prev(), b'd'); + assert_eq!(cursor.curr(), 0x00); + assert_eq!(cursor.next(), 0x00); } } diff --git a/crates/oxide/src/extractor/arbitrary_property_machine.rs b/crates/oxide/src/extractor/arbitrary_property_machine.rs index 8fd13e394..a43253339 100644 --- a/crates/oxide/src/extractor/arbitrary_property_machine.rs +++ b/crates/oxide/src/extractor/arbitrary_property_machine.rs @@ -74,7 +74,7 @@ impl Machine for ArbitraryPropertyMachine { #[inline] fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { - match cursor.curr.into() { + match cursor.curr().into() { // Start of an arbitrary property Class::OpenBracket => { self.start_pos = cursor.pos; @@ -97,8 +97,8 @@ impl Machine for ArbitraryPropertyMachine { let len = cursor.input.len(); while cursor.pos < len { - match cursor.curr.into() { - Class::Dash => match cursor.next.into() { + match cursor.curr().into() { + Class::Dash => match cursor.next().into() { // Start of a CSS variable // // E.g.: `[--my-color:red]` @@ -137,7 +137,7 @@ impl ArbitraryPropertyMachine { fn parse_property_variable(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { match self.css_variable_machine.next(cursor) { MachineState::Idle => self.restart(), - MachineState::Done(_) => match cursor.next.into() { + MachineState::Done(_) => match cursor.next().into() { // End of the CSS variable, must be followed by a `:` // // E.g.: `[--my-color:red]` @@ -165,8 +165,8 @@ impl Machine for ArbitraryPropertyMachine { let len = cursor.input.len(); let start_of_value_pos = cursor.pos; while cursor.pos < len { - match cursor.curr.into() { - Class::Escape => match cursor.next.into() { + match cursor.curr().into() { + Class::Escape => match cursor.next().into() { // An escaped whitespace character is not allowed // // E.g.: `[color:var(--my-\ color)]` @@ -181,7 +181,7 @@ impl Machine for ArbitraryPropertyMachine { }, Class::OpenParen | Class::OpenBracket | Class::OpenCurly => { - if !self.bracket_stack.push(cursor.curr) { + if !self.bracket_stack.push(cursor.curr()) { return self.restart(); } cursor.advance(); @@ -190,7 +190,7 @@ impl Machine for ArbitraryPropertyMachine { Class::CloseParen | Class::CloseBracket | Class::CloseCurly if !self.bracket_stack.is_empty() => { - if !self.bracket_stack.pop(cursor.curr) { + if !self.bracket_stack.pop(cursor.curr()) { return self.restart(); } cursor.advance(); @@ -227,7 +227,7 @@ impl Machine for ArbitraryPropertyMachine { Class::Slash if start_of_value_pos == cursor.pos => return self.restart(), // String interpolation-like syntax is not allowed. E.g.: `[${x}]` - Class::Dollar if matches!(cursor.next.into(), Class::OpenCurly) => { + Class::Dollar if matches!(cursor.next().into(), Class::OpenCurly) => { return self.restart() } diff --git a/crates/oxide/src/extractor/arbitrary_value_machine.rs b/crates/oxide/src/extractor/arbitrary_value_machine.rs index f5a612df1..cbef62c42 100644 --- a/crates/oxide/src/extractor/arbitrary_value_machine.rs +++ b/crates/oxide/src/extractor/arbitrary_value_machine.rs @@ -32,7 +32,7 @@ impl Machine for ArbitraryValueMachine { #[inline] fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { // An arbitrary value must start with an open bracket - if Class::OpenBracket != cursor.curr.into() { + if Class::OpenBracket != cursor.curr().into() { return MachineState::Idle; } @@ -42,8 +42,8 @@ impl Machine for ArbitraryValueMachine { let len = cursor.input.len(); while cursor.pos < len { - match cursor.curr.into() { - Class::Escape => match cursor.next.into() { + match cursor.curr().into() { + Class::Escape => match cursor.next().into() { // An escaped whitespace character is not allowed // // E.g.: `[color:var(--my-\ color)]` @@ -61,7 +61,7 @@ impl Machine for ArbitraryValueMachine { }, Class::OpenParen | Class::OpenBracket | Class::OpenCurly => { - if !self.bracket_stack.push(cursor.curr) { + if !self.bracket_stack.push(cursor.curr()) { return self.restart(); } cursor.advance(); @@ -70,7 +70,7 @@ impl Machine for ArbitraryValueMachine { Class::CloseParen | Class::CloseBracket | Class::CloseCurly if !self.bracket_stack.is_empty() => { - if !self.bracket_stack.pop(cursor.curr) { + if !self.bracket_stack.pop(cursor.curr()) { return self.restart(); } cursor.advance(); @@ -96,7 +96,7 @@ impl Machine for ArbitraryValueMachine { Class::Whitespace => return self.restart(), // String interpolation-like syntax is not allowed. E.g.: `[${x}]` - Class::Dollar if matches!(cursor.next.into(), Class::OpenCurly) => { + Class::Dollar if matches!(cursor.next().into(), Class::OpenCurly) => { return self.restart() } diff --git a/crates/oxide/src/extractor/arbitrary_variable_machine.rs b/crates/oxide/src/extractor/arbitrary_variable_machine.rs index 6990e110b..e8d475557 100644 --- a/crates/oxide/src/extractor/arbitrary_variable_machine.rs +++ b/crates/oxide/src/extractor/arbitrary_variable_machine.rs @@ -80,13 +80,13 @@ impl Machine for ArbitraryVariableMachine { #[inline(always)] fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { - match cursor.curr.into() { + match cursor.curr().into() { // Arbitrary variables start with `(` followed by a CSS variable // // E.g.: `(--my-variable)` // ^^ // - Class::OpenParen => match cursor.next.into() { + Class::OpenParen => match cursor.next().into() { Class::Dash => { self.start_pos = cursor.pos; cursor.advance(); @@ -117,7 +117,7 @@ impl Machine for ArbitraryVariableMachine { fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { match self.css_variable_machine.next(cursor) { MachineState::Idle => self.restart(), - MachineState::Done(_) => match cursor.next.into() { + MachineState::Done(_) => match cursor.next().into() { // A CSS variable followed by a `,` means that there is a fallback // // E.g.: `(--my-color,red)` @@ -134,7 +134,7 @@ impl Machine for ArbitraryVariableMachine { _ => { cursor.advance(); - match cursor.curr.into() { + match cursor.curr().into() { // End of an arbitrary variable, must be followed by `)` Class::CloseParen => self.done(self.start_pos, cursor), @@ -156,7 +156,7 @@ impl Machine for ArbitraryVariableMachine { let len = cursor.input.len(); while cursor.pos < len { - match cursor.curr.into() { + match cursor.curr().into() { // Valid data type characters // // E.g.: `(length:--my-length)` @@ -169,7 +169,7 @@ impl Machine for ArbitraryVariableMachine { // // E.g.: `(length:--my-length)` // ^ - Class::Colon => match cursor.next.into() { + Class::Colon => match cursor.next().into() { Class::Dash => { cursor.advance(); return self.transition::().next(cursor); @@ -196,8 +196,8 @@ impl Machine for ArbitraryVariableMachine { fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { let len = cursor.input.len(); while cursor.pos < len { - match cursor.curr.into() { - Class::Escape => match cursor.next.into() { + match cursor.curr().into() { + Class::Escape => match cursor.next().into() { // An escaped whitespace character is not allowed // // E.g.: `(--my-\ color)` @@ -212,7 +212,7 @@ impl Machine for ArbitraryVariableMachine { }, Class::OpenParen | Class::OpenBracket | Class::OpenCurly => { - if !self.bracket_stack.push(cursor.curr) { + if !self.bracket_stack.push(cursor.curr()) { return self.restart(); } cursor.advance(); @@ -221,7 +221,7 @@ impl Machine for ArbitraryVariableMachine { Class::CloseParen | Class::CloseBracket | Class::CloseCurly if !self.bracket_stack.is_empty() => { - if !self.bracket_stack.pop(cursor.curr) { + if !self.bracket_stack.pop(cursor.curr()) { return self.restart(); } cursor.advance(); @@ -253,7 +253,7 @@ impl Machine for ArbitraryVariableMachine { Class::Whitespace => return self.restart(), // String interpolation-like syntax is not allowed. E.g.: `[${x}]` - Class::Dollar if matches!(cursor.next.into(), Class::OpenCurly) => { + Class::Dollar if matches!(cursor.next().into(), Class::OpenCurly) => { return self.restart() } diff --git a/crates/oxide/src/extractor/candidate_machine.rs b/crates/oxide/src/extractor/candidate_machine.rs index c82cc3707..a755cf02a 100644 --- a/crates/oxide/src/extractor/candidate_machine.rs +++ b/crates/oxide/src/extractor/candidate_machine.rs @@ -32,14 +32,14 @@ impl Machine for CandidateMachine { while cursor.pos < len { // Skip ahead for known characters that will never be part of a candidate. No need to // run any sub-machines. - if cursor.curr.is_ascii_whitespace() { + if cursor.curr().is_ascii_whitespace() { self.reset(); cursor.advance(); continue; } // Candidates don't start with these characters, so we can skip ahead. - if matches!(cursor.curr, b':' | b'"' | b'\'' | b'`') { + if matches!(cursor.curr(), b':' | b'"' | b'\'' | b'`') { self.reset(); cursor.advance(); continue; @@ -56,7 +56,7 @@ impl Machine for CandidateMachine { // E.g.: `Some Class` // ^ ^ Invalid, we can jump ahead to the next boundary // - if matches!(cursor.curr, b'<' | b'A'..=b'Z') { + if matches!(cursor.curr(), b'<' | b'A'..=b'Z') { if let Some(offset) = cursor.input[cursor.pos..] .iter() .position(|&c| is_valid_before_boundary(&c)) diff --git a/crates/oxide/src/extractor/css_variable_machine.rs b/crates/oxide/src/extractor/css_variable_machine.rs index 4e19c7111..7beb827ee 100644 --- a/crates/oxide/src/extractor/css_variable_machine.rs +++ b/crates/oxide/src/extractor/css_variable_machine.rs @@ -20,7 +20,7 @@ impl Machine for CssVariableMachine { #[inline] fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { // CSS Variables must start with `--` - if Class::Dash != cursor.curr.into() || Class::Dash != cursor.next.into() { + if Class::Dash != cursor.curr().into() || Class::Dash != cursor.next().into() { return MachineState::Idle; } @@ -30,11 +30,11 @@ impl Machine for CssVariableMachine { cursor.advance_twice(); while cursor.pos < len { - match cursor.curr.into() { + match cursor.curr().into() { // https://drafts.csswg.org/css-syntax-3/#ident-token-diagram // Class::AllowedCharacter | Class::Dash => { - match cursor.next.into() { + match cursor.next().into() { // Valid character followed by a valid character or an escape character // // E.g.: `--my-variable` @@ -58,7 +58,7 @@ impl Machine for CssVariableMachine { } } - Class::Escape => match cursor.next.into() { + Class::Escape => match cursor.next().into() { // An escaped whitespace character is not allowed // // In CSS it is allowed, but in the context of a class it's not because then we diff --git a/crates/oxide/src/extractor/mod.rs b/crates/oxide/src/extractor/mod.rs index add0de4ac..a37d6a0a4 100644 --- a/crates/oxide/src/extractor/mod.rs +++ b/crates/oxide/src/extractor/mod.rs @@ -86,7 +86,7 @@ impl<'a> Extractor<'a> { { let cursor = &mut self.cursor.clone(); while cursor.pos < len { - if cursor.curr.is_ascii_whitespace() { + if cursor.curr().is_ascii_whitespace() { cursor.advance(); continue; } @@ -104,7 +104,7 @@ impl<'a> Extractor<'a> { let cursor = &mut self.cursor.clone(); while cursor.pos < len { - if cursor.curr.is_ascii_whitespace() { + if cursor.curr().is_ascii_whitespace() { cursor.advance(); continue; } @@ -147,7 +147,7 @@ impl<'a> Extractor<'a> { let cursor = &mut self.cursor.clone(); while cursor.pos < len { - if cursor.curr.is_ascii_whitespace() { + if cursor.curr().is_ascii_whitespace() { cursor.advance(); continue; } @@ -238,7 +238,7 @@ mod tests { use std::hint::black_box; fn pre_process_input(input: &str, extension: &str) -> String { - let input = crate::scanner::pre_process_input(input.as_bytes(), extension); + let input = crate::scanner::pre_process_input(input.as_bytes().to_vec(), extension); String::from_utf8(input).unwrap() } diff --git a/crates/oxide/src/extractor/modifier_machine.rs b/crates/oxide/src/extractor/modifier_machine.rs index 22668a828..811bc9ded 100644 --- a/crates/oxide/src/extractor/modifier_machine.rs +++ b/crates/oxide/src/extractor/modifier_machine.rs @@ -31,14 +31,14 @@ impl Machine for ModifierMachine { #[inline] fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { // A modifier must start with a `/`, everything else is not a valid start of a modifier - if Class::Slash != cursor.curr.into() { + if Class::Slash != cursor.curr().into() { return MachineState::Idle; } let start_pos = cursor.pos; cursor.advance(); - match cursor.curr.into() { + match cursor.curr().into() { // Start of an arbitrary value: // // ``` @@ -70,9 +70,9 @@ impl Machine for ModifierMachine { Class::ValidStart => { let len = cursor.input.len(); while cursor.pos < len { - match cursor.curr.into() { + match cursor.curr().into() { Class::ValidStart | Class::ValidInside => { - match cursor.next.into() { + match cursor.next().into() { // Only valid characters are allowed, if followed by another valid character Class::ValidStart | Class::ValidInside => cursor.advance(), diff --git a/crates/oxide/src/extractor/named_utility_machine.rs b/crates/oxide/src/extractor/named_utility_machine.rs index d218dbeb7..24dc027b2 100644 --- a/crates/oxide/src/extractor/named_utility_machine.rs +++ b/crates/oxide/src/extractor/named_utility_machine.rs @@ -52,8 +52,8 @@ impl Machine for NamedUtilityMachine { #[inline] fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { - match cursor.curr.into() { - Class::AlphaLower => match cursor.next.into() { + match cursor.curr().into() { + Class::AlphaLower => match cursor.next().into() { // Valid single character utility in between quotes // // E.g.: `
` @@ -90,7 +90,7 @@ impl Machine for NamedUtilityMachine { // // E.g.: `-mx-2.5` // ^^ - Class::Dash => match cursor.next.into() { + Class::Dash => match cursor.next().into() { Class::AlphaLower => { self.start_pos = cursor.pos; cursor.advance(); @@ -118,7 +118,7 @@ impl Machine for NamedUtilityMachine { let len = cursor.input.len(); while cursor.pos < len { - match cursor.curr.into() { + match cursor.curr().into() { // Followed by a boundary character, we are at the end of the utility. // // E.g.: `'flex'` @@ -132,14 +132,14 @@ impl Machine for NamedUtilityMachine { // E.g.: `:div="{ flex: true }"` (JavaScript object syntax) // ^ Class::AlphaLower | Class::AlphaUpper => { - if is_valid_after_boundary(&cursor.next) || { + if is_valid_after_boundary(&cursor.next()) || { // Or any of these characters // // - `:`, because of JS object keys // - `/`, because of modifiers // - `!`, because of important matches!( - cursor.next.into(), + cursor.next().into(), Class::Colon | Class::Slash | Class::Exclamation ) } { @@ -150,7 +150,7 @@ impl Machine for NamedUtilityMachine { cursor.advance() } - Class::Dash => match cursor.next.into() { + Class::Dash => match cursor.next().into() { // Start of an arbitrary value // // E.g.: `bg-[#0088cc]` @@ -196,7 +196,7 @@ impl Machine for NamedUtilityMachine { _ => return self.restart(), }, - Class::Underscore => match cursor.next.into() { + Class::Underscore => match cursor.next().into() { // Valid characters _if_ followed by another valid character. These characters are // only valid inside of the utility but not at the end of the utility. // @@ -225,14 +225,14 @@ impl Machine for NamedUtilityMachine { // ^ // E.g.: `:div="{ flex: true }"` (JavaScript object syntax) // ^ - _ if is_valid_after_boundary(&cursor.next) || { + _ if is_valid_after_boundary(&cursor.next()) || { // Or any of these characters // // - `:`, because of JS object keys // - `/`, because of modifiers // - `!`, because of important matches!( - cursor.next.into(), + cursor.next().into(), Class::Colon | Class::Slash | Class::Exclamation ) } => @@ -249,11 +249,11 @@ impl Machine for NamedUtilityMachine { // E.g.: `px-2.5` // ^^^ Class::Dot => { - if !matches!(cursor.prev.into(), Class::Number) { + if !matches!(cursor.prev().into(), Class::Number) { return self.restart(); } - if !matches!(cursor.next.into(), Class::Number) { + if !matches!(cursor.next().into(), Class::Number) { return self.restart(); } @@ -278,7 +278,7 @@ impl Machine for NamedUtilityMachine { // Class::Number => { if !matches!( - cursor.prev.into(), + cursor.prev().into(), Class::Dash | Class::Underscore | Class::Dot @@ -290,7 +290,7 @@ impl Machine for NamedUtilityMachine { } if !matches!( - cursor.next.into(), + cursor.next().into(), Class::Dot | Class::Number | Class::AlphaLower @@ -314,7 +314,7 @@ impl Machine for NamedUtilityMachine { // ^^ // ``` Class::Percent => { - if !matches!(cursor.prev.into(), Class::Number) { + if !matches!(cursor.prev().into(), Class::Number) { return self.restart(); } diff --git a/crates/oxide/src/extractor/named_variant_machine.rs b/crates/oxide/src/extractor/named_variant_machine.rs index b6de8220b..285fe6ab5 100644 --- a/crates/oxide/src/extractor/named_variant_machine.rs +++ b/crates/oxide/src/extractor/named_variant_machine.rs @@ -81,8 +81,8 @@ impl Machine for NamedVariantMachine { #[inline(always)] fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { - match cursor.curr.into() { - Class::AlphaLower | Class::Star => match cursor.next.into() { + match cursor.curr().into() { + Class::AlphaLower | Class::Star => match cursor.next().into() { // Valid single character variant, must be followed by a `:` // // E.g.: `
` @@ -134,8 +134,8 @@ impl Machine for NamedVariantMachine { let len = cursor.input.len(); while cursor.pos < len { - match cursor.curr.into() { - Class::Dash => match cursor.next.into() { + match cursor.curr().into() { + Class::Dash => match cursor.next().into() { // Start of an arbitrary value // // E.g.: `data-[state=pending]:`. @@ -192,7 +192,7 @@ impl Machine for NamedVariantMachine { }; } - Class::Underscore => match cursor.next.into() { + Class::Underscore => match cursor.next().into() { // Valid characters _if_ followed by another valid character. These characters are // only valid inside of the variant but not at the end of the variant. // @@ -240,11 +240,11 @@ impl Machine for NamedVariantMachine { // E.g.: `2.5xl:flex` // ^^^ Class::Dot => { - if !matches!(cursor.prev.into(), Class::Number) { + if !matches!(cursor.prev().into(), Class::Number) { return self.restart(); } - if !matches!(cursor.next.into(), Class::Number) { + if !matches!(cursor.next().into(), Class::Number) { return self.restart(); } @@ -263,7 +263,7 @@ impl Machine for NamedVariantMachine { impl NamedVariantMachine { #[inline(always)] fn parse_arbitrary_end(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { - match cursor.next.into() { + match cursor.next().into() { Class::Slash => { cursor.advance(); self.transition::().next(cursor) @@ -285,7 +285,7 @@ impl Machine for NamedVariantMachine { fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { match self.modifier_machine.next(cursor) { MachineState::Idle => self.restart(), - MachineState::Done(_) => match cursor.next.into() { + MachineState::Done(_) => match cursor.next().into() { // Modifier must be followed by a `:` // // E.g.: `group-hover/name:` @@ -308,7 +308,7 @@ impl Machine for NamedVariantMachine { #[inline(always)] fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { - match cursor.curr.into() { + match cursor.curr().into() { // The end of a variant must be the `:` // // E.g.: `hover:` diff --git a/crates/oxide/src/extractor/pre_processors/clojure.rs b/crates/oxide/src/extractor/pre_processors/clojure.rs index ecf02b4ff..3fb4ce601 100644 --- a/crates/oxide/src/extractor/pre_processors/clojure.rs +++ b/crates/oxide/src/extractor/pre_processors/clojure.rs @@ -32,14 +32,14 @@ impl PreProcessor for Clojure { let mut cursor = cursor::Cursor::new(&content); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Consume strings as-is b'"' => { result[cursor.pos] = b' '; cursor.advance(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Escaped character, skip ahead to the next character b'\\' => cursor.advance_twice(), @@ -57,8 +57,8 @@ impl PreProcessor for Clojure { // Discard line comments until the end of the line. // Comments start with `;;` - b';' if matches!(cursor.next, b';') => { - while cursor.pos < len && cursor.curr != b'\n' { + b';' if matches!(cursor.next(), b';') => { + while cursor.pos < len && cursor.curr() != b'\n' { result[cursor.pos] = b' '; cursor.advance(); } @@ -70,7 +70,7 @@ impl PreProcessor for Clojure { cursor.advance(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // A `.` surrounded by digits is a decimal number, so we don't want to replace it. // // E.g.: @@ -78,8 +78,8 @@ impl PreProcessor for Clojure { // gap-1.5 // ^ // ``` - b'.' if cursor.prev.is_ascii_digit() - && cursor.next.is_ascii_digit() => + b'.' if cursor.prev().is_ascii_digit() + && cursor.next().is_ascii_digit() => { // Keep the `.` as-is } @@ -95,7 +95,7 @@ impl PreProcessor for Clojure { result[cursor.pos] = b' '; } // End of keyword. - _ if !is_keyword_character(cursor.curr) => { + _ if !is_keyword_character(cursor.curr()) => { result[cursor.pos] = b' '; break; } @@ -110,11 +110,11 @@ impl PreProcessor for Clojure { // Handle quote with a list, e.g.: `'(…)` // and with a vector, e.g.: `'[…]` - b'\'' if matches!(cursor.next, b'[' | b'(') => { + b'\'' if matches!(cursor.next(), b'[' | b'(') => { result[cursor.pos] = b' '; cursor.advance(); result[cursor.pos] = b' '; - let end = match cursor.curr { + let end = match cursor.curr() { b'[' => b']', b'(' => b')', _ => unreachable!(), @@ -122,7 +122,7 @@ impl PreProcessor for Clojure { // Consume until the closing `]` while cursor.pos < len { - match cursor.curr { + match cursor.curr() { x if x == end => { result[cursor.pos] = b' '; break; @@ -134,7 +134,7 @@ impl PreProcessor for Clojure { cursor.advance(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Escaped character, skip ahead to the next character b'\\' => cursor.advance_twice(), @@ -157,14 +157,14 @@ impl PreProcessor for Clojure { } // Handle quote with a keyword, e.g.: `'bg-white` - b'\'' if !cursor.next.is_ascii_whitespace() => { + b'\'' if !cursor.next().is_ascii_whitespace() => { result[cursor.pos] = b' '; cursor.advance(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // End of keyword. - _ if !is_keyword_character(cursor.curr) => { + _ if !is_keyword_character(cursor.curr()) => { result[cursor.pos] = b' '; break; } diff --git a/crates/oxide/src/extractor/pre_processors/elixir.rs b/crates/oxide/src/extractor/pre_processors/elixir.rs index 87b89a2a9..7737a6b2d 100644 --- a/crates/oxide/src/extractor/pre_processors/elixir.rs +++ b/crates/oxide/src/extractor/pre_processors/elixir.rs @@ -13,13 +13,13 @@ impl PreProcessor for Elixir { while cursor.pos < content.len() { // Look for a sigil marker - if cursor.curr != b'~' { + if cursor.curr() != b'~' { cursor.advance(); continue; } // Scan charlists, strings, and wordlists - if !matches!(cursor.next, b'c' | b'C' | b's' | b'S' | b'w' | b'W') { + if !matches!(cursor.next(), b'c' | b'C' | b's' | b'S' | b'w' | b'W') { cursor.advance(); continue; } @@ -27,7 +27,7 @@ impl PreProcessor for Elixir { cursor.advance_twice(); // Match the opening for a sigil - if !matches!(cursor.curr, b'(' | b'[' | b'{') { + if !matches!(cursor.curr(), b'(' | b'[' | b'{') { continue; } @@ -35,19 +35,19 @@ impl PreProcessor for Elixir { result[cursor.pos] = b' '; // Scan until we find a balanced closing one and replace it too - bracket_stack.push(cursor.curr); + bracket_stack.push(cursor.curr()); while cursor.pos < content.len() { cursor.advance(); - match cursor.curr { + match cursor.curr() { // Escaped character, skip ahead to the next character b'\\' => cursor.advance_twice(), b'(' | b'[' | b'{' => { - bracket_stack.push(cursor.curr); + bracket_stack.push(cursor.curr()); } b')' | b']' | b'}' if !bracket_stack.is_empty() => { - bracket_stack.pop(cursor.curr); + bracket_stack.pop(cursor.curr()); if bracket_stack.is_empty() { // Replace the closing bracket with a space diff --git a/crates/oxide/src/extractor/pre_processors/haml.rs b/crates/oxide/src/extractor/pre_processors/haml.rs index a131618ab..99b7bd052 100644 --- a/crates/oxide/src/extractor/pre_processors/haml.rs +++ b/crates/oxide/src/extractor/pre_processors/haml.rs @@ -101,7 +101,7 @@ impl PreProcessor for Haml { let mut last_newline_position = 0; while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Escape the next character b'\\' => { cursor.advance_twice(); @@ -119,7 +119,7 @@ impl PreProcessor for Haml { b'-' if cursor.input[last_newline_position..cursor.pos] .iter() .all(u8::is_ascii_whitespace) - && matches!(cursor.next, b'#') => + && matches!(cursor.next(), b'#') => { // Just consume the comment let updated_last_newline_position = @@ -159,7 +159,7 @@ impl PreProcessor for Haml { // Override the last known newline position last_newline_position = end; - let replaced = pre_process_input(ruby_code, "rb"); + let replaced = pre_process_input(ruby_code.to_vec(), "rb"); result.replace_range(start..end, replaced); } @@ -190,7 +190,7 @@ impl PreProcessor for Haml { // digit. // E.g.: `bg-red-500.2xl:flex` // ^^^ - if cursor.prev.is_ascii_digit() && cursor.next.is_ascii_digit() { + if cursor.prev().is_ascii_digit() && cursor.next().is_ascii_digit() { let mut next_cursor = cursor.clone(); next_cursor.advance(); @@ -213,11 +213,11 @@ impl PreProcessor for Haml { if bracket_stack.is_empty() { result[cursor.pos] = b' '; } - bracket_stack.push(cursor.curr); + bracket_stack.push(cursor.curr()); } b')' | b']' | b'}' if !bracket_stack.is_empty() => { - bracket_stack.pop(cursor.curr); + bracket_stack.pop(cursor.curr()); // Replace closing bracket with a space if bracket_stack.is_empty() { @@ -257,7 +257,7 @@ impl Haml { // :url => { :action => "add", :id => product.id }, // :update => { :success => "cart", :failure => "error" } // ``` - let evaluation_type = cursor.curr; + let evaluation_type = cursor.curr(); let block_indentation_level = cursor .pos @@ -267,17 +267,17 @@ impl Haml { let mut last_newline_position = last_known_newline_position; // Consume until the end of the line first - while cursor.pos < len && cursor.curr != b'\n' { + while cursor.pos < len && cursor.curr() != b'\n' { cursor.advance(); } // Block is already done, aka just a line - if evaluation_type == b'=' && cursor.prev != b',' { + if evaluation_type == b'=' && cursor.prev() != b',' { return cursor.pos; } 'outer: while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Escape the next character b'\\' => { cursor.advance_twice(); @@ -289,7 +289,7 @@ impl Haml { last_newline_position = cursor.pos; // We are done with this block - if evaluation_type == b'=' && cursor.prev != b',' { + if evaluation_type == b'=' && cursor.prev() != b',' { break; } @@ -300,11 +300,11 @@ impl Haml { // Skip whitespace and compute the indentation level x if x.is_ascii_whitespace() => { // Find first non-whitespace character - while cursor.pos < len && cursor.curr.is_ascii_whitespace() { - if cursor.curr == b'\n' { + while cursor.pos < len && cursor.curr().is_ascii_whitespace() { + if cursor.curr() == b'\n' { last_newline_position = cursor.pos; - if evaluation_type == b'=' && cursor.prev != b',' { + if evaluation_type == b'=' && cursor.prev() != b',' { // We are done with this block break 'outer; } diff --git a/crates/oxide/src/extractor/pre_processors/json.rs b/crates/oxide/src/extractor/pre_processors/json.rs index 809eb501e..cd457050c 100644 --- a/crates/oxide/src/extractor/pre_processors/json.rs +++ b/crates/oxide/src/extractor/pre_processors/json.rs @@ -11,13 +11,13 @@ impl PreProcessor for Json { let mut cursor = cursor::Cursor::new(content); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Consume strings as-is b'"' => { cursor.advance(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Escaped character, skip ahead to the next character b'\\' => cursor.advance_twice(), diff --git a/crates/oxide/src/extractor/pre_processors/markdown.rs b/crates/oxide/src/extractor/pre_processors/markdown.rs index 63dc37d1e..d2acc6a58 100644 --- a/crates/oxide/src/extractor/pre_processors/markdown.rs +++ b/crates/oxide/src/extractor/pre_processors/markdown.rs @@ -13,7 +13,7 @@ impl PreProcessor for Markdown { let mut in_directive = false; while cursor.pos < len { - match (in_directive, cursor.curr) { + match (in_directive, cursor.curr()) { (false, b'{') => { result[cursor.pos] = b' '; in_directive = true; diff --git a/crates/oxide/src/extractor/pre_processors/pug.rs b/crates/oxide/src/extractor/pre_processors/pug.rs index 7b44fcf5b..70dfe5683 100644 --- a/crates/oxide/src/extractor/pre_processors/pug.rs +++ b/crates/oxide/src/extractor/pre_processors/pug.rs @@ -15,7 +15,7 @@ impl PreProcessor for Pug { let mut bracket_stack = BracketStack::default(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Only replace `.` with a space if it's not surrounded by numbers. E.g.: // // ```diff @@ -43,7 +43,7 @@ impl PreProcessor for Pug { // digit. // E.g.: `bg-red-500.2xl:flex` // ^^^ - if cursor.prev.is_ascii_digit() && cursor.next.is_ascii_digit() { + if cursor.prev().is_ascii_digit() && cursor.next().is_ascii_digit() { let mut next_cursor = cursor.clone(); next_cursor.advance(); @@ -68,17 +68,17 @@ impl PreProcessor for Pug { // // However, we also need to make sure that we keep the parens that are part of the // utility class. E.g.: `bg-(--my-color)`. - b'(' if bracket_stack.is_empty() && !matches!(cursor.prev, b'-' | b'/') => { + b'(' if bracket_stack.is_empty() && !matches!(cursor.prev(), b'-' | b'/') => { result[cursor.pos] = b' '; - bracket_stack.push(cursor.curr); + bracket_stack.push(cursor.curr()); } b'(' | b'[' | b'{' => { - bracket_stack.push(cursor.curr); + bracket_stack.push(cursor.curr()); } b')' | b']' | b'}' if !bracket_stack.is_empty() => { - bracket_stack.pop(cursor.curr); + bracket_stack.pop(cursor.curr()); } // Consume everything else diff --git a/crates/oxide/src/extractor/pre_processors/ruby.rs b/crates/oxide/src/extractor/pre_processors/ruby.rs index 89ac1ee7b..1f2221414 100644 --- a/crates/oxide/src/extractor/pre_processors/ruby.rs +++ b/crates/oxide/src/extractor/pre_processors/ruby.rs @@ -68,7 +68,8 @@ impl PreProcessor for Ruby { } let body = &content_as_str[body_start..body_end]; - let replaced = pre_process_input(body.as_bytes(), &lang.to_ascii_lowercase()); + let replaced = + pre_process_input(body.as_bytes().to_vec(), &lang.to_ascii_lowercase()); result.replace_range(body_start..body_end, replaced); break; @@ -77,12 +78,12 @@ impl PreProcessor for Ruby { // Ruby extraction while cursor.pos < len { - match cursor.curr { + match cursor.curr() { b'"' => { cursor.advance(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Escaped character, skip ahead to the next character b'\\' => cursor.advance_twice(), @@ -102,7 +103,7 @@ impl PreProcessor for Ruby { cursor.advance(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Escaped character, skip ahead to the next character b'\\' => cursor.advance_twice(), @@ -123,12 +124,12 @@ impl PreProcessor for Ruby { // Except for strict locals, these are defined in a `<%# locals: … %>`. Checking if // the comment is preceded by a `%` should be enough without having to perform more // parsing logic. Worst case we _do_ scan a few comments. - b'#' if !matches!(cursor.prev, b'%') => { + b'#' if !matches!(cursor.prev(), b'%') => { result[cursor.pos] = b' '; cursor.advance(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // End of the comment b'\n' => break, @@ -148,7 +149,7 @@ impl PreProcessor for Ruby { } // Looking for `%w`, `%W`, or `%p` - if cursor.curr != b'%' || !matches!(cursor.next, b'w' | b'W' | b'p') { + if cursor.curr() != b'%' || !matches!(cursor.next(), b'w' | b'W' | b'p') { cursor.advance(); continue; } @@ -156,7 +157,7 @@ impl PreProcessor for Ruby { cursor.advance_twice(); // Boundary character - let boundary = match cursor.curr { + let boundary = match cursor.curr() { b'[' => b']', b'(' => b')', b'{' => b'}', @@ -177,11 +178,11 @@ impl PreProcessor for Ruby { cursor.advance(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Skip escaped characters b'\\' => { // Use backslash to embed spaces in the strings. - if cursor.next == b' ' { + if cursor.next() == b' ' { result[cursor.pos] = b' '; } @@ -190,19 +191,19 @@ impl PreProcessor for Ruby { // Start of a nested bracket b'[' | b'(' | b'{' => { - bracket_stack.push(cursor.curr); + bracket_stack.push(cursor.curr()); } // End of a nested bracket b']' | b')' | b'}' if !bracket_stack.is_empty() => { - if !bracket_stack.pop(cursor.curr) { + if !bracket_stack.pop(cursor.curr()) { // Unbalanced cursor.advance(); } } // End of the pattern, replace the boundary character with a space - _ if cursor.curr == boundary => { + _ if cursor.curr() == boundary => { if boundary != b'\n' { result[cursor.pos] = b' '; } diff --git a/crates/oxide/src/extractor/pre_processors/rust.rs b/crates/oxide/src/extractor/pre_processors/rust.rs index 6404fffb5..34c9e0c91 100644 --- a/crates/oxide/src/extractor/pre_processors/rust.rs +++ b/crates/oxide/src/extractor/pre_processors/rust.rs @@ -33,7 +33,7 @@ impl Rust { let mut bracket_stack = bracket_stack::BracketStack::default(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Escaped character, skip ahead to the next character b'\\' => { cursor.advance_twice(); @@ -46,7 +46,7 @@ impl Rust { cursor.advance(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Escaped character, skip ahead to the next character b'\\' => cursor.advance_twice(), @@ -89,7 +89,7 @@ impl Rust { // digit. // E.g.: `bg-red-500.2xl:flex` // ^^^ - if cursor.prev.is_ascii_digit() && cursor.next.is_ascii_digit() { + if cursor.prev().is_ascii_digit() && cursor.next().is_ascii_digit() { let mut next_cursor = cursor.clone(); next_cursor.advance(); @@ -103,11 +103,11 @@ impl Rust { } b'[' => { - bracket_stack.push(cursor.curr); + bracket_stack.push(cursor.curr()); } b']' if !bracket_stack.is_empty() => { - bracket_stack.pop(cursor.curr); + bracket_stack.pop(cursor.curr()); } // Consume everything else diff --git a/crates/oxide/src/extractor/pre_processors/slim.rs b/crates/oxide/src/extractor/pre_processors/slim.rs index 6d41e5a09..eacfb55ee 100644 --- a/crates/oxide/src/extractor/pre_processors/slim.rs +++ b/crates/oxide/src/extractor/pre_processors/slim.rs @@ -15,7 +15,7 @@ impl PreProcessor for Slim { let mut bracket_stack = BracketStack::default(); while cursor.pos < len { - match cursor.curr { + match cursor.curr() { // Only replace `.` with a space if it's not surrounded by numbers. E.g.: // // ```diff @@ -43,7 +43,7 @@ impl PreProcessor for Slim { // digit. // E.g.: `bg-red-500.2xl:flex` // ^^^ - if cursor.prev.is_ascii_digit() && cursor.next.is_ascii_digit() { + if cursor.prev().is_ascii_digit() && cursor.next().is_ascii_digit() { let mut next_cursor = cursor.clone(); next_cursor.advance(); @@ -65,7 +65,7 @@ impl PreProcessor for Slim { // class=%w[bg-blue-500 w-10 h-10] // ] // ``` - b'%' if matches!(cursor.next, b'w' | b'W') + b'%' if matches!(cursor.next(), b'w' | b'W') && matches!(cursor.input.get(cursor.pos + 2), Some(b'[' | b'(' | b'{')) => { result[cursor.pos] = b' '; // Replace `%` @@ -73,7 +73,7 @@ impl PreProcessor for Slim { result[cursor.pos] = b' '; // Replace `w` cursor.advance(); result[cursor.pos] = b' '; // Replace `[` or `(` or `{` - bracket_stack.push(cursor.curr); + bracket_stack.push(cursor.curr()); cursor.advance(); // Move past the bracket continue; } @@ -96,10 +96,10 @@ impl PreProcessor for Slim { // Instead of listing all boundary characters, let's list the characters we know // will be invalid instead. b'[' if bracket_stack.is_empty() - && matches!(cursor.prev, b'a'..=b'z' | b'A'..=b'Z' | b'0'..=b'9') => + && matches!(cursor.prev(), b'a'..=b'z' | b'A'..=b'Z' | b'0'..=b'9') => { result[cursor.pos] = b' '; - bracket_stack.push(cursor.curr); + bracket_stack.push(cursor.curr()); } // In Slim the class name shorthand can be followed by a parenthesis. E.g.: @@ -114,17 +114,17 @@ impl PreProcessor for Slim { // // However, we also need to make sure that we keep the parens that are part of the // utility class. E.g.: `bg-(--my-color)`. - b'(' if bracket_stack.is_empty() && !matches!(cursor.prev, b'-' | b'/') => { + b'(' if bracket_stack.is_empty() && !matches!(cursor.prev(), b'-' | b'/') => { result[cursor.pos] = b' '; - bracket_stack.push(cursor.curr); + bracket_stack.push(cursor.curr()); } b'(' | b'[' | b'{' => { - bracket_stack.push(cursor.curr); + bracket_stack.push(cursor.curr()); } b')' | b']' | b'}' if !bracket_stack.is_empty() => { - bracket_stack.pop(cursor.curr); + bracket_stack.pop(cursor.curr()); } // Consume everything else diff --git a/crates/oxide/src/extractor/pre_processors/vue.rs b/crates/oxide/src/extractor/pre_processors/vue.rs index 119e2a3d2..1b9bf1668 100644 --- a/crates/oxide/src/extractor/pre_processors/vue.rs +++ b/crates/oxide/src/extractor/pre_processors/vue.rs @@ -14,13 +14,12 @@ pub struct Vue; impl PreProcessor for Vue { fn process(&self, content: &[u8]) -> Vec { let mut result = content.to_vec(); - let content_as_str = std::str::from_utf8(content).unwrap(); for (_, [lang, body]) in TEMPLATE_REGEX .captures_iter(content_as_str) .map(|c| c.extract()) { - let replaced = pre_process_input(body.as_bytes(), lang); + let replaced = pre_process_input(body.as_bytes().to_vec(), lang); result = result.replace(body, replaced); } diff --git a/crates/oxide/src/extractor/string_machine.rs b/crates/oxide/src/extractor/string_machine.rs index 9148a1fbf..4fad49faa 100644 --- a/crates/oxide/src/extractor/string_machine.rs +++ b/crates/oxide/src/extractor/string_machine.rs @@ -30,20 +30,20 @@ impl Machine for StringMachine { #[inline] fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { - if Class::Quote != cursor.curr.into() { + if Class::Quote != cursor.curr().into() { return MachineState::Idle; } // Start of a string let len = cursor.input.len(); let start_pos = cursor.pos; - let end_char = cursor.curr; + let end_char = cursor.curr(); cursor.advance(); while cursor.pos < len { - match cursor.curr.into() { - Class::Escape => match cursor.next.into() { + match cursor.curr().into() { + Class::Escape => match cursor.next().into() { // An escaped whitespace character is not allowed Class::Whitespace => return MachineState::Idle, @@ -52,7 +52,7 @@ impl Machine for StringMachine { }, // End of the string - Class::Quote if cursor.curr == end_char => return self.done(start_pos, cursor), + Class::Quote if cursor.curr() == end_char => return self.done(start_pos, cursor), // Any kind of whitespace is not allowed Class::Whitespace => return MachineState::Idle, diff --git a/crates/oxide/src/extractor/utility_machine.rs b/crates/oxide/src/extractor/utility_machine.rs index b9b1038cd..c116ec4e4 100644 --- a/crates/oxide/src/extractor/utility_machine.rs +++ b/crates/oxide/src/extractor/utility_machine.rs @@ -27,12 +27,12 @@ impl Machine for UtilityMachine { #[inline] fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { - match cursor.curr.into() { + match cursor.curr().into() { // LEGACY: Important marker Class::Exclamation => { self.legacy_important = true; - match cursor.next.into() { + match cursor.next().into() { // Start of an arbitrary property // // E.g.: `![color:red]` @@ -78,7 +78,7 @@ impl UtilityMachine { fn parse_arbitrary_property(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { match self.arbitrary_property_machine.next(cursor) { MachineState::Idle => self.restart(), - MachineState::Done(_) => match cursor.next.into() { + MachineState::Done(_) => match cursor.next().into() { // End of arbitrary property, but there is a potential modifier. // // E.g.: `[color:#0088cc]/` @@ -109,7 +109,7 @@ impl UtilityMachine { fn parse_named_utility(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { match self.named_utility_machine.next(cursor) { MachineState::Idle => self.restart(), - MachineState::Done(_) => match cursor.next.into() { + MachineState::Done(_) => match cursor.next().into() { // End of a named utility, but there is a potential modifier. // // E.g.: `bg-red-500/` @@ -140,7 +140,7 @@ impl UtilityMachine { fn parse_modifier(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { match self.modifier_machine.next(cursor) { MachineState::Idle => self.restart(), - MachineState::Done(_) => match cursor.next.into() { + MachineState::Done(_) => match cursor.next().into() { // A modifier followed by a modifier is invalid Class::Slash => self.restart(), diff --git a/crates/oxide/src/extractor/variant_machine.rs b/crates/oxide/src/extractor/variant_machine.rs index 23aa61798..9ec2adc09 100644 --- a/crates/oxide/src/extractor/variant_machine.rs +++ b/crates/oxide/src/extractor/variant_machine.rs @@ -16,7 +16,7 @@ impl Machine for VariantMachine { #[inline] fn next(&mut self, cursor: &mut cursor::Cursor<'_>) -> MachineState { - match cursor.curr.into() { + match cursor.curr().into() { // Start of an arbitrary variant // // E.g.: `[&:hover]:` @@ -48,7 +48,7 @@ impl VariantMachine { start_pos: usize, cursor: &mut cursor::Cursor<'_>, ) -> MachineState { - match cursor.next.into() { + match cursor.next().into() { // End of an arbitrary value, must be followed by a `:` // // E.g.: `[&:hover]:` diff --git a/crates/oxide/src/fast_skip.rs b/crates/oxide/src/fast_skip.rs index 488e61bd6..54930548c 100644 --- a/crates/oxide/src/fast_skip.rs +++ b/crates/oxide/src/fast_skip.rs @@ -10,7 +10,7 @@ pub fn fast_skip(cursor: &Cursor) -> Option { return None; } - if !cursor.curr.is_ascii_whitespace() { + if !cursor.curr().is_ascii_whitespace() { return None; } diff --git a/crates/oxide/src/scanner/detect_sources.rs b/crates/oxide/src/scanner/detect_sources.rs index 9dc340fba..c519cd00d 100644 --- a/crates/oxide/src/scanner/detect_sources.rs +++ b/crates/oxide/src/scanner/detect_sources.rs @@ -30,11 +30,9 @@ fn sort_by_dir_and_name(a: &DirEntry, z: &DirEntry) -> Ordering { pub fn resolve_globs( base: PathBuf, - dirs: &[PathBuf], + dirs: &FxHashSet, extensions: &FxHashSet, ) -> Vec { - let allowed_paths: FxHashSet = FxHashSet::from_iter(dirs.iter().cloned()); - // A list of known extensions + a list of extensions we found in the project. let mut found_extensions: FxHashSet = FxHashSet::from_iter(KNOWN_EXTENSIONS.iter().map(|x| x.to_string())); @@ -72,7 +70,7 @@ pub fn resolve_globs( continue; } - if !allowed_paths.contains(path) { + if !dirs.contains(path) { let mut path = path; while let Some(parent) = path.parent() { if parent == base { @@ -113,7 +111,7 @@ pub fn resolve_globs( continue; } - if !allowed_paths.contains(path) { + if !dirs.contains(path) { continue; } diff --git a/crates/oxide/src/scanner/init_tracing.rs b/crates/oxide/src/scanner/init_tracing.rs new file mode 100644 index 000000000..2a570ff20 --- /dev/null +++ b/crates/oxide/src/scanner/init_tracing.rs @@ -0,0 +1,70 @@ +use std::fs::OpenOptions; +use std::io::{self, Write}; +use std::path::Path; +use std::sync::{self, Arc, Mutex}; +use tracing_subscriber::fmt::writer::BoxMakeWriter; + +pub static SHOULD_TRACE: sync::LazyLock = sync::LazyLock::new( + || matches!(std::env::var("DEBUG"), Ok(value) if value.eq("*") || (value.contains("tailwindcss:oxide") && !value.contains("-tailwindcss:oxide"))), +); + +fn dim(input: &str) -> String { + format!("\u{001b}[2m{input}\u{001b}[22m") +} + +fn blue(input: &str) -> String { + format!("\u{001b}[34m{input}\u{001b}[39m") +} + +fn highlight(input: &str) -> String { + format!("{}{}{}", dim(&blue("`")), blue(input), dim(&blue("`"))) +} + +struct MutexWriter(Arc>); + +impl Write for MutexWriter { + fn write(&mut self, buf: &[u8]) -> io::Result { + self.0.lock().unwrap().write(buf) + } + + fn flush(&mut self) -> io::Result<()> { + self.0.lock().unwrap().flush() + } +} + +pub fn init_tracing() { + if !*SHOULD_TRACE { + return; + } + + let file_path = format!("tailwindcss-{}.log", std::process::id()); + let file = OpenOptions::new() + .create(true) + .append(true) + .open(&file_path) + .unwrap_or_else(|_| panic!("Failed to open {file_path}")); + + let file_path = Path::new(&file_path); + let absolute_file_path = dunce::canonicalize(file_path) + .unwrap_or_else(|_| panic!("Failed to canonicalize {file_path:?}")); + eprintln!( + "{} Writing debug info to: {}\n", + dim("[DEBUG]"), + highlight(absolute_file_path.as_path().to_str().unwrap()) + ); + + let file = Arc::new(Mutex::new(file)); + + let writer: BoxMakeWriter = BoxMakeWriter::new({ + let file = file.clone(); + move || Box::new(MutexWriter(file.clone())) as Box + }); + + _ = tracing_subscriber::fmt() + .with_max_level(tracing::Level::INFO) + .with_span_events(tracing_subscriber::fmt::format::FmtSpan::ACTIVE) + .with_writer(writer) + .with_ansi(false) + .compact() + .try_init(); +} diff --git a/crates/oxide/src/scanner/mod.rs b/crates/oxide/src/scanner/mod.rs index ec6aea642..6d6cabc2a 100644 --- a/crates/oxide/src/scanner/mod.rs +++ b/crates/oxide/src/scanner/mod.rs @@ -1,5 +1,6 @@ pub mod auto_source_detection; pub mod detect_sources; +pub mod init_tracing; pub mod sources; use crate::extractor::{Extracted, Extractor}; @@ -14,15 +15,13 @@ use bstr::ByteSlice; use fast_glob::glob_match; use fxhash::{FxHashMap, FxHashSet}; use ignore::{gitignore::GitignoreBuilder, WalkBuilder}; +use init_tracing::{init_tracing, SHOULD_TRACE}; use rayon::prelude::*; use std::collections::{BTreeMap, BTreeSet}; -use std::fs::OpenOptions; -use std::io::{self, Write}; use std::path::{Path, PathBuf}; -use std::sync::{self, Arc, Mutex}; +use std::sync::{Arc, Mutex}; use std::time::SystemTime; use tracing::event; -use tracing_subscriber::fmt::writer::BoxMakeWriter; // @source "some/folder"; // This is auto source detection // @source "some/folder/**/*"; // This is auto source detection @@ -34,70 +33,6 @@ use tracing_subscriber::fmt::writer::BoxMakeWriter; // // @source "do-include-me.bin"; // `.bin` is typically ignored, but now it's explicit so should be included // @source "git-ignored.html"; // A git ignored file that is listed explicitly, should be scanned -static SHOULD_TRACE: sync::LazyLock = sync::LazyLock::new( - || matches!(std::env::var("DEBUG"), Ok(value) if value.eq("*") || (value.contains("tailwindcss:oxide") && !value.contains("-tailwindcss:oxide"))), -); - -fn dim(input: &str) -> String { - format!("\u{001b}[2m{input}\u{001b}[22m") -} - -fn blue(input: &str) -> String { - format!("\u{001b}[34m{input}\u{001b}[39m") -} - -fn highlight(input: &str) -> String { - format!("{}{}{}", dim(&blue("`")), blue(input), dim(&blue("`"))) -} - -fn init_tracing() { - if !*SHOULD_TRACE { - return; - } - - let file_path = format!("tailwindcss-{}.log", std::process::id()); - let file = OpenOptions::new() - .create(true) - .append(true) - .open(&file_path) - .unwrap_or_else(|_| panic!("Failed to open {file_path}")); - - let file_path = Path::new(&file_path); - let absolute_file_path = dunce::canonicalize(file_path) - .unwrap_or_else(|_| panic!("Failed to canonicalize {file_path:?}")); - eprintln!( - "{} Writing debug info to: {}\n", - dim("[DEBUG]"), - highlight(absolute_file_path.as_path().to_str().unwrap()) - ); - - let file = Arc::new(Mutex::new(file)); - - let writer: BoxMakeWriter = BoxMakeWriter::new({ - let file = file.clone(); - move || Box::new(MutexWriter(file.clone())) as Box - }); - - _ = tracing_subscriber::fmt() - .with_max_level(tracing::Level::INFO) - .with_span_events(tracing_subscriber::fmt::format::FmtSpan::ACTIVE) - .with_writer(writer) - .with_ansi(false) - .compact() - .try_init(); -} - -struct MutexWriter(Arc>); - -impl Write for MutexWriter { - fn write(&mut self, buf: &[u8]) -> io::Result { - self.0.lock().unwrap().write(buf) - } - - fn flush(&mut self) -> io::Result<()> { - self.0.lock().unwrap().flush() - } -} #[derive(Debug, Clone)] pub enum ChangedContent { @@ -129,26 +64,32 @@ pub struct Scanner { /// The walker to detect all files that we have to scan walker: Option, - /// All changed content that we have to parse - changed_content: Vec, - /// All found extensions extensions: FxHashSet, - /// All CSS files we want to scan for CSS variable usage - css_files: Vec, - /// All files that we have to scan - files: Vec, + files: FxHashSet, /// All directories, sub-directories, etc… we saw during source detection - dirs: Vec, + dirs: FxHashSet, /// All generated globs, used for setting up watchers globs: Option>, /// Track unique set of candidates candidates: FxHashSet, + + /// Track mtimes for files so re-scans can skip unchanged files. + /// Only populated after the first scan completes (to avoid unnecessary + /// metadata calls on initial build). + mtimes: FxHashMap, + + /// Whether we've completed at least one full scan. When false, we skip + /// mtime tracking entirely so the initial build stays fast. + has_scanned_once: bool, + + /// Whether sources have been scanned since the last `scan()` call + sources_scanned: bool, } impl Scanner { @@ -163,7 +104,6 @@ impl Scanner { } let sources = Sources::new(public_source_entries_to_private_source_entries(sources)); - if *SHOULD_TRACE { event!(tracing::Level::INFO, "Optimized sources:"); for source in sources.iter() { @@ -171,29 +111,26 @@ impl Scanner { } } + let walker = create_walker(&sources); + Self { - sources: sources.clone(), - walker: create_walker(sources), + sources, + walker, ..Default::default() } } pub fn scan(&mut self) -> Vec { - self.scan_sources(); + self.sources_scanned = false; - // TODO: performance improvement, bail early if we don't have any changed content - // if self.changed_content.is_empty() { - // return vec![]; - // } + let (scanned_blobs, css_files) = self.discover_sources(); - let _new_candidates = self.extract_candidates(); + self.extract_candidates(scanned_blobs, css_files); - // Make sure we have a sorted list of candidates - let mut candidates = self.candidates.iter().cloned().collect::>(); - candidates.par_sort_unstable(); - - // Return all candidates instead of only the new ones - candidates + // Return all candidates sorted + let mut result = self.candidates.iter().cloned().collect::>(); + result.par_sort_unstable(); + result } #[tracing::instrument(skip_all)] @@ -208,7 +145,7 @@ impl Scanner { // Raw content can be parsed directly, no need to verify if the file exists and is allowed // to be scanned. - self.changed_content.extend(changed_contents); + let mut content_to_scan: Vec = changed_contents; // Fully resolve all files let changed_files = changed_files @@ -232,7 +169,7 @@ impl Scanner { }); // All known files are allowed to be scanned - self.changed_content.extend(known_files); + content_to_scan.extend(known_files); // Figure out if the new unknown files are allowed to be scanned if !new_unknown_files.is_empty() { @@ -252,8 +189,8 @@ impl Scanner { // When the file is found on disk it means that all the rules pass. We can // extract the current file and remove it from the list of passed in files. if file == path { - self.files.push(path.to_path_buf()); // Track for future use - self.changed_content.push(changed_file.clone()); // Track for parsing + self.files.insert(path.to_path_buf()); // Track for future use + content_to_scan.push(changed_file.clone()); // Track for parsing drop_file_indexes.push(idx); } } @@ -274,18 +211,17 @@ impl Scanner { } } - self.extract_candidates() + // Read all content into blobs for extraction + let blobs = read_all_files(content_to_scan); + self.extract_candidates(blobs, vec![]) } #[tracing::instrument(skip_all)] - fn extract_candidates(&mut self) -> Vec { - let changed_content = self.changed_content.drain(..).collect::>(); - - // Extract all candidates from the changed content - let mut new_candidates = parse_all_blobs(read_all_files(changed_content)); + fn extract_candidates(&mut self, blobs: Vec>, css_files: Vec) -> Vec { + // Extract all candidates from the pre-read blobs + let mut new_candidates = parse_all_blobs(blobs); // Extract all CSS variables from the CSS files - let css_files = self.css_files.drain(..).collect::>(); if !css_files.is_empty() { let css_variables = extract_css_variables(read_all_files( css_files @@ -297,63 +233,23 @@ impl Scanner { new_candidates.extend(css_variables); } - // Only compute the new candidates and ignore the ones we already have. This is for - // subsequent calls to prevent serializing the entire set of candidates every time. - let mut new_candidates = new_candidates - .into_par_iter() - .filter(|candidate| !self.candidates.contains(candidate)) - .collect::>(); - - new_candidates.par_sort_unstable(); + // Only keep candidates we haven't seen before + for existing in self.candidates.iter() { + new_candidates.remove(existing); + } // Track new candidates for subsequent calls - self.candidates.par_extend(new_candidates.clone()); + self.candidates.extend(new_candidates.iter().cloned()); - new_candidates - } + let mut result: Vec = new_candidates.into_iter().collect(); + result.par_sort_unstable(); - #[tracing::instrument(skip_all)] - fn scan_sources(&mut self) { - let Some(walker) = &mut self.walker else { - return; - }; - - for entry in walker.build().filter_map(Result::ok) { - let path = entry.into_path(); - let Ok(metadata) = path.metadata() else { - continue; - }; - if metadata.is_dir() { - self.dirs.push(path); - } else if metadata.is_file() { - let extension = path - .extension() - .and_then(|x| x.to_str()) - .unwrap_or_default(); // In case the file has no extension - - match extension { - // Special handing for CSS files, we don't want to extract candidates from - // these files, but we do want to extract used CSS variables. - "css" => { - self.css_files.push(path.clone()); - } - _ => { - self.changed_content.push(ChangedContent::File( - path.to_path_buf(), - extension.to_owned(), - )); - } - } - - self.extensions.insert(extension.to_owned()); - self.files.push(path); - } - } + result } #[tracing::instrument(skip_all)] pub fn get_files(&mut self) -> Vec { - self.scan_sources(); + let _ = self.discover_sources(); self.files .par_iter() @@ -367,7 +263,7 @@ impl Scanner { return globs.clone(); } - self.scan_sources(); + let _ = self.discover_sources(); let mut globs = vec![]; for source in self.sources.iter() { @@ -456,6 +352,99 @@ impl Scanner { }) .collect() } + + #[tracing::instrument(skip_all)] + fn discover_sources(&mut self) -> (Vec>, Vec) { + if self.sources_scanned { + return (vec![], vec![]); + } + self.sources_scanned = true; + + let Some(walker) = &mut self.walker else { + return (vec![], vec![]); + }; + + // Use synchronous walk for the initial build (lower overhead) and parallel + // walk for subsequent calls (watch mode) where the overhead is amortised. + let all_entries = if self.has_scanned_once { + walk_parallel(walker) + } else { + walk_synchronous(walker) + }; + + let mut css_files: Vec = vec![]; + let mut content_paths: Vec<(PathBuf, String)> = Vec::new(); + let mut seen_files: FxHashSet = FxHashSet::default(); + + for (path, is_dir, extension) in all_entries { + if is_dir { + self.dirs.insert(path); + } else { + // Deduplicate: parallel walk can visit the same file from multiple threads + if !seen_files.insert(path.clone()) { + continue; + } + + // On re-scans, check mtime to skip unchanged files. + // On the first scan we skip this entirely to avoid extra + // metadata syscalls. + let changed = if self.has_scanned_once { + let current_mtime = std::fs::metadata(&path) + .ok() + .and_then(|m| m.modified().ok()); + + match current_mtime { + Some(mtime) => { + let prev = self.mtimes.insert(path.clone(), mtime); + prev.is_none_or(|prev| prev != mtime) + } + None => true, + } + } else { + true + }; + + match extension.as_str() { + // Special handing for CSS files, we don't want to extract candidates from + // these files, but we do want to extract used CSS variables. + "css" => { + if changed { + css_files.push(path.clone()); + } + } + _ => { + if changed { + content_paths.push((path.clone(), extension.clone())); + } + } + } + + self.extensions.insert(extension); + self.files.insert(path); + } + } + + // Read + preprocess all discovered files in parallel + let scanned_blobs: Vec> = content_paths + .into_par_iter() + .filter_map(|(path, ext)| { + let content = std::fs::read(&path).ok()?; + event!(tracing::Level::INFO, "Reading {:?}", path); + let processed = pre_process_input(content, &ext); + if processed.is_empty() { + None + } else { + Some(processed) + } + }) + .collect(); + + if !self.has_scanned_once { + self.has_scanned_once = true; + } + + (scanned_blobs, css_files) + } } fn read_changed_content(c: ChangedContent) -> Option> { @@ -474,26 +463,26 @@ fn read_changed_content(c: ChangedContent) -> Option> { ChangedContent::Content(contents, extension) => (contents.into_bytes(), extension), }; - Some(pre_process_input(&content, &extension)) + Some(pre_process_input(content, &extension)) } -pub fn pre_process_input(content: &[u8], extension: &str) -> Vec { +pub fn pre_process_input(content: Vec, extension: &str) -> Vec { use crate::extractor::pre_processors::*; match extension { - "clj" | "cljs" | "cljc" => Clojure.process(content), - "heex" | "eex" | "ex" | "exs" => Elixir.process(content), - "cshtml" | "razor" => Razor.process(content), - "haml" => Haml.process(content), - "json" => Json.process(content), - "md" | "mdx" => Markdown.process(content), - "pug" => Pug.process(content), - "rb" | "erb" => Ruby.process(content), - "slim" | "slang" => Slim.process(content), - "svelte" => Svelte.process(content), - "rs" => Rust.process(content), - "vue" => Vue.process(content), - _ => content.to_vec(), + "clj" | "cljs" | "cljc" => Clojure.process(&content), + "heex" | "eex" | "ex" | "exs" => Elixir.process(&content), + "cshtml" | "razor" => Razor.process(&content), + "haml" => Haml.process(&content), + "json" => Json.process(&content), + "md" | "mdx" => Markdown.process(&content), + "pug" => Pug.process(&content), + "rb" | "erb" => Ruby.process(&content), + "slim" | "slang" => Slim.process(&content), + "svelte" => Svelte.process(&content), + "rs" => Rust.process(&content), + "vue" => Vue.process(&content), + _ => content, } } @@ -512,23 +501,23 @@ fn read_all_files(changed_content: Vec) -> Vec> { } #[tracing::instrument(skip_all)] -fn extract_css_variables(blobs: Vec>) -> Vec { +fn extract_css_variables(blobs: Vec>) -> FxHashSet { extract(blobs, |mut extractor| { extractor.extract_variables_from_css() }) } #[tracing::instrument(skip_all)] -fn parse_all_blobs(blobs: Vec>) -> Vec { +fn parse_all_blobs(blobs: Vec>) -> FxHashSet { extract(blobs, |mut extractor| extractor.extract()) } #[tracing::instrument(skip_all)] -fn extract(blobs: Vec>, handle: H) -> Vec +fn extract(blobs: Vec>, handle: H) -> FxHashSet where H: Fn(Extractor) -> Vec + std::marker::Sync, { - let mut result: Vec<_> = blobs + blobs .par_iter() .flat_map(|blob| blob.par_split(|x| *x == b'\n')) .filter_map(|blob| { @@ -554,22 +543,97 @@ where }) .into_iter() .map(|s| unsafe { String::from_utf8_unchecked(s.to_vec()) }) - .collect(); - - // SAFETY: Unstable sort is faster and in this scenario it's also safe because we are - // guaranteed to have unique candidates. - result.par_sort_unstable(); - - result + .collect() } -/// Create a walker for the given sources to detect all the files that we have to scan. +type WalkEntry = (PathBuf, bool, String); + +/// Walk the file system synchronously. Used for the initial build where the overhead of spawning +/// parallel walker threads is not worth it. +#[tracing::instrument(skip_all)] +fn walk_synchronous(walker: &mut WalkBuilder) -> Vec { + let mut entries = Vec::new(); + + for entry in walker.build().filter_map(Result::ok) { + let is_dir = entry.file_type().map(|ft| ft.is_dir()).unwrap_or(false); + let path = entry.into_path(); + + if is_dir { + entries.push((path, true, String::new())); + } else { + let ext = path + .extension() + .and_then(|x| x.to_str()) + .unwrap_or_default() + .to_owned(); + entries.push((path, false, ext)); + } + } + + entries +} + +/// Walk the file system in parallel. Used in watch mode where the parallel walker overhead is +/// amortised across many rebuilds and subsequent calls are much faster. +#[tracing::instrument(skip_all)] +fn walk_parallel(walker: &mut WalkBuilder) -> Vec { + struct FlushOnDrop { + local: Vec, + shared: Arc>>, + } + + impl Drop for FlushOnDrop { + fn drop(&mut self) { + if !self.local.is_empty() { + self.shared.lock().unwrap().append(&mut self.local); + } + } + } + + let collected: Arc>> = Arc::new(Mutex::new(Vec::new())); + + walker.build_parallel().run(|| { + let mut buf = FlushOnDrop { + local: Vec::with_capacity(256), + shared: collected.clone(), + }; + + Box::new(move |entry| { + let Ok(entry) = entry else { + return ignore::WalkState::Continue; + }; + + let is_dir = entry.file_type().map(|ft| ft.is_dir()).unwrap_or(false); + let path = entry.into_path(); + + if is_dir { + buf.local.push((path, true, String::new())); + } else { + let ext = path + .extension() + .and_then(|x| x.to_str()) + .unwrap_or_default() + .to_owned(); + buf.local.push((path, false, ext)); + } + + if buf.local.len() >= 256 { + buf.shared.lock().unwrap().append(&mut buf.local); + } + + ignore::WalkState::Continue + }) + }); + + // All threads have finished and flushed their buffers via FlushOnDrop::drop + Arc::try_unwrap(collected).unwrap().into_inner().unwrap() +} + +/// Sets up a WalkBuilder with all source roots, gitignore rules, and source pattern matching. /// -/// The `mtimes` map is used to keep track of the last modified time of each file. This is used to -/// determine if a file or folder has changed since the last scan and we can skip folders that -/// haven't changed. -fn create_walker(sources: Sources) -> Option { - let mtimes: Arc>> = Default::default(); +/// This is the common setup shared between the full walker (with mtime tracking for re-scans) +/// and the parallel walker (without mtime tracking for the initial scan). +fn create_walker(sources: &Sources) -> Option { let mut other_roots: FxHashSet<&PathBuf> = FxHashSet::default(); let mut first_root: Option<&PathBuf> = None; let mut ignores: BTreeMap<&PathBuf, BTreeSet> = Default::default(); @@ -717,75 +781,68 @@ fn create_walker(sources: Sources) -> Option { builder.add_gitignore(ignore); } - builder.filter_entry({ - move |entry| { - let path = entry.path(); + // Pre-compute source matching data to avoid allocations in the hot filter_entry path + let auto_bases: Vec = sources + .iter() + .filter_map(|source| match source { + SourceEntry::Auto { base } | SourceEntry::External { base } => Some(base.clone()), + _ => None, + }) + .collect(); - // Ensure the entries are matching any of the provided source patterns (this is - // necessary for manual-patterns that can filter the file extension) - if path.is_file() { - let mut matches = false; - for source in sources.iter() { - match source { - SourceEntry::Auto { base } | SourceEntry::External { base } => { - if path.starts_with(base) { - matches = true; - break; - } - } - SourceEntry::Pattern { base, pattern } => { - let mut pattern = pattern.to_string(); - // Ensure that the pattern is pinned to the base path. - if !pattern.starts_with("/") { - pattern = format!("/{pattern}"); - } + let pattern_sources: Vec<(PathBuf, String)> = sources + .iter() + .filter_map(|source| match source { + SourceEntry::Pattern { base, pattern } => { + let normalized = if pattern.starts_with("/") { + pattern.to_string() + } else { + format!("/{pattern}") + }; + Some((base.clone(), normalized)) + } + _ => None, + }) + .collect(); - // Check if path starts with base, if so, remove the prefix and check the remainder against the pattern - let remainder = path.strip_prefix(base); - if remainder.is_ok_and(|remainder| { - let mut path_str = remainder.to_string_lossy().to_string(); - if !path_str.starts_with("/") { - path_str = format!("/{path_str}"); - } - glob_match(pattern, path_str.as_bytes()) - }) { - matches = true; - break; - } - } - _ => {} - } - } + // Source pattern matching filter (lock-free, safe for parallel walking) + builder.filter_entry(move |entry| { + let path = entry.path(); - if !matches { - return false; + // Ensure the entries are matching any of the provided source patterns (this is + // necessary for manual-patterns that can filter the file extension) + if path.is_file() { + let mut matches = false; + + for base in &auto_bases { + if path.starts_with(base) { + matches = true; + break; } } - let mut mtimes = mtimes.lock().unwrap(); - let current_time = match entry.metadata() { - Ok(metadata) if metadata.is_file() => { - if let Ok(time) = metadata.modified() { - Some(time) - } else { - None + if !matches { + for (base, pattern) in &pattern_sources { + let remainder = path.strip_prefix(base); + if remainder.is_ok_and(|remainder| { + let mut path_str = remainder.to_string_lossy().to_string(); + if !path_str.starts_with("/") { + path_str = format!("/{path_str}"); + } + glob_match(pattern, path_str.as_bytes()) + }) { + matches = true; + break; } } - _ => None, - }; + } - let previous_time = - current_time.and_then(|time| mtimes.insert(entry.clone().into_path(), time)); - - match (current_time, previous_time) { - (Some(current), Some(prev)) if prev == current => false, - _ => { - event!(tracing::Level::INFO, "Discovering {:?}", path); - - true - } + if !matches { + return false; } } + + true }); Some(builder)