Instead of writing debug logs (triggered by using `DEBUG=*`) in the root
of the project as `tailwindcss-<pid>.log`, we will write them to
`.tailwindcss/logs/scanner-<timestamp>-<pid>.log` instead.
The reasoning for this is that we won't pollute the root of the project.
Another reason is that if you don't ignore `.log` files via
`.gitignore`, it could be that you end up in a loop if you use `DEBUG=*
vite` because a new `.log` file might trigger a new build. We noticed
this while looking at
https://github.com/tailwindlabs/tailwindcss/discussions/20382.
The `.tailwindcss` folder will come with a `.gitignore` file that
ignores everything (`*`).
## Test plan
1. Nothing really changed here, all tests should pass
Testing this in a local project, it looks like this:
<img width="483" height="165"
alt="file-35bfc43606c556f2380a161c7595d103"
src="https://github.com/user-attachments/assets/629519c8-8f2b-46ee-8046-9ba3749f319a"
/>
## Summary
Replace the three unchecked scanner conversions in
`crates/oxide/src/scanner/mod.rs` with checked conversions that omit
only extracted slices that are not valid UTF-8. Apply the check in the
shared `extract` pipeline so initial scans, incremental `scan_content`
calls, and CSS-variable extraction cannot insert invalid strings into
scanner state, and apply the same policy to both branches of
`get_candidates_with_positions` while preserving byte offsets and the
legacy `-[]` restoration. Keep the extractor's byte-oriented CSS
identifier classification unchanged: accepting non-ASCII bytes during
extraction is useful for valid multibyte code points, while the
conversion boundary is the authoritative place to enforce the `String`
contract.
The scanner currently converts extracted byte slices with unchecked
UTF-8 constructors at the shared extraction boundary and both
candidate-with-position branches. A source file containing a stray
continuation byte can therefore produce an invalid `String`, violating
Rust's string invariant and allowing corrupted candidates to persist in
a long-lived scanner. The thread provides a deterministic reproduction
using invalid bytes inside an arbitrary value, so the problem no longer
depends on reproducing the originally reported Turbopack race. Valid
candidates found alongside malformed byte sequences must continue to be
returned normally.
Fixes#20368
## Test plan
Not applicable to this change.
AI was used for assistance.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Robin Malfait <malfait.robin@gmail.com>
Preface: For full transparency, I used AI (Claude) to help dig into this
issue we're having and write some of the explanation below.
Our local Laravel application (with Docker) takes between 2-6s seconds
(re)building `app.css` after each Blade file update. After checking with
Claude, there seems to be some potential unnecessary traversing of big
(gitignored) directories, specifically `vendor` and `storage`.
As far as I understand it, two scans happen in tailwindcss:
- a first scan, the content scan. This one reads the project files
(auto-detect). It respects .gitignore, so it correctly skips vendor/ and
storage/.
- a second scan, the glob resolution (resolve_globs() in the oxide
crate). It builds watch patterns that the Vite plugin registers, so a
file save can trigger a CSS rebuild. It walks the project again, but
this walk does not respect .gitignore. It only skips a hardcoded list of
directory names from `ignored-content-dirs.txt` (node_modules, .git,
venv, ...). vendor/ and storage/ are not on that list, so it descends
into them and stats every file. These directories can never produce a
watch pattern, because the content scan never visited them. That work is
perhaps wasted?
Maybe why this went unnoticed: on a native filesystem a stat takes about
1µs, so 57k of them is around 0.06s. A Docker bind mount multiplies that
by roughly 100. And the big directory in a typical JS project is
`node_modules`, which the name list already catches; `vendor/` is the
PHP ecosystem's `node_modules`, and it is not on that list. (#19616 /
#19632 fixed the same problem for `scanner.files`; the CLI never reads
`scanner.globs`, so it was never affected.)
The fix: the second walk now skips any directory the scan never visited,
the same pruning the first walk (line 73) already does.
Measured on our local project: `get_globs()` after `scan()` went from
9.9s to somewhere between 0.2 and 1.0s across runs, with identical
output (35 globs). A template save now reloads in about 0.3s instead of
4.
## Test plan
- `cargo test` in `crates/oxide`: 66 tests pass, including the glob
assertions for nested and gitignored directories in `tests/scanner.rs`.
- Compared the sorted `get_globs()` output before and after on the
project above: identical.
I've also set up a repo that reproduces the issue in a fresh Laravel
app:
https://github.com/marickvantuil/tailwind-scan-reproduce
The results:
| Environment | vendor/ | scan | get_globs | syscalls under vendor/ |
| --- | --- | --- | --- | --- |
| GitHub Actions (native FS) | present | 5.0 ms | 48.8 ms | 11,410 |
| GitHub Actions (native FS) | absent | 4.5 ms | 0.9 ms | n/a |
| macOS, Docker bind mount | present | 15.8 ms | 599.4 ms | 11,410 |
| macOS, Docker bind mount | absent | 16.5 ms | 6.5 ms | n/a |
---------
Co-authored-by: Marick van Tuil <marick@dolphiq.nl>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Robin Malfait <malfait.robin@gmail.com>
## Summary
Fixes#20386.
Ruby lets you close a `%w`/`%W` literal with any non-alphanumeric
character, but the pre-processor only recognized `[`, `(`, `{`, `#` and
a space. So `%w<bg-green-400>` and `%w|bg-blue-400|` were skipped and
the classes inside them never made it into the output.
The boundary lookup now maps `<` to `>` alongside the other paired
delimiters, and falls back to treating any other non-alphanumeric,
non-whitespace character as its own closing delimiter.
## Test plan
Added pre-processor and extraction cases for `%w<…>`, `%w|…|`, `%w:…:`
and `%w!…!` in `crates/oxide/src/extractor/pre_processors/ruby.rs`. They
fail on `main` and pass with this change.
```
cargo test -p tailwindcss-oxide
```
---------
Co-authored-by: Robin Malfait <malfait.robin@gmail.com>
This PR fixes an issue where changes to a symlinked file wouldn't result
in hot-reload when using `@tailwincdss/vite`. This issue also exists in
the other packages such as `@tailwincdss/postcss`,
`@tailwincdss/webpack` and `@tailwincdss/cli`.
The issue is that we watch the symlinked file, but not the "real" file
for changes. If the source of the symlinked file lives in another folder
that is not covered by auto-source detection or by any of the `@source`
directives, then changes to that file won't trigger a change.
To solve this, if a file is symlinked or lives in a symlinked folder,
then we will make sure that the `scanner.files` contains the real path /
canonicalized path to the real file as well just so we can detect
changes in that file.
Fixes: #20346Closes: #20347
## Test plan
1. Added a regression test for `@tailwindcss/vite`
2. Added tests in the scanner code itself
3. Manually tested on the reproduction:
| | Initial state | After change |
| ---: | --- | --- |
| **Before** | <img width="3200" height="1800"
alt="file-f8616573eec3150ae484fd279204c6d7"
src="https://github.com/user-attachments/assets/2734ecc3-b5b6-420e-820d-8a4a8fcdd7f5"
/> | <img width="3200" height="1800"
alt="file-0e93fd3c2c9694c844b098616a3208d2"
src="https://github.com/user-attachments/assets/fd86859b-420b-476b-80f6-e94d02e8b07b"
/> |
| **After** | <img width="3200" height="1800"
alt="file-f8616573eec3150ae484fd279204c6d7"
src="https://github.com/user-attachments/assets/2734ecc3-b5b6-420e-820d-8a4a8fcdd7f5"
/> | <img width="3200" height="1800"
alt="file-eb7b37430ea1363dddeac7226007afe4"
src="https://github.com/user-attachments/assets/0a95b1a4-a357-491d-b57f-78460c0fc9db"
/> |
[ci-all]
---------
Co-authored-by: Nic <162764842+Nic-Polumeyv@users.noreply.github.com>
This PR fixes an issue where an `@source` pointing to a file in a nested
folder was not scanned when a later `@source` pointed to a file in a
parent folder.
E.g.:
```css
@source "./nested/index.html";
@source "./index.html";
```
When using `@source` pointing to a specific file, then we want to make
sure that we ignore _other_ files since they are not listed explicitly.
To ensure that these patterns don't read other files, we inject a `*`
ignore pattern before it. You can think of the above being expanded to:
```rs
Ignored { base: "/project/src/nested", pattern: "*" }
Pattern { base: "/project/src/nested", pattern: "/index.html" }
Ignored { base: "/project/src", pattern: "*" }
Pattern { base: "/project/src", pattern: "/index.html" }
```
The problem with this is that the `Ignored { base: "/project/src",
pattern: "*" }` pattern results in ignoring the `nested` folder as well.
This means that we never even walk into the `nested` folder, so the
earlier `@source "./nested/index.html"` never matches anything.
We could switch the order in user land, but that's going to be hard to
maintain (and order matters for undoing/redoing earlier rules, so we
can't re-order internally either). Instead, we can scope the ignore
pattern to the _current_ path only, and not deeply nested. In other
words, the pattern should become:
```diff
- *
+ /*
```
It's a very subtle difference, but the pattern from above will now
become:
```diff
Ignored { base: "/project/src/nested", pattern: "*" }
Pattern { base: "/project/src/nested", pattern: "/index.html" }
- Ignored { base: "/project/src", pattern: "*" }
+ Ignored { base: "/project/src", pattern: "/*" }
Pattern { base: "/project/src", pattern: "/index.html" }
```
We already do this when an unrestricted root (e.g. `@source "./nested"`)
lives inside the base of a restricted pattern. This works because every
source base is also its own walk root, and a `/*` pattern only matches
direct children so it can't ignore anything when walking from the nested
root itself.
This PR extends that same check to restricted pattern bases: if another
`@source` pattern has its base nested inside the current base, we emit
`/*` instead of `*`.
Note that we only relax the pattern to `/*` when such a nested root
actually exists. Sibling folders that no `@source` points into (e.g. an
`ignore-me` folder next to `nested`) are still direct children, so they
still match `/*` and are never walked.
Fixes: #20333
## Test plan
1. Added a regression test with the reproduction setup
2. Added a similar test with another sibling folder that should still be
ignored
3. Tested it against the actual reproduction:
Before:
<img width="797" height="463" alt="image"
src="https://github.com/user-attachments/assets/893bb872-f5cf-4c5a-bfb5-2dc1043b5a39"
/>
After:
<img width="794" height="460" alt="image"
src="https://github.com/user-attachments/assets/a59aae45-68ad-4fe3-b21f-b568f535d24f"
/>
Notice that the `text-green-500` now appears as expected.
This PR re-adds the `--poll` option to the `@tailwindcss/cli` that we
had in Tailwind CSS v3, but didn't in Tailwind CSS v4.
In Tailwind CSS v4, we started using `@parcel/watcher` instead of
chokidar for our watcher in the CLI. However, this currently doesn't
support a `--poll` option.
This PR implements our own `--poll` option such that you can use it in
environments where fs events don't work properly (e.g. Docker).
Polling can be enabled by using `--watch --poll`, in this case we will
poll every `250ms` (I'm open for a different default value). You can
also pick your own interval by using `--watch --poll 500` which is
defined in milliseconds.
The `--poll` option will be less efficient than a normal `--watch`. But
if you are in a situation where you can't use `--watch` on its own then
this is a good fallback.
One thing you can do today is run the build command manually. If you do
have some tooling that _does_ work on your machine (such as
[`watchexec`](https://github.com/watchexec/watchexec)) then you can
automatically perform a full build. The biggest downside of this
approach is that you are doing a full build every time, instead of an
incremental build.
With this PR, we try to fix that by still allowing incremental builds.
This should result in the same behavior as the normal `--watch`
function:
1. First run, will trigger a full build
1. When any of the source files changes:
- If all classes were already known, then it will be a no-op, but you
will see a log in the terminal about it.
- If a new class is detected, then the CSS will be updated, but it will
be much more efficient than a full rebuild
1. When the input CSS file changes, or any of its dependencies, then a
full rebuild will be triggered (such that your new `@utility` are
available, and `@theme` values are updated).
The implementation is a little bit more complex just because I didn't
want to spam the terminal output even if we are polling every `250ms`.
In the Oxide scanner we do track the modified times of each file. Every
`250ms` we traverse the file system and skip the files that we know
didn't change (since the mtime is the same). If the file was touched,
then we will parse it again to extract possible Tailwind CSS classes. We
will also track which files were scanned such that we can know whether
we have to trigger a full-rebuild or not (in case the input.css file or
any of its dependencies was changed).
Fixes: #18109Fixes: #18540Fixes: #15750
## Test plan
1. Existing tests pass
2. An integration test has been added for the `--poll` option
3. Tested it on the tailwindcss.com codebase:
<img width="679" height="320" alt="image"
src="https://github.com/user-attachments/assets/af858da5-3e07-448d-86bd-8eacdf3bf8d1"
/>
Annotated:
```
≈ tailwindcss v4.3.2
Done in 105ms Initial build
Done in 3ms Saved a file that resulted in a no-op
Done in 2ms Saved a file that resulted in a no-op
Done in 3ms Saved a file that resulted in a no-op
Done in 53ms Saved a file with a new class
Done in 2ms Saved a file that resulted in a no-op
Done in 2ms Saved a file that resulted in a no-op
Done in 3ms Saved a file that resulted in a no-op
Done in 88ms Saved a the input.css file
Done in 3ms Saved a file that resulted in a no-op
Polling for changes…
```
We check the file system every `250ms` by default, but we won't log to
prevent spamming the terminal.
This PR handles template toolkit syntax as a pre-processor step such
that `%]` and `[%` are seen as valid boundary characters.
This is handled for the `.tt`, `.tt2` and `.tx` file extensions. It's
not handled if this syntax is used in `.html` files because then
everybody pays a pre processor cost even if you don't need this syntax
in most cases.
This now ensures that a `template.tx` like this:
```html
<div class="[% IF $is_open %]bg-white/40[% ELSE %]bg-white/10[% END %]"></div>
<!-- ^^^^^^^^^^^ ^^^^^^^^^^^ -->
```
Extracts the classes in between those conditions correctly.
This also fixes a small issue related to Maud, a template engine for
Rust where conditionals like `p.text-black[condition]` caused the
`text-black` class not to be extracted. This is fixed as part of this PR
because it was commented on the linked issue.
Fixes: #20233
## Test plan
1. Added a new extractor
2. Added regression tests
3. All existing tests pass
This PR fixes an issue where a `@source` pointing to a concrete file
could result in scanning the entire parent folder instead of only
looking for the file we are actually interested in.
When we optimize a `@source`, we move all the static parts of the
pattern to the `base`. This means that a `@source` like this:
```css
@source "../../app.config.ts";
```
Resolves to:
```rs
SourceEntry::Pattern { base: "/Users", pattern: "/app.config.ts" }
```
When walking the `base`, we would only emit an `!app.config.ts` rule.
This means that _everything_ in the `/Users` folder is still walked, and
the result is then thrown away. If you take a look at the `gitignore`
equivalent (which is what we build behind the scenes), then we would
essentially create the following:
```gitignore
!app.config.ts
```
But if you know how `gitignore` files work, then you know that this does
force `app.config.ts` to _not_ be ignored, but it doesn't say anything
about all the other files/folders.
The fix is to restrict the `base` so that we ignore everything in the
folder, and then explicitly re-include only the pattern we care about.
Fixing this would result in the following:
```gitignore
*
!foo.ts
```
In the reproduction from #20255 this takes the build from appearing to
hang (~22s) down to ~20ms on my machine.
There are a few edge cases we have to be careful about:
**Multiple patterns for the same base.** If we have multiple `@source`
directives for the same folder:
```css
@source "./src/foo.ts";
@source "./src/bar.ts";
```
Then blindly emitting `*` for each one would result in this `.gitignore`
equivalent:
```gitignore
*
!foo.ts
*
!bar.ts
```
Notice that the second `*` would end up ignoring `foo.ts` again. To
avoid this, we only emit the `*` rule once per `base`.
**Dynamic parts in intermediate folders.** If the pattern still contains
a `*` in one of its folders:
```css
@source "./src/ba*/*.html";
```
This resolves to:
```rs
SourceEntry::Pattern { base: "/src", pattern: "/ba*/*.html" }
```
If we now inject the `*` rule for `/src`, then we would never walk the
`ba*` folders (e.g. `bar` or `baz`). To fix this, we add inverse rules
for each parent segment of the pattern:
```gitignore
* ← ignore everything
!/ba*/ ← except for the `ba*/` folders, so we walk into them
!/ba*/*.html ← then scan the `*.html` files in them
```
**Bases already covered by a broader source.** If the `base` is already
included (or nested) under an unrestricted source (an `Auto`/`External`
source, or a `Pattern` containing `**`), then we leave it alone.
Restricting it would incorrectly hide siblings that the broader source
is supposed to pick up. For example:
```css
@source "**/*";
@source "./src/components/button.html";
```
Here the `**/*` source should keep auto-detecting every file, so we must
_not_ restrict `src/components` just because there's a more specific
`@source` pointing to it.
Source order is preserved throughout, so later `@source not …` rules can
still override an earlier restricted source.
Fixes: #20255
## Test plan
1. Added unit tests realted to this `sources` logic
2. Added integration like tests for the scanner itself
3. All other tests should pass as-s
4. Should work on each OS [ci-all]
This PR fixes an issue where we were over-scanning because we lost
pattern information when a `@source` ended in `**/*` and contained other
dynamic parts in the glob pattern.
Noticed this while debugging and working on #20214. Let's say you have
the following:
```css
@source './blog/*/foo/bar/baz/**/*';
```
We make sure that folders or patterns ending in `**/*` are converted to
"auto sources", meaning that auto content detection should be used in
these folders.
However, when doing so, we would only take the `base` path of the glob.
And since we have "dynamic" parts (the `*`) in the pattern, that should
not be the case.
So the `@source` from above, would be turned into:
```rs
SourceEntry::Auto { base = "/Users/projects/project/blog" }
```
Notice that we lose all the information related to `/*/foo/bar/baz/`.
While this would technically still work, it also means that we are
scanning **too many files and folders** because we're only interested in
folders in the `blog` folder that also contain `foo/bar/baz`.
If the `*` wasn't there, then this would be correct, because then we
would've moved the `/blog/foo/bar/baz` part to the `base` ahead of time,
and the pattern would just be `/**/*`. That would result in:
```rs
SourceEntry::Auto { base = "/Users/projects/project/blog/foo/bar/baz" }
```
This PR fixes that, by _not_ converting it to an auto-source, and
instead convert it to a pattern:
```rs
SourceEntry::Pattern {
base: "/Users/projects/project/blog",
pattern: "/*/foo/bar/baz/**/*"
}
```
Notice that the static part `blog` is still moved to the base path. But
the rest stays in the `pattern` part as expected.
## Test plan
1. Added a failing test to make sure this doesn't happen anymore
2. Existing tests pass
3. Tested this on the tailwindcss.com codebase using `@source
"../blog/tailwindcss*/**/*";`
```diff
diff --git a/./tailwindcss-55526.log b/./tailwindcss-55527.log
index ce79c5da..03e97598 100644
--- a/./tailwindcss-55526.log
+++ b/./tailwindcss-55527.log
@@ -2,50 +2,9 @@ INFO tailwindcss_oxide::scanner: Provided sources:
INFO tailwindcss_oxide::scanner: Source: PublicSourceEntry { base: "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/app", pattern: "../blog/tailwindcss*/**/*", negated: false }
INFO tailwindcss_oxide::scanner: Source: PublicSourceEntry { base: "/Users/robin/.bun/bin", pattern: "bun", negated: true }
INFO tailwindcss_oxide::scanner: Optimized sources:
-INFO tailwindcss_oxide::scanner: Source: Auto { base: "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog" }
+INFO tailwindcss_oxide::scanner: Source: Pattern { base: "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog", pattern: "/tailwindcss*/**/*" }
INFO tailwindcss_oxide::scanner: Source: Ignored { base: "/Users/robin/.bun/bin", pattern: "/bun" }
INFO discover_sources: tailwindcss_oxide::scanner: enter
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-05-23-headless-ui-v1-6-tailwind-ui-team-management/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-06-23-tailwind-templates-and-all-access/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-08-17-tailwind-framer-motion-template-and-tailwind-jobs/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-09-09-new-personal-website-heroicons-2-headless-ui-v17/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2022-12-15-protocol-api-documentation-template/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2023-04-24-new-changelog-template-and-the-biggest-tailwind-ui-update-ever/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2023-07-18-tailwind-connect-2023-recap/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2023-08-07-meet-studio-our-new-agency-template/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2024-05-24-catalyst-application-layouts/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2024-05-30-prettier-plugin-collapse-whitespace/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2024-06-21-headless-ui-v2-1/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2024-09-12-radiant-a-beautiful-new-marketing-site-template/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/2025-05-14-compass-course-starter-kit/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/automatic-class-sorting-with-prettier/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/building-react-and-vue-support-for-tailwind-ui/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/building-the-tailwind-blog/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/designing-tailwind-ui-ecommerce/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/from-900-to-1-how-we-hired-robin-malfait/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-unstyled-accessible-ui-components/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v1-4/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v1-5/demo.tsx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v1-5/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v1/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v2/examples/HeadlessUIV2Examples.tsx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v2/examples/StateAttributesExample.tsx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v2/examples/anchor-positioning.tsx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/headless-ui-v2/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/heroicons-micro/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/heroicons-v1/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/hiring-a-design-engineer-and-staff-engineer/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/introducing-catalyst/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/introducing-heroicons/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/introducing-linting-for-tailwindcss-intellisense/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/introducing-tailwind-play/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/just-in-time-the-next-generation-of-tailwind-css/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/multi-line-truncation-with-tailwindcss-line-clamp/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/simon-vrachliotis-joins-tailwind-labs/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/standalone-cli/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwind-plus/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwind-ui-ecommerce/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwind-ui-now-with-react-and-vue-support/index.mdx"
INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-1-5/index.mdx"
INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-1-6/index.mdx"
INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-1-7/index.mdx"
@@ -68,10 +27,4 @@ INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/t
INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-v4-beta/index.mdx"
INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-v4/color-palette.tsx"
INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/tailwindcss-v4/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/utility-friendly-transitions-with-tailwindui-react/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/vanilla-js-support-for-tailwind-plus/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/welcoming-brad-cornes-to-the-tailwind-team/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/welcoming-david-luhr-to-tailwind-labs/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/welcoming-james-mcdonald-to-tailwind-labs/index.mdx"
-INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/tailwindlabs/tailwindcss.com/src/blog/whats-new-in-tailwindcss-on-youtube/index.mdx"
INFO discover_sources: tailwindcss_oxide::scanner: exit
```
Notice now that a lot of files are skipped as expected since we-re not
accidentally over-scanning now.
[ci-all]
This PR fixes an issue where a `@source` that's pointing to a folder
that is git ignored, is also ignored by the `@source` even if it's
explicitly added.
Internally, we convert `@source` directives from `PublicSourceEntry`s to
`SourceEntry`s where we have dedicated enum branches for `Auto`,
`Pattern`, `Ignored` and `External`.
The `Auto` one accepts a `base` path, and will be used for auto content
detection. However, these paths will make use of all the default auto
content detection rules, which includes git ignore rules.
We also have `External` where we link to something that's "external" to
the current repo. We can probably improve this name, but it's external
in the sense that it won't show up on GitHub for example, aka ignored.
We have some content dirs that we ignore by default, such as the
`node_modules` folder. When you do use `@source` with `node_modules` in
the path, then we will mark it as an `external` resource which does not
look at the `gitignore` related rules and allowing it to be included
this way.
The idea with this is that, even though the folder is ignored by
default, you can still include files from the folder by explicitly using
the `@source` directive.
The issue as seen in #19844 is using `vendor/` instead of
`node_modules/` which is _not_ ignored by default. While we can add
`vendor/` to this same ignored dirs list, it will result in a breaking
change because this folder is often used by the Laravel community to
store some resources in.
This PR fixes this problem by not only looking at the content dirs we
ignore by default, but also looking at the actual git ignore state of
this folder. If it turns out that this is ignored, then we promote the
`Auto` source to an `External` source.
Fixes: #19844Closes: #20057
## Test plan
1. Added integration tests for this situation
2. Ran the fix on the reproduction from #19844. If we run the CLI with
the `DEBUG=*` environment variable, the log file produces these results:
```diff
diff --git a/./tailwindcss-29207.log b/./tailwindcss-30381.log
index bc3017c..921af0e 100644
--- a/./tailwindcss-29207.log
+++ b/./tailwindcss-30381.log
@@ -6,8 +6,9 @@ INFO tailwindcss_oxide::scanner: Source: PublicSourceEntry { base: "/Users/robin
INFO tailwindcss_oxide::scanner: Optimized sources:
INFO tailwindcss_oxide::scanner: Source: Pattern { base: "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/app/design/frontend/theme", pattern: "/**/*.phtml" }
INFO tailwindcss_oxide::scanner: Source: Pattern { base: "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/app/design/frontend/theme", pattern: "/**/*.xml" }
-INFO tailwindcss_oxide::scanner: Source: Auto { base: "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/vendor/acme/theme" }
+INFO tailwindcss_oxide::scanner: Source: External { base: "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/vendor/acme/theme" }
INFO tailwindcss_oxide::scanner: Source: Ignored { base: "/Users/robin/.fnm/node-versions/v26.1.0/installation/bin", pattern: "/node" }
INFO discover_sources: tailwindcss_oxide::scanner: enter
-INFO discover_sources: tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/app/design/frontend/theme/index.phtml"
+INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/app/design/frontend/theme/index.phtml"
+INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/GrimLink/tailwind-gitignore-bug/vendor/acme/theme/module/templates/component.phtml"
INFO discover_sources: tailwindcss_oxide::scanner: exit
```
We're checking some `.gitignore` related files, so let's check on each
OS [ci-all]
This PR fixes an issue when working with `@source` and `@source not`
that involves symlinks.
Internally our sources are mapped to a source entry where we have a
`base` path and a `pattern`. You can think about this where we insert a
`.gitignore` file in the `base` path for the given pattern.
However, we optimize these entries to move as many "static" parts into
the base path. For example:
```css
@source "./some/folder/here/*.html";
```
Is mapped to something like:
```ts
{ base: "/projects/my-project", pattern: "./some/folder/here/*.html" }
```
We then optimize it by turning it into:
```ts
{ base: "/projects/my-project/some/folder/here", pattern: "*.html" }
```
While doing this, we also use `dunce::canonicalize` to resolve the
actual paths on disk. This means that a symlink is resolved to their
real paths. This can cause issues because the "real" path is not what
you wrote in the `@source` directives.
So before, it could be that you have this:
```css
@source "./some/symlinked-folder/here/*.html";
```
Which was mapped to this on the Rust side:
```ts
{ base: "/projects/my-project", pattern: "./some/symlinked-folder/here/*.html" }
```
But was then optimized to:
```ts
{ base: "/projects/my-project/some/actual-folder/here", pattern: "*.html" }
```
...and we lost the `symlinked-folder` information. This causes issues as
seen in #17985.
With this PR, we keep the symlinked information in those globs since
that's what you wrote in those `@source` directives.
While setting up integration tests, I stumbled upon an issue because I
wanted to test that ignoring a symlinked folder, but including a single
particular file of that ignored folder resulted in that file being
ignored as well. Let's look at an example:
```css
@source '../lib';
@source not '../lib/ignored';
@source '../lib/ignored/except.html';
```
Earlier I mentioned that we create `.gitignore` files based on these
`@source` directives. In this case, when we're dealing with a folder, we
use `**/*` as the contents.
Looking at the example above, we should essentially have something like
this:
```gitignore
# lib/.gitignore
# @source '../lib'
!**/*
# lib/ignored/.gitignore
# @source '../lib/ignored'
**/*
# @source '../lib/ignored/except.html'
!except.html
```
Since it's a `.gitignore` file, we have to invert the globs. But the bug
I noticed is that in reality the result of those gitignores didn't look
like the above, it looked like:
```gitignore
# lib/.gitignore
# @source '../lib'
!**/*
# lib/ignored/.gitignore
# @source '../lib/ignored/except.html'
!except.html
# @source '../lib/ignored'
**/*
```
Notice how the `!except.html` and `**/*` are flipped. When dealing with
`.gitignore` files, the order is important.
This was caused because internally we kept a `BTreeMap` of `BTreeSet`s
where the map was the base path and a set of patterns. The patterns were
sorted because of the `BTreeSet`... which is not what we want.
Fixes: #17985Closes: #20091
## Test plan
1. Added new tests in the scanner tests (on the Rust side)
2. Added integration tests with a symlink to another folder, outside of
the current folder
2. Added integration tests with a symlink to another folder, inside of
the current folder
2. Added integration tests to ensure that the order of `@source` files
with a folder + file is sorted correctly.
3. Since we're dealing with symlinks in these tests, let's test all OSes
[ci-all]
## Summary
This PR fixes a panic that occurs when the Ruby or Vue preprocessors
encounter files with invalid UTF-8 bytes.
**The issue:**
- `ruby.rs:37` and `vue.rs:18` used
`std::str::from_utf8(content).unwrap()`
- This panics when processing files containing invalid UTF-8 bytes
**Error message:**
```
thread panicked at crates/oxide/src/extractor/pre_processors/ruby.rs:37:59:
called `Result::unwrap()` on an `Err` value: Utf8Error { valid_up_to: 45, error_len: Some(1) }
```
**The fix:**
- Wrap UTF-8 conversion in `if let Ok(...)` to gracefully handle invalid
UTF-8
- Skip regex-based template extraction when UTF-8 conversion fails
- Allow byte-level processing to continue (in Ruby's case)
This can happen in Rails projects when:
- Binary files are inadvertently scanned
- Files contain non-UTF-8 encodings
- Files are truncated at multi-byte character boundaries during parallel
processing
## Test plan
- [x] Added `test_invalid_utf8_does_not_panic` test for Ruby
preprocessor
- [x] Added `test_valid_utf8_with_multibyte_chars` test for Ruby
preprocessor
- [x] Added `test_invalid_utf8_does_not_panic` test for Vue preprocessor
- [x] All existing tests pass (`cargo test pre_processors` - 43 tests)
---------
Co-authored-by: Robin Malfait <malfait.robin@gmail.com>
This PR fixes an issue where `addClass(opacity-50)` and
`removeClass(opacity-50)` in Twig templates isn't extracted properly.
This fixes that by adding a pre-processor for `.twig` files. This is a
simple initial implementation where we drop the `(` and `)` from the
`addClass` and `removeClass` functions. We don't do any special handling
around escaped characters or parenthesis inside of strings. We will add
them when there is a use case for it. Until then, we'll keep it simple.
If the real API would've been `addClass('opacity-50')` then it would've
worked out of the box.
A workaround you can use today is by using spaces `addClass( opacity-50
)` that works as well.
Fixes: #19458Closes: #20110
## Test plan
1. Added new tests for `twig` extraction
2. Added tests with nested `(` and `)` e.g. `addClass(p-(--value))`
which should properly extarct `p-(--value)`
Fixes a few minor typos across the codebase (e.g. 'overriden' ->
'overridden', 're-use' -> 'reuse').
---------
Co-authored-by: Robin Malfait <malfait.robin@gmail.com>
## Summary
This specializes the `.jsonl` and `.ndjson` file extensions so they're
preprocessed like JSON instead of by the standard scanner. This prevents
them from creating thousands of sub machines and reduces scanning time
(see #17125 where this was done for `.json` files).
It seems reasonable to handle new-line delimited JSON files as well
otherwise scanning these files can take quite a long time.
It's quite unlikely that these will contain classes so, alternatively,
these *could* go in the binary extensions list so they get ignored
entirely.
## Test plan
I ran manual tests inside the `oxide` crate against some large-ish JSONL
files (5MB–15MB). These changes bring down scanning time from 2s–3s on
my M3 Max (via `cargo test --release …`) to less than 20ms.
I also ran tests through a full CLI build pipeline on a low-spec linux
box. This change brought scanning time down from ~90s to ~300ms for a
single ~15MB file.
This PR is an attempt to make the upgrade tooling more stable.
### TL;DR
1. When migrating from Tailwind CSS v3 → Tailwind CSS v4, only migrate
files listed in the `config.content` instead of relying on v4's auto
content detection feature
2. Skip writing files that have not been changed
3. Write changed files in a safe way: first write to a temporary file,
then rename the file atomically
4. Never migrate files that are git ignored, even if they are listed in
the `config.content` file
5. Always ignore `.env` and `.env.*` files when scanning for files. Most
people will have this in their `.gitignore` file, but if not, then this
is a fallback mechanism.
---
Looking at the #18972 issue, it looks like some people are running into
weird situations where some of the contents is just gone.
I have never been able to reproduce this on my own devices and in my own
projects unfortunately. But there is definitely _something_ happening
that's not right that people are running into.
Therefore, this PR is an attempt to fix what I think _might_ be wrong,
but I'm not 100% sure if these fixes are enough, or if something else is
still happening here.
This builds on top of the #19779 PR which has some small fixes, but is
incomplete to make this work.
### What's happening
Looking at some of the comments, it looks like a few things are
happening such as:
1. The upgrade tool is emptying out my files — it looks like these are
only happening if you ctrl+c while the process is taking a while. It
could be that a lot of files are being checked and therefore the tooling
looks like its stuck.
6. The upgrade tool is looking at files it shouldn't look at — in
Tailwind CSS v4 we have this concept of the auto-content detection. This
means that we will look at any plain text file that is not git ignored.
### Fixes
#### Emptying out files
The files being emptied looks like it's because how `fs.writeFile`
behaves by default. It opens the file handle with the `w` flag, which
will first truncate the file before writing the new contents. This is
not a single atomic operation, so a killed process in the middle will
cause invalid state.
When we migrate your template files, everything is happening in promises
to migrate things at the same time. When a lot of files are being
scanned, truncating might have happened already before we write the new
content. Since we migrate a bunch of files in parallel, a ctrl+c could
cause data loss in multiple files.
To mitigate this, I switched to an alternative way of writing files.
1. First, we do some quick checks where if the contents didn't change we
just bail out immediately. Files that don't include Tailwind CSS classes
won't change, and therefore we don't need to override these files with
the same contents.
2. When the migrated contents is empty, we bail out as well. I'm 100%
sure that this is not the spot where the "emptying out" happens, I still
believe it happens in the `writeFile` itself, but added it just in case.
7. Next, I introduced a safe write, where we first write to a temporary
file in the same folder. We could write it to `/tmp`, but then we can't
guarantee that we are on the same file system.
If we ctrl+c at this stage, then the worst case scenario is that you
have additional temporary files in your project, but your original files
are still there.
Once that file was written, we will use the atomic `fs.rename`. This
should be atomic as long as we are on the same file system, so either
the rename didn't happen yet, or it completed.
I added an integration test for this, but I had to change the
`writeFile` implementation slightly. In the test, we will truncate the
file first, after that we will write the new contents. This is so that
we have enough time to kill the current process and allows us to verify
that we didn't clear out the file. Again, this is a hacky way of testing
this, just because I can't reproduce this issue myself, let alone
reproduce it reliable in a CI environment.
Note: we are also using `realpath` to make sure that we are updating the
real file. Otherwise, if we were dealing with a symlinked file, we would
override the symlink with a "hard" copy instead.
#### Touching files that should not be touched
During the migration, we rely on the Tailwind CSS v4 auto detection
logic which means that it will scan any plain text file that is not git
ignored. Therefore changes to php files could happen because in theory
they could contain Tailwind CSS classes.
To solve this, when migrating from Tailwind CSS v3 to Tailwind CSS v4,
we will _only_ take the sources into account that were listed in the
`config.content` array. Since this was a requirement in Tailwind CSS v3,
it should be safe to rely on this array.
Additionally, this will make sure that we are dealing with way fewer
files to migrate as well.
On top of that, files that match the patterns in the content array that
are git ignored will also be skipped. This is to prevent that we mutate
files in `node_modules` for example.
In one of the comments I read that `.env` files were emptied out. In
most cases people will have these files gitignored but I explicitly
added `.env` and `.env.*` as files to never ever touch by default when
scanning.
Last but not least, this also updates the output a little bit of the
upgrade tool in case we skip content files (because of git ignore) and
if we changed a file.
<img width="1122" height="1376" alt="image"
src="https://github.com/user-attachments/assets/318fdbbf-e319-4c7e-9648-ee9283842624"
/>
- "Git ignored folder, skipping: `./node_modules`": this is because the
content array looks like this while the `node_modules` are being
ignored:
<img width="1090" height="398" alt="image"
src="https://github.com/user-attachments/assets/7d694720-5671-47ec-bb2e-f24c5f2c4248"
/>
- "Migrated
`./resources/views/vendor/filament-panels/components/logo.blade.php`":
this is because **I** made a change to showcase this feature.
Fixes: #18972Closes: #19779
### Test plan
1. Existing tests still pass
1. Added a dedicated integration test to ensure that we only take
`config.content` into account when migrating from Tailwind CSS v3 to
Tailwind CSS v4 projects.
1. Added a dedicated integration test to make sure that files listed in
`config.content` that are also git ignored, will still be skipped.
1. Added a dedicated integration test to ensure that when `writeFile` is
cancelled mid-write that our old files are still present.
1. Added a dedicated integration test to ensure that we ignore `.env`
and `.env.*` files even if you didn't git ignore them.
[ci-all] To verify on Windows
---------
Co-authored-by: Sami <sychocouldy@gmail.com>
This PR ensures that interpolated expressions in Ruby syntax are
correctly extracted.
The issue was that we ignore comments in Ruby syntax (which start with
`#`). We already made an exception for locals (`<%# locals: … %>`), but
we also need to handle interpolated expressions (`#{ … }`) in the same
way because they are not comments.
Fixes: #19728
## Test plan
1. Existing tests pass
2. Added a regression test for this scenario
3. Tested using the extractor on the given code snippet:
<img width="1461" height="1856" alt="image"
src="https://github.com/user-attachments/assets/b48f7042-ca9d-4133-87ae-4c37c633c073"
/>
Notice that the `w-100` gets extracted now.
This PR improves the performance of Oxide when scanning large codebases.
The `Oxide` API, looks something like this:
```ts
let scanner = new Scanner({ sources })
let candidates = scanner.scan() // Found candidates
let files = scanner.files // Scanned files
let globs = scanner.globs // Scanned globs
```
The `files` and `globs` are used to tell PostCSS, Vite, webpack etc
which files to watch for changes.
The `.scan()` operation extracts the candidates from the source files.
You can think of these as potential Tailwind CSS classes.
In all these scenarios we have to walk the file system and find files
that match the `sources`.
### 1. Prevent multiple file system walks
The first big win came from the fact that accessing `.files` after a
`.scan()` also does an entire walk of the file system (for the given
`sources`), which is unnecessary because we just walked the file system.
This is something that's not really an issue in smaller codebases
because we have `mtime` tracking. We don't re-scan a file if its `mtime`
hasn't changed since the last scan. However, in large codebases with
thousands of files, even walking the file system to check `mtime`s can
be expensive.
### 2. Use parallel file system walking
Another big win is to use a parallel file system walker instead of a
synchronous one. The big problem here is that the parallel build has
20ms-50ms of overhead which is noticeable on small codebases. We don't
really know if you have a small or big codebase ahead of time, so maybe
some kind of hint in the future would be useful.
So the solution I settled on right now is to use a synchronous walker
for the initial scan, and then switch to a parallel walker for
subsequent scans (think dev mode). This gives us the best of both
worlds: fast initial scan on small codebases, and fast re-scans on large
codebases.
Caveat: if you use the `@tailwindcss/cli` we know exactly which files
changed so we can just re-scan those files directly without walking the
file system at all. But in `@tailwindcss/postcss` we don't know which
files changed, so we have to walk the file system to check `mtime`s.
While this improvement is nice, it resulted in an annoying issue related
to `mtime` tracking. Since the parallel walker processes files in
parallel, the `mtime` was typed as `Arc<Mutex<FxHashMap<PathBuf,
SystemTime>>>` so to avoid locking, I decided to only walk the files
here and collect their paths. Then later we check the `mtime` to know
whether to re-scan them or not.
Initially I just removed the `mtime` tracking altogether. But it did
have an impact when actually extracting candidates from those files, so
I added it back later.
### 3. Delaying work
I was still a bit annoyed by the fact that we had to track `mtime`
values for every file. This seems like annoying overhead, especially
when doing a single build (no dev mode).
So the trick I applied here is to only start tracking `mtime` values
after the initial scan.
This means that, in dev mode, we would do this:
1. Walk entire file system to track files.
2. On a subsequent scan, walk entire file system (again) and start
tracking `mtime` values. This time, we use the parallel walker instead
of the synchronous one.
3. On further scans, only re-scan files whose `mtime` has changed
The trade-off here is that on the second scan we always re-scan all
files, even if they haven't changed. Since this typically only happens
in dev mode, I think this is an acceptable trade-off especially if the
initial build is therefor faster this way.
### 3. Small wins
There are also a few small wins in here that I would like to mention but
that are less significant:
1. Pre-computed normalized `source` patterns instead of in every walker
filter call.
2. Tried to avoid some allocations in various places. For example the
`pre_process_input` always called `content.to_vec()` which allocates.
Instead we now accept an owned `Vec<u8>` so we don't have to call
`.to_vec()` in the default case (in my testing, this is ~92% of the time
in the codebases I checked).
3. Made the `Cursor` struct smaller, which is used a lot during
candidate extraction.
### Benchmarks
Now for the fun stuff, the benchmarks!
<details>
<summary>The code for the benchmarks</summary>
```ts
import path from 'node:path'
import { bench, boxplot, do_not_optimize, run, summary } from 'mitata'
import { Scanner as ScannerPr } from '/path/to/repo/with/pr/branch/tailwindcss/crates/node'
import { Scanner as ScannerMain } from '/path/to/repo/with/main/branch/tailwindcss/crates/node'
let base = '/path/to/some/codebase'
let sources = [{ base, pattern: '**/*', negated: false }]
// Verify the results are the same before benchmarking
let scannerPr = new ScannerPr({ sources })
let scannerMain = new ScannerMain({ sources })
{
let aCandidates = scannerPr.scan()
let bCandidates = scannerMain.scan()
if (aCandidates.length !== bCandidates.length) {
throw new Error(`Mismatch in candidate count: ${aCandidates.length} vs ${bCandidates.length}`)
}
for (let i = 0; i < aCandidates.length; i++) {
if (aCandidates[i] !== bCandidates[i]) {
throw new Error(`Mismatch in candidate at index ${i}: ${aCandidates[i]} vs ${bCandidates[i]}`)
}
}
let aFiles = scannerPr.files
let bFiles = scannerMain.files
if (aFiles.length !== bFiles.length) {
throw new Error(`Mismatch in file count: ${aFiles.length} vs ${bFiles.length}`)
}
for (let i = 0; i < aFiles.length; i++) {
if (aFiles[i] !== bFiles[i]) {
throw new Error(`Mismatch in file at index ${i}: ${aFiles[i]} vs ${bFiles[i]}`)
}
}
console.log('Scanned', aFiles.length, 'files')
console.log('Extracted', aCandidates.length, 'candidates')
console.log('Base =', base)
console.log()
}
summary(() => {
boxplot(() => {
bench('PR (build, .scan()))', function* () {
yield {
[0]() {
return new ScannerPr({ sources })
},
bench(scanner: ScannerPr) {
do_not_optimize(scanner.scan())
},
}
})
bench('main (build, .scan()))', function* () {
yield {
[0]() {
return new ScannerMain({ sources })
},
bench(scanner: ScannerMain) {
do_not_optimize(scanner.scan())
},
}
})
})
})
summary(() => {
boxplot(() => {
bench('PR (build, .scan() + .files)', function* () {
yield {
[0]() {
return new ScannerPr({ sources })
},
bench(scanner: ScannerPr) {
do_not_optimize(scanner.scan())
do_not_optimize(scanner.files)
},
}
})
bench('main (build, .scan() + .files)', function* () {
yield {
[0]() {
return new ScannerMain({ sources })
},
bench(scanner: ScannerMain) {
do_not_optimize(scanner.scan())
do_not_optimize(scanner.files)
},
}
})
})
})
summary(() => {
boxplot(() => {
bench('PR (watch, .scan()))', function* () {
yield {
bench() {
do_not_optimize(scannerPr.scan())
},
}
})
bench('main (watch, .scan()))', function* () {
yield {
bench() {
do_not_optimize(scannerMain.scan())
},
}
})
})
})
summary(() => {
boxplot(() => {
bench('PR (watch, .scan() + .files)', function* () {
yield {
bench() {
do_not_optimize(scannerPr.scan())
do_not_optimize(scannerPr.files)
},
}
})
bench('main (watch, .scan() + .files)', function* () {
yield {
bench() {
do_not_optimize(scannerMain.scan())
do_not_optimize(scannerMain.files)
},
}
})
})
})
await run()
```
</details>
#### tailwindcss.com codebase
```
Scanned 462 files
Extracted 13200 candidates
Base = /Users/robin/github.com/tailwindlabs/tailwindcss.com
clk: ~3.09 GHz
cpu: Apple M1 Max
runtime: bun 1.3.3 (arm64-darwin)
```
In these benchmarks the `PR` one is consistently faster than `main`.
It's not by a lot but that's mainly because the codebase itself isn't
that big. It is a codebase with _a lot_ of candidates though, but not
that many files.
The candidate extraction was already pretty fast, so the wins here
mainly come from avoiding re-walking the file system when accessing
`.files`, and from delaying `mtime` tracking until after the initial
scan.
**Single initial build**:
It's not a lot, but it's a bit faster. This is due to avoiding tracking
the `mtime` values initially and making some small optimizations related
to the struct size and allocations.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (build, .scan())) 22.87 ms/iter 23.28 ms █
(21.49 ms … 25.68 ms) 23.98 ms ▂ ▂ ▂ █ ▂▂
(832.00 kb … 2.69 mb) 1.41 mb ▆▆▆▆█▆▆█▁▆▆█▁▁█▁▆██▁▆
main (build, .scan())) 25.67 ms/iter 26.12 ms █ █ █
(24.54 ms … 27.74 ms) 27.06 ms █ █ █ ███
(432.00 kb … 2.78 mb) 996.00 kb ██▁████▁█▁████▁█▁▁█▁█
┌ ┐
╷ ┌─────┬──┐ ╷
PR (build, .scan())) ├────┤ │ ├─────┤
╵ └─────┴──┘ ╵
╷ ┌─────┬──┐ ╷
main (build, .scan())) ├──┤ │ ├───────┤
╵ └─────┴──┘ ╵
└ ┘
21.49 ms 24.28 ms 27.06 ms
summary
PR (build, .scan()))
1.12x faster than main (build, .scan()))
```
**Single initial build + accessing `.files`**:
We don't have to re-walk the entire file system even if we're just
dealing with ~462 scanned files.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (build, .scan() + .files) 22.54 ms/iter 22.99 ms █ ▂
(21.41 ms … 25.86 ms) 24.26 ms █ ▅ ▅▅█▅ ▅▅
(368.00 kb … 2.05 mb) 853.00 kb █▇█▇▇████▇▇██▁▁▇▁▁▇▁▇
main (build, .scan() + .files) 32.15 ms/iter 32.17 ms █ ▂
(30.78 ms … 36.22 ms) 35.75 ms ▅█ ▅█ ▅
(400.00 kb … 2.45 mb) 952.00 kb ██▁██▇▇█▇▁▁▁▁▁▁▁▁▁▁▁▇
┌ ┐
╷┌──┬┐ ╷
PR (build, .scan() + .files) ├┤ │├───┤
╵└──┴┘ ╵
╷┌───┬ ╷
main (build, .scan() + .files) ├┤ │──────────┤
╵└───┴ ╵
└ ┘
21.41 ms 28.58 ms 35.75 ms
summary
PR (build, .scan() + .files)
1.43x faster than main (build, .scan() + .files)
```
**Watch/dev mode, only scanning**:
This now switches to the parallel walker, but since it's not a super big
codebase we don't see a huge win here yet.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (watch, .scan())) 6.85 ms/iter 7.22 ms █▄
(6.34 ms … 7.94 ms) 7.91 ms ▄██▃
( 64.00 kb … 688.00 kb) 452.82 kb ▃████▆▂▂▁▂▁▂▁▁▁▅█▆▃▅▃
main (watch, .scan())) 7.92 ms/iter 8.08 ms █ █ ▃ █▃▃
(7.41 ms … 8.71 ms) 8.68 ms █▆█▆▃█████
( 0.00 b … 64.00 kb) 19.20 kb ▆▄██████████▆▁▆▆█▄▄▄▆
┌ ┐
╷ ┌──────┬──────┐ ╷
PR (watch, .scan())) ├──┤ │ ├────────────┤
╵ └──────┴──────┘ ╵
╷ ┌───┬──┐ ╷
main (watch, .scan())) ├────┤ │ ├───────────┤
╵ └───┴──┘ ╵
└ ┘
6.34 ms 7.51 ms 8.68 ms
summary
PR (watch, .scan()))
1.16x faster than main (watch, .scan()))
```
**Watch/dev mode, scanning + accessing `.files`**:
Again we avoid re-walking the entire file system when accessing
`.files`.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (watch, .scan() + .files) 12.10 ms/iter 12.74 ms █ █ █ █ ▃▃▃
(10.69 ms … 13.89 ms) 13.81 ms █ █▂▂▂ ▇█▂█▂███▇
(128.00 kb … 10.73 mb) 5.23 mb █▆████▁█████████▆▆▆▆▆
main (watch, .scan() + .files) 14.44 ms/iter 14.74 ms █
(13.93 ms … 15.33 ms) 15.18 ms ███▅ █ ▅ ▅
( 16.00 kb … 80.00 kb) 39.51 kb █▅████▁███▅▁█████▅▁▅▅
┌ ┐
╷ ┌──────┬──────┐ ╷
PR (watch, .scan() + .files) ├──────┤ │ ├─────────┤
╵ └──────┴──────┘ ╵
╷ ┌───┬──┐ ╷
main (watch, .scan() + .files) ├─┤ │ ├───┤
╵ └───┴──┘ ╵
└ ┘
10.69 ms 12.93 ms 15.18 ms
summary
PR (watch, .scan() + .files)
1.19x faster than main (watch, .scan() + .files)
```
#### Synthetic 5000 files codebase
Based on the instructions from #19616 I created a codebase with 5000
files. Each file contains a `flex` class and a unique class like
`content-['/path/to/file']` to ensure we have a decent amount of unique
candidates.
You can test the script yourself by running this:
```
mkdir -p fixtures/app-5000/src/components/{auth,dashboard,settings,profile,notifications,messages,search,navigation,footer,sidebar}/sub{001..500} && for dir in fixtures/app-5000/src/components/*/sub*; do echo "export const Component = () => <div className=\"flex content-['$dir']\">test</div>" > "$dir/index.tsx"; done && find fixtures/app-5000/src/components -type f | wc -lc
```
```
Scanned 5000 files
Extracted 5005 candidates
Base = /Users/robin/github.com/RobinMalfait/playground/scanner-benchmarks/fixtures/app-5000
clk: ~3.08 GHz
cpu: Apple M1 Max
runtime: bun 1.3.3 (arm64-darwin)
```
**Single initial build**:
As expected not a super big win here because it's a single build. But
there is a noticeable improvement.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (build, .scan())) 217.27 ms/iter 211.97 ms █
(205.99 ms … 289.53 ms) 214.33 ms ▅ ▅ ▅▅▅▅█▅▅ ▅
( 3.34 mb … 4.25 mb) 3.72 mb █▁▁▁▁▁▁█▁███████▁▁▁▁█
main (build, .scan())) 249.26 ms/iter 239.88 ms █
(231.51 ms … 381.66 ms) 241.01 ms ▅ ▅ ▅▅ ▅ █▅ ▅▅▅
( 4.22 mb … 4.78 mb) 4.49 mb █▁▁▁▁█▁██▁▁█▁▁██▁▁███
┌ ┐
╷ ┌─────╷──┬
PR (build, .scan())) ├────┤ ┤ │
╵ └─────╵──┴
╷ ┌───────╷
main (build, .scan())) ├───┤ ┤
╵ └───────╵
└ ┘
205.99 ms 223.50 ms 241.01 ms
summary
PR (build, .scan()))
1.15x faster than main (build, .scan()))
```
**Single initial build + accessing `.files`**:
Now things are getting interesting. Almost a 2x speedup by avoiding
re-walking the file system when accessing `.files`.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (build, .scan() + .files) 216.35 ms/iter 214.53 ms █ █ █
(211.00 ms … 242.64 ms) 221.45 ms █ █▅█ ▅▅ ▅ ▅
( 2.97 mb … 4.47 mb) 3.97 mb █▁███▁██▁▁▁▁▁▁█▁▁▁▁▁█
main (build, .scan() + .files) 414.79 ms/iter 406.05 ms ██
(396.72 ms … 542.30 ms) 413.69 ms ▅ ██▅ ▅▅ ▅ ▅ ▅
( 5.19 mb … 6.03 mb) 5.63 mb █▁▁▁███▁██▁█▁█▁▁▁▁▁▁█
┌ ┐
┌┬╷
PR (build, .scan() + .files) ││┤
└┴╵
╷┌──╷
main (build, .scan() + .files) ├┤ ┤
╵└──╵
└ ┘
211.00 ms 312.34 ms 413.69 ms
summary
PR (build, .scan() + .files)
1.92x faster than main (build, .scan() + .files)
```
**Watch/dev mode, only scanning**:
This is where we see bigger wins because now we're using the parallel
walker.
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (watch, .scan())) 76.26 ms/iter 77.41 ms █
(73.56 ms … 79.02 ms) 77.81 ms ▅ ▅ ▅ ▅▅ ▅▅▅ ▅ █
( 2.53 mb … 5.52 mb) 3.06 mb █▁▁▁█▁▁█▁██▁███▁▁▁█▁█
main (watch, .scan())) 166.71 ms/iter 165.14 ms █ █
(161.49 ms … 198.26 ms) 168.99 ms █ ▅█ ▅▅▅ ▅ ▅ ▅
( 1.08 mb … 2.72 mb) 1.24 mb █▁██▁███▁▁█▁▁█▁▁▁▁▁▁█
┌ ┐
╷┬┐
PR (watch, .scan())) ├││
╵┴┘
╷┌─┬╷
main (watch, .scan())) ├┤ │┤
╵└─┴╵
└ ┘
73.56 ms 121.28 ms 168.99 ms
summary
PR (watch, .scan()))
2.19x faster than main (watch, .scan()))
```
**Watch/dev mode, scanning + accessing `.files`**:
This is the biggest win of them all because we have all the benefits
combined:
1. Avoiding re-walking the file system when accessing `.files`
2. Using the parallel walker for faster file system walking
```
benchmark avg (min … max) p75 / p99 (min … top 1%)
--------------------------------------------- -------------------------------
PR (watch, .scan() + .files) 84.04 ms/iter 84.84 ms █
(80.96 ms … 87.53 ms) 87.27 ms ▅▅ ▅▅ ▅▅ █▅ ▅ ▅
( 15.42 mb … 31.34 mb) 22.16 mb ██▁▁▁██▁██▁██▁█▁▁▁▁▁█
main (watch, .scan() + .files) 338.59 ms/iter 353.89 ms █
(321.87 ms … 378.43 ms) 358.70 ms █ █
( 2.39 mb … 2.45 mb) 2.42 mb ███▁▁██▁▁▁▁▁▁▁▁▁▁██▁█
┌ ┐
┬┐
PR (watch, .scan() + .files) ││
┴┘
┌──┬─┐╷
main (watch, .scan() + .files) │ │ ├┤
└──┴─┘╵
└ ┘
80.96 ms 219.83 ms 358.70 ms
summary
PR (watch, .scan() + .files)
4.03x faster than main (watch, .scan() + .files)
```
## Test plan
1. All existing tests still pass
2. All public APIs remain the same
3. In the benchmarks I'm sharing, I first verify that the candidates
returned and the files returned are the same before and after the
change.
4. Benchmarked against real codebases, and against a synthetic large
codebase (5000 files).
Fixes: #19616
Fixes: #19481
This PR improves the Ruby extractor to better handle strict locals. We
recently introduced skipping comments in the Ruby extractor (PR #19243
for #19239) by ignoring comments that start with `#` until the end of
the line.
Strict locals are implemented like this:
```ruby
<%# locals: (css: "text-amber-600") %>
```
Notice the `#` after the `<%`, we considered this a comment and ignored
it.
This PR changes that behavior slightly where we skip comments that are
preceded by `%`. This means that `<%# anything here _will_ be scanned
%>`. This should solve the strict locals case, and normal comments will
still be skipped.
We can be more strict in the future if needed, but I think that this
should be a good solution for both scenarios.
### Test plan
1. Added a test to ensure we extract candidates in strict locals
2. Added a regression test for issue #19239 where we introduced skipping
comments in the Ruby extractor
3. Other existing tests are still passing
We can also verify the extracted candidates:
(it's subtle, but you can see that the class is being extracted now)
<img width="1187" height="1376" alt="image"
src="https://github.com/user-attachments/assets/74bbfd79-9db4-4a5b-bd8d-25f1565c6bfd"
/>
This PR adds support for Maud templates in Rust.
We already had some pre-processing for Rust but for Leptos `class:`
syntax. This PR now added a dedicated Rust pre-processor that handles
Leptos and Maud syntax.
We only start pre-processing Maud templates if the Rust file includes
the `html!` macro.
## Test plan
Looking at the extractor, you can see that we now do extract the proper
classes in Maud templates:
<img width="1076" height="1856" alt="image"
src="https://github.com/user-attachments/assets/e649e1de-289e-466f-8fab-44a938a47dd5"
/>
Fixes: #18984
This PR fixes an issue where the `'` syntax in ClojureScript was not
handled properly, resulting in missing extracted classes.
This PR now supports the following ClojureScript syntaxes:
```cljs
; Keyword
(print 'text-red-500)
; List
(print '(flex flex-col underline))
; Vector
(print '[flex flex-col underline])
```
### Test plan
1. Added regression tests
2. Verified that we extract classes correctly now in various scenarios:
Top is before, bottom is with this PR:
<img width="1335" height="1862" alt="image"
src="https://github.com/user-attachments/assets/746aa073-25f8-41f8-b71c-ba83a33065aa"
/>
Fixes: #18882
This PR fixes an issue where `.hdr` files were scanned for candidates
even though it's a binary file.
As a workaround today, you could use:
```css
@source not "**/*.hdr";
```
To ignore `.hdr` files, but let's bake it in by default instead.
Fixes: #18733
## Summary
Slang is basically a Slim template language for Crystal language, so the
very same Slim parser works fine.
Slim template: https://github.com/slim-template/slim
Slang template: https://github.com/jeromegn/slang
## Test plan
Create a simple slang file with some tailwind-css and check if the CSS
is being extracted:
```slim
doctype html
html
head
title This is a title
body.min-h-screen
header.stick.top-0.z-10
section.max-w-4xl.mx-auto.p-4.flex.items-center.justify-between
h1.text-3xl.font-medium This is a slang file
```
To test it, get any slim template, rename the extension to .slang
Fixes#17851
---------
Co-authored-by: Jordan Pittman <jordan@cryptica.me>
Fixes#18431
The `~W(…)` syntax in Elixir is a _sigil_. It's basically another way to
write strings, arrays of strings, etc… The syntax lets you use a handful
of surrounding brackets like `~W(…)`, `~W[…]`, `~W{…}`, `~W"…"`, etc… to
let you write lists without necessarily having to escape characters.
In v3 our extractor was able to pick these up but in v4 Oxide does not.
I've added a preprocessor for Elixir files so we can modify the code
before our main extractor sees it.
Now things like this: `~W(text-white bg-gray-600)` will get turned into
` ~W text-white bg-gray-600 ` which can easily be processed by our
extractor.
The sigils we support are:
- `~s` / `~S` (strings)
- `~w` / `~W` (word lists)
- `~c` / `~C` (charlists)
We're specifically detecting the use of `(…)`, `[…]`, and `{…}` as using
quotes already works today.
---------
Co-authored-by: Robin Malfait <malfait.robin@gmail.com>
## Summary
In a form like,
```clojure
(if condition :bg-white :bg-black)
```
`:bg-black` will fail to extract, while `:bg-white` is extracted as
expected. This PR fixes this case, implements more comprehensive
candidate filtering, and supersedes a previous PR.
Having recently submitted a PR for handling another special case with
Clojure keywords (the presence of `:` inside of keywords), I thought it
best to invert the previous strategy: Instead of handling special cases
one by one, consume keywords according to the Clojure reader spec.
Consume nothing else, other than strings.
Because of this, this PR is a tad more invasive rather than additive,
for which I apologize. The strategy is this:
- Strings begin with a `"` and ends with an unescaped `"`. Consume
everything between these delimiters (existing case).
- Keywords begin with `:`, and end with whitespace, or one out of a
small set of specific reserved characters. Everything else is a valid
character in a keyword. Consume everything between these delimiters, and
apply the class splitting previously contained in the outer loop. My
previous special case handling of `:` inside of keywords in #18338 is
now redundant (and is removed), as this is a more general solution.
- Discard _everything else_.
I'm hoping that a strategy that is based on Clojure's definition of
strings and keywords will pre-empt any further issues with edge cases.
Closes#18344.
## Test plan
- Added failing tests.
- `cargo test` -> failure
- Added fix
- `cargo test` -> success
---------
Co-authored-by: Jordan Pittman <jordan@cryptica.me>
## Summary
Taking a shot at fixing my own complaint from #18336.
Added failing test and fix for extracting classes containing a
pseudo-class from Clojure keywords. Eg., `:hover:text` ->
`"hover:text"`. This would previously produce `["hover", "text"]`.
## Test plan
- Add a failing test.
- Verify that it fails with `cargo test`.
- Add fix.
- Verify that `cargo test` has no further complaints.
ATT: @RobinMalfait
---------
Co-authored-by: Jordan Pittman <jordan@cryptica.me>
This PR fixes a Haml pre-processing issue where a crash occurs if there
is no trailing `\n` at the end of the file and the code before it was
considered Ruby code.
This happens in situations where Ruby code was used. E.g.:
```
- index = 0
- index += 1
```
In this situation when we see the `-` on the second line, then we will
find the whole indented block and parse it as Ruby code instead. The
block ands at the last `\n`, but since we reach the end of the file,
there is no `\n`. Right now we incorrectly reset the internal cursor to
the last known `\n` position. This means that the `start` of the Ruby
block will be _after_ the `end` of the Ruby block and the Haml parser
will crash.
To solve this, once we reach the end of the file, we don't reset the
cursor to the wrong position.
Fixes:
https://github.com/tailwindlabs/tailwindcss/issues/17379#issuecomment-2910108646
## Test plan
1. Added a regression test that did fail before the fix, and doesn't
anymore
<!--
👋 Hey, thanks for your interest in contributing to Tailwind!
**Please ask first before starting work on any significant new
features.**
It's never a fun experience to have your pull request declined after
investing a lot of time and effort into a new feature. To avoid this
from happening, we request that contributors create a discussion to
first discuss any significant new features.
For more info, check out the contributing guide:
https://github.com/tailwindcss/tailwindcss/blob/main/.github/CONTRIBUTING.md
-->
Fixes#18092
## Summary
Using the Svelte preprocessor for Rust files we can support Leptos
`class:` attributes syntax.
## Test plan
```
pnpm t
```
---------
Co-authored-by: Robin Malfait <malfait.robin@gmail.com>
This PR fixes an issue in the Clojure pre-processor where candidates
including `.` characters were not extracted correctly.
The solution here is to only replace the `.` with a ` ` when the `.` is
not surrounded by numbers. This means that:
```
:.foo.bar
```
Becomes
```
: foo bar
```
But
```
:.gap-1.5.flex
```
Becomes
```
: gap-1.5 flex
```
This way the `gap-1.5` is correctly extracted.
## Test plan
1. Added a test for this case
2. Tested this in the extractor tool as well. Notice how the `gap-1.5`
is correctly extracted here.
<img width="1247" alt="image"
src="https://github.com/user-attachments/assets/f5dd2600-5c5e-4ad8-88af-4e5be44340f5"
/>
Fixes: 17760
This PR fixes a bug where a class like `header_1` wasn't properly
extracted because we didn't allow an `_` before a number.
This PR fixes that by allowing an `_` before a number.
Fixes: #17960
## Test plan
1. Added a test to verify this works
2. Existing tests work
Used the visualizer tool for this to verify that the `header_1` class is
being extracted:
<img width="1816" alt="image"
src="https://github.com/user-attachments/assets/fdc21602-0e2b-4e4e-92a1-19c4f4f5393f"
/>
This PR improves the debug logging by also adding the provided `@source`
to the log file. It will also print the optimized sources (the ones we
expand and mark as `Auto` or `Pattern`).
In the logs, this will look like this:
```
2025-05-09T11:03:32.906478Z INFO tailwindcss_oxide::scanner: Provided sources:
2025-05-09T11:03:32.906544Z INFO tailwindcss_oxide::scanner: Source: PublicSourceEntry { base: "/Users/robin/github.com/RobinMalfait/spreadsheet", pattern: "**/*", negated: false }
2025-05-09T11:03:32.906589Z INFO tailwindcss_oxide::scanner: Optimized sources:
2025-05-09T11:03:32.906595Z INFO tailwindcss_oxide::scanner: Source: Auto { base: "/Users/robin/github.com/RobinMalfait/spreadsheet" }
```
Or if you have more sources:
```
2025-05-09T11:06:54.767546Z INFO tailwindcss_oxide::scanner: Provided sources:
2025-05-09T11:06:54.767660Z INFO tailwindcss_oxide::scanner: Source: PublicSourceEntry { base: "/Users/robin/github.com/RobinMalfait/spreadsheet", pattern: "**/*", negated: false }
2025-05-09T11:06:54.767987Z INFO tailwindcss_oxide::scanner: Source: PublicSourceEntry { base: "/Users/robin/github.com/RobinMalfait/spreadsheet/app", pattern: "./routes/*.{jsx,tsx}", negated: false }
2025-05-09T11:06:54.767992Z INFO tailwindcss_oxide::scanner: Source: PublicSourceEntry { base: "/Users/robin/github.com/RobinMalfait/spreadsheet/app", pattern: "./utils/*.ts", negated: false }
2025-05-09T11:06:54.768450Z INFO tailwindcss_oxide::scanner: Optimized sources:
2025-05-09T11:06:54.768455Z INFO tailwindcss_oxide::scanner: Source: Auto { base: "/Users/robin/github.com/RobinMalfait/spreadsheet" }
2025-05-09T11:06:54.768459Z INFO tailwindcss_oxide::scanner: Source: Pattern { base: "/Users/robin/github.com/RobinMalfait/spreadsheet/app/routes", pattern: "*.jsx" }
2025-05-09T11:06:54.768462Z INFO tailwindcss_oxide::scanner: Source: Pattern { base: "/Users/robin/github.com/RobinMalfait/spreadsheet/app/routes", pattern: "*.tsx" }
2025-05-09T11:06:54.768466Z INFO tailwindcss_oxide::scanner: Source: Pattern { base: "/Users/robin/github.com/RobinMalfait/spreadsheet/app/utils", pattern: "*.ts" }
```
Closes#15452
This PR adds more directories to the list of ignored content dirs. These
are now handled the same as `node_modules`:
- Contents of this directory are ignored by default to avoid scanning
dependency trees
- Explicit `@import`s inside these folders are now treated as
_external_, bypassing any `.gitignore` files.
The new extensions are:
- Version control systems: `.hg`, `.svn`
- Bundler caches: `.venv`, `venv`, `.yarn`
- Framework caches: `.next`, `.turbo`, `.parcel-cache`, `__pycache__`,
`.svelte-kit`
## Test plan
Verified with the repro of #15452 by renaming the ignored directory to
`.venv` and installing a local tarball:
<img width="1283" alt="Screenshot 2025-05-06 at 13 14 55"
src="https://github.com/user-attachments/assets/3265acbb-e121-47b3-ac6c-e174770f8234"
/>
This PR improves the debugability of the scanner when using `DEBUG=*` by
writing to a `tailwindcss-{pid}.log` file in the current working
directory.
It will include all the tracing information from the scanner. This PR
also introduces `Discovering {path}` and `Reading {path}` logs.
- `Discovering {path}` — this is logged when we are traversing the file
system looking for files. We use the `ignore` crate, and log this
information in the `filter_entry` callback. If a file was already
ignored by `.gitignore` files, this won't show up, but it also means
that we will not read it.
- `Reading {path}` — this is when we are actually reading the file so we
can start extracting potential Tailwind CSS classes.
These will give you some insights in what paths are being scanned, and
if we get stuck, where we get stuck.
Also, we are appending to the file. In the log below, you can already
see that a `tailwindcss-<number>.log` file exists already even though it
didn't exist before running the command. This should make it easier to
debug if we get stuck on a specific file/folder because the file will be
populated with information.
There are a few reasons for appending to a file:
1. There is a lot of output, so spamming the stdout/stderr is not ideal
2. If you run the same command again, after changing your `@source`
directives, you could diff the outputs. (although, the timestamps will
be different)
3. When using `DEBUG=*`, a lot of other tools also output debug
information, so writing to a file should make this better.
<details>
<summary>Example log</summary>
```log
2025-05-06T23:13:45.912292Z INFO scan_sources: tailwindcss_oxide::scanner: enter
2025-05-06T23:13:45.912697Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/out.css"
2025-05-06T23:13:45.912716Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/tailwindcss-61347.log"
2025-05-06T23:13:45.912748Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app"
2025-05-06T23:13:45.912786Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/test"
2025-05-06T23:13:45.912814Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/test/utils.ts"
2025-05-06T23:13:45.912851Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/utils"
2025-05-06T23:13:45.912873Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/utils/flatten.ts"
2025-05-06T23:13:45.912884Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/utils/matrix.ts"
2025-05-06T23:13:45.912893Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/utils/default-map.ts"
2025-05-06T23:13:45.912904Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/tailwind.css"
2025-05-06T23:13:45.912914Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/root.tsx"
2025-05-06T23:13:45.912936Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain"
2025-05-06T23:13:45.912962Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/expression.ts"
2025-05-06T23:13:45.912972Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/walk-ast.ts"
2025-05-06T23:13:45.912995Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature"
2025-05-06T23:13:45.913019Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature/parser.test.ts"
2025-05-06T23:13:45.913029Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature/validate.test.ts"
2025-05-06T23:13:45.913039Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature/tokenizer.ts"
2025-05-06T23:13:45.913048Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature/validate.ts"
2025-05-06T23:13:45.913058Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature/parser.ts"
2025-05-06T23:13:45.913067Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature/tokenizer.test.ts"
2025-05-06T23:13:45.913077Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/ast.ts"
2025-05-06T23:13:45.913086Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/version-control.ts"
2025-05-06T23:13:45.913095Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/tokenizer.ts"
2025-05-06T23:13:45.913105Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/type-checker.test.ts"
2025-05-06T23:13:45.913121Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/evaluation-result.ts"
2025-05-06T23:13:45.913505Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/tmp.test.ts"
2025-05-06T23:13:45.913514Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/version-control.test.ts"
2025-05-06T23:13:45.913523Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/tokenizer.test.ts"
2025-05-06T23:13:45.913531Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/evaluation.ts"
2025-05-06T23:13:45.913554Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions"
2025-05-06T23:13:45.913583Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/sequence.test.ts"
2025-05-06T23:13:45.913592Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/date.ts"
2025-05-06T23:13:45.913601Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/lookup.test.ts"
2025-05-06T23:13:45.913613Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/statistics.ts"
2025-05-06T23:13:45.913622Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/intrinsics.test.ts"
2025-05-06T23:13:45.913631Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/math.ts"
2025-05-06T23:13:45.913640Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/engineering.ts"
2025-05-06T23:13:45.913648Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/types.test.ts"
2025-05-06T23:13:45.913656Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/text.ts"
2025-05-06T23:13:45.913665Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/text.test.ts"
2025-05-06T23:13:45.913673Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/intrinsics.ts"
2025-05-06T23:13:45.913681Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/logic.ts"
2025-05-06T23:13:45.913689Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/date.test.ts"
2025-05-06T23:13:45.913697Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/types.ts"
2025-05-06T23:13:45.913705Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/logic.test.ts"
2025-05-06T23:13:45.913713Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/lookup.ts"
2025-05-06T23:13:45.913720Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/sequence.ts"
2025-05-06T23:13:45.913728Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/engineering.test.ts"
2025-05-06T23:13:45.913741Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/index.ts"
2025-05-06T23:13:45.913749Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/math.test.ts"
2025-05-06T23:13:45.913757Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/statistics.test.ts"
2025-05-06T23:13:45.913783Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/__snapshots__"
2025-05-06T23:13:45.913817Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/spreadsheet.test.ts"
2025-05-06T23:13:45.913826Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/type-checker.ts"
2025-05-06T23:13:45.913833Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/function-utils.ts"
2025-05-06T23:13:45.913841Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/spreadsheet.ts"
2025-05-06T23:13:45.913849Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/expression.test.ts"
2025-05-06T23:13:45.913857Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/evaluation.test.ts"
2025-05-06T23:13:45.913879Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/routes"
2025-05-06T23:13:45.913896Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/app/routes/_index.tsx"
2025-05-06T23:13:45.914172Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/README.md"
2025-05-06T23:13:45.914197Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/public"
2025-05-06T23:13:45.914228Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/public/fonts"
2025-05-06T23:13:45.914268Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/package.json"
2025-05-06T23:13:45.914289Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/scripts"
2025-05-06T23:13:45.914310Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/scripts/generate-documentation.ts"
2025-05-06T23:13:45.914332Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/.github"
2025-05-06T23:13:45.914383Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/.github/workflows"
2025-05-06T23:13:45.914410Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/.github/workflows/ci.yml"
2025-05-06T23:13:45.914420Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/tsconfig.json"
2025-05-06T23:13:45.914455Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/vite.config.ts"
2025-05-06T23:13:45.914486Z INFO scan_sources: tailwindcss_oxide::scanner: Discovering "/Users/robin/github.com/RobinMalfait/spreadsheet/biome.json"
2025-05-06T23:13:45.914512Z INFO scan_sources: tailwindcss_oxide::scanner: exit
2025-05-06T23:13:45.914515Z INFO extract_candidates: tailwindcss_oxide::scanner: enter
2025-05-06T23:13:45.914518Z INFO extract_candidates:read_all_files: tailwindcss_oxide::scanner: enter
2025-05-06T23:13:45.914524Z INFO extract_candidates:read_all_files: tailwindcss_oxide::scanner: Reading 58 file(s)
2025-05-06T23:13:45.914808Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/tailwindcss-61347.log"
2025-05-06T23:13:45.914990Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/routes/_index.tsx"
2025-05-06T23:13:45.915138Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/.github/workflows/ci.yml"
2025-05-06T23:13:45.915140Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/statistics.test.ts"
2025-05-06T23:13:45.915145Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature/tokenizer.ts"
2025-05-06T23:13:45.915163Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/tsconfig.json"
2025-05-06T23:13:45.915153Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/engineering.ts"
2025-05-06T23:13:45.915226Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/utils/matrix.ts"
2025-05-06T23:13:45.915229Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/walk-ast.ts"
2025-05-06T23:13:45.915372Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/test/utils.ts"
2025-05-06T23:13:45.915578Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/ast.ts"
2025-05-06T23:13:45.915599Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature/validate.ts"
2025-05-06T23:13:45.915637Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/version-control.ts"
2025-05-06T23:13:45.915647Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature/parser.ts"
2025-05-06T23:13:45.915657Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/vite.config.ts"
2025-05-06T23:13:45.915680Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/types.test.ts"
2025-05-06T23:13:45.915681Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/utils/flatten.ts"
2025-05-06T23:13:45.915701Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature/parser.test.ts"
2025-05-06T23:13:45.915706Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/README.md"
2025-05-06T23:13:45.915691Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/utils/default-map.ts"
2025-05-06T23:13:45.915701Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/spreadsheet.test.ts"
2025-05-06T23:13:45.915730Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/package.json"
2025-05-06T23:13:45.915753Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/tokenizer.ts"
2025-05-06T23:13:45.915731Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/biome.json"
2025-05-06T23:13:45.915787Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature/validate.test.ts"
2025-05-06T23:13:45.915822Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/signature/tokenizer.test.ts"
2025-05-06T23:13:45.915826Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/text.ts"
2025-05-06T23:13:45.915885Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/root.tsx"
2025-05-06T23:13:45.915885Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/types.ts"
2025-05-06T23:13:45.915886Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/type-checker.ts"
2025-05-06T23:13:45.915990Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/type-checker.test.ts"
2025-05-06T23:13:45.915995Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/text.test.ts"
2025-05-06T23:13:45.915998Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/logic.ts"
2025-05-06T23:13:45.916000Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/intrinsics.ts"
2025-05-06T23:13:45.916024Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/sequence.ts"
2025-05-06T23:13:45.916056Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/function-utils.ts"
2025-05-06T23:13:45.916058Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/scripts/generate-documentation.ts"
2025-05-06T23:13:45.916070Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/expression.ts"
2025-05-06T23:13:45.916075Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/logic.test.ts"
2025-05-06T23:13:45.916063Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/date.test.ts"
2025-05-06T23:13:45.916119Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/evaluation-result.ts"
2025-05-06T23:13:45.916120Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/index.ts"
2025-05-06T23:13:45.916148Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/expression.test.ts"
2025-05-06T23:13:45.916152Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/engineering.test.ts"
2025-05-06T23:13:45.916193Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/lookup.ts"
2025-05-06T23:13:45.916219Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/tmp.test.ts"
2025-05-06T23:13:45.916228Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/tokenizer.test.ts"
2025-05-06T23:13:45.916245Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/evaluation.test.ts"
2025-05-06T23:13:45.916256Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/spreadsheet.ts"
2025-05-06T23:13:45.916253Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/lookup.test.ts"
2025-05-06T23:13:45.916267Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/sequence.test.ts"
2025-05-06T23:13:45.916287Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/math.test.ts"
2025-05-06T23:13:45.916286Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/version-control.test.ts"
2025-05-06T23:13:45.916317Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/intrinsics.test.ts"
2025-05-06T23:13:45.916323Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/evaluation.ts"
2025-05-06T23:13:45.916354Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/statistics.ts"
2025-05-06T23:13:45.916562Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/date.ts"
2025-05-06T23:13:45.916609Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/domain/functions/math.ts"
2025-05-06T23:13:45.916676Z INFO extract_candidates:read_all_files: tailwindcss_oxide::scanner: exit
2025-05-06T23:13:45.916682Z INFO extract_candidates:parse_all_blobs: tailwindcss_oxide::scanner: enter
2025-05-06T23:13:45.916688Z INFO extract_candidates:parse_all_blobs:extract: tailwindcss_oxide::scanner: enter
2025-05-06T23:13:45.918271Z INFO extract_candidates:parse_all_blobs:extract: tailwindcss_oxide::scanner: exit
2025-05-06T23:13:45.918282Z INFO extract_candidates:parse_all_blobs: tailwindcss_oxide::scanner: exit
2025-05-06T23:13:45.918286Z INFO extract_candidates:read_all_files: tailwindcss_oxide::scanner: enter
2025-05-06T23:13:45.918288Z INFO extract_candidates:read_all_files: tailwindcss_oxide::scanner: Reading 2 file(s)
2025-05-06T23:13:45.918315Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/app/tailwind.css"
2025-05-06T23:13:45.918504Z INFO tailwindcss_oxide::scanner: Reading "/Users/robin/github.com/RobinMalfait/spreadsheet/out.css"
2025-05-06T23:13:45.918512Z INFO extract_candidates:read_all_files: tailwindcss_oxide::scanner: exit
2025-05-06T23:13:45.918519Z INFO extract_candidates:extract_css_variables: tailwindcss_oxide::scanner: enter
2025-05-06T23:13:45.918522Z INFO extract_candidates:extract_css_variables:extract: tailwindcss_oxide::scanner: enter
2025-05-06T23:13:45.918635Z INFO extract_candidates:extract_css_variables:extract: tailwindcss_oxide::scanner: exit
2025-05-06T23:13:45.918640Z INFO extract_candidates:extract_css_variables: tailwindcss_oxide::scanner: exit
2025-05-06T23:13:45.919059Z INFO extract_candidates: tailwindcss_oxide::scanner: exit
```
</details>
We also output where we are writing the file to. This looks like this
when using the CLI:
<img width="1462" alt="image"
src="https://github.com/user-attachments/assets/79c2cc95-adea-4bbd-a4f1-101de45726f5"
/>
Last but not least, this also ignores `.log` files by default
## Test plan
Ran the CLI (but you can use any tool real, since this is implemented in
Oxide) with the `DEBUG=*` flag.
The file generated, looks like the example I shared above.
This PR fixes a Windows CI issue due to the recently merged #17846
There might be better ways to solve this, but essentially we want to
make sure we are always dealing with `\n` and not `\r\n`. This PR fixes
that by replacing all `\r\n` with `\n` in the tests.
This PR fixes and improves the HAML extractor by ensuring that whenever
we detect lines or blocks of Ruby code (`-`, `=` and `~`) characters at
the beginning of the line, we treat them as Ruby code.
Fixes: #17813
## Test Plan
1. Existing tests pass
2. Changed 1 existing test which embedded Ruby syntax
3. Added a dedicated test to ensure the HAML file in the linked issue is
parsed correctly
Running this in the internal extractor tool you can see that the
`w-[12px]`, `w-[16px]`, `h-[12px]`, and `h-[16px]` are properly
extracted.
Note: the `mr-12px` is also extracted, but not highlighted because this
is not a valid Tailwind CSS class.
<img width="1816" alt="image"
src="https://github.com/user-attachments/assets/fc5929ca-bc71-47d2-b21b-7abeec86f54d"
/>
This PR ignores `.db` files by default. We already ignored `.sqlite` and
`.sqlite3` files but we didn't ignore `.db` files which is a common
extension for database files as well.
Due to the binary nature of `.db` files, scanning these could result in
hard to debug errors such as:
Resolves https://github.com/tailwindlabs/tailwindcss/issues/17699
GeoJSON files are giant JSON files for geographic data structures and
will never contain Tailwind classes, but because they are often huge
they will slow down the build a lot if scanned.
---------
Co-authored-by: Adam Wathan <4323180+adamwathan@users.noreply.github.com>
We generate a glob to ignore binary extensions that looks something like
this:
```
*.{mp4,pages,exe,…}
```
However, if you have a folder that happens to end in `.pages` for
example, then this folder will be ignored in its entirety.
To solve this, we added a new flag to the `Gitignore` struct so we can
register a bunch of ignore rules that _only_ apply to paths and not
folders.
Fixes: #17569
## Test plan
- Added a unit test
---------
Co-authored-by: Philipp Spiess <hello@philippspiess.com>
This PR fixes an issue where the Ruby `%w[…]` syntax causes utilities to
not be properly extracted.
This PR will now ensure that it does get extracted correctly.
Given this input:
```slim
div[
class=%w[bg-blue-500 w-10 h-10]
]
div[
class=%w[w-10 bg-green-500 h-10]
]
```
Before this PR, we extracted everything but the `bg-blue-500`. The
`w-10` was extracted but not because of the second div, but because of
the first one.
Fixes: #17542
## Test plan
1. Added a test to ensure it's working correctly.
Looking at the extractor tool, you can see that it now gets extracted
correctly. Top is before, bottom is with this change.
<img width="1199" alt="image"
src="https://github.com/user-attachments/assets/028d9abd-8917-438c-a423-88ba887b7f97"
/>
Robin Malfait