Skip to content

Fix windows path locator performance. #15177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Eric Snow (ericsnowcurrently) merged 34 commits into from
Feb 1, 2021
Merged

Fix windows path locator performance. #15177

Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
19b3ec8
Activate the Windows $PATH locator.
ericsnowcurrently Jan 19, 2021
2c69245
Allow passing fs.Dirent to convertFileType().
ericsnowcurrently Jan 6, 2021
6aa00e2
Factor out readDir().
ericsnowcurrently Jan 19, 2021
989ded1
Time out after 1 second when reading a directory.
ericsnowcurrently Jan 19, 2021
cdd06f7
Move a comment to the right spot.
ericsnowcurrently Jan 19, 2021
44768a3
Add an optional timeout handler to readDir().
ericsnowcurrently Jan 19, 2021
3d7ae02
Use a timeout handler in WindowsPathEnvVarLocator().
ericsnowcurrently Jan 19, 2021
6bca176
Fix an async bug.
ericsnowcurrently Jan 19, 2021
1089976
Combine identical if blocks.
ericsnowcurrently Jan 19, 2021
87532c8
lint
ericsnowcurrently Jan 20, 2021
4ee4292
Move checkBin to the right spot.
ericsnowcurrently Jan 20, 2021
84b3ced
Remove timeout code.
ericsnowcurrently Jan 20, 2021
cc63376
Drop externalDependencies.listDir().
ericsnowcurrently Jan 20, 2021
4ecb931
iterExecutables() -> walkSubTree().
ericsnowcurrently Jan 20, 2021
2ac0db9
Add iterPythonExecutablesInDir().
ericsnowcurrently Jan 20, 2021
ad8c2b0
Use iterPythonExecutablesInDir() in DirFilesLocator.
ericsnowcurrently Jan 20, 2021
dd1178a
Add isDirWatchable().
ericsnowcurrently Jan 20, 2021
f61e435
Add DirFilesWatchingLocator.
ericsnowcurrently Jan 20, 2021
cb3ca6b
Only watch watchable directories.
ericsnowcurrently Jan 20, 2021
8d325d5
Fix the "watchable" condition.
ericsnowcurrently Jan 21, 2021
4c38d35
Factor out resolvePythonExeGlobs() and make watchLocationForPythonBin…
ericsnowcurrently Jan 21, 2021
ec0690f
Support a flat structure in FSWatchingLocator.
ericsnowcurrently Jan 21, 2021
fb0270e
Watch only a flat structure in DirFilesWatchingLocator.
ericsnowcurrently Jan 21, 2021
5c85900
Fix the FSWatchingLocator tests.
ericsnowcurrently Jan 21, 2021
88b2f84
Add a note about what gets checked.
ericsnowcurrently Jan 26, 2021
9f52799
Do not bother watching any of the $PATH directories.
ericsnowcurrently Jan 26, 2021
7e3175f
Drop DirFilesWatchingLocator.
ericsnowcurrently Jan 26, 2021
5b9b922
Pass PythonEnvStructure in the FSWatchingLocator options.
ericsnowcurrently Jan 28, 2021
4689b17
Use a flat env structure for WindowsStoreLocator.
ericsnowcurrently Jan 28, 2021
1f344c0
isDirWatchable() -> checkDirWatchable() (and return a reason).
ericsnowcurrently Jan 28, 2021
ba9aa1e
Log unwatchable dirs instead of failing.
ericsnowcurrently Jan 28, 2021
dc919d9
Make the unwatchable check before calling startWatchers().
ericsnowcurrently Jan 28, 2021
8f95844
Move the enableGlobalWatchers check earlier.
ericsnowcurrently Jan 28, 2021
1040ca2
Filter out unwatchable roots early in FSWatchingLocator.initWatchers().
ericsnowcurrently Jan 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix the FSWatchingLocator tests.
  • Loading branch information
@ericsnowcurrently
ericsnowcurrently committed Jan 28, 2021
commit 5c85900a64c7b582872acb1ea5211fca93deefd4
87 changes: 65 additions & 22 deletions src/test/pythonEnvironments/discovery/locators/lowLevel/fsWatchingLocator.unit.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

import * as assert from 'assert';
import * as sinon from 'sinon';
import { getOSType, OSType } from '../../../../../client/common/utils/platform';
import { Disposables } from '../../../../../client/common/utils/resourceLifecycle';
import { PythonEnvInfo, PythonEnvKind } from '../../../../../client/pythonEnvironments/base/info';
import { IPythonEnvsIterator } from '../../../../../client/pythonEnvironments/base/locator';
Expand All @@ -14,6 +15,8 @@ import * as externalDeps from '../../../../../client/pythonEnvironments/common/e
import * as binWatcher from '../../../../../client/pythonEnvironments/common/pythonBinariesWatcher';

suite('File System Watching Locator Tests', () => {
const baseDir = '/this/is/a/fake/path';
const callback = async () => Promise.resolve(PythonEnvKind.System);
let inExperimentStub: sinon.SinonStub;
let watchLocationStub: sinon.SinonStub;

Expand All @@ -30,13 +33,13 @@ suite('File System Watching Locator Tests', () => {
});

class TestWatcher extends FSWatchingLocator {
constructor(watcherKind: FSWatcherKind) {
super(
() => '/this/is/a/fake/path',
async () => Promise.resolve(PythonEnvKind.System),
{},
watcherKind,
);
constructor(
watcherKind: FSWatcherKind,
opts: {
noTree?: boolean;
} = {},
) {
super(() => baseDir, callback, opts, watcherKind);
}

public async initialize() {
Expand All @@ -54,24 +57,64 @@ suite('File System Watching Locator Tests', () => {
}
}

const watcherKinds = [FSWatcherKind.Global, FSWatcherKind.Workspace];
const watcherExperiment = [true, false];
['standard', 'flat', 'default'].forEach((structure) => {
let noTree: boolean | undefined = structure === 'standard';
if (structure === 'default') {
noTree = undefined;
}
suite(`${structure} structure`, () => {
const expected =
getOSType() === OSType.Windows
? [
// The first one is the basename glob.
'python.exe',
'*/python.exe',
'*/Scripts/python.exe',
]
: [
// The first one is the basename glob.
'python',
'*/python',
'*/bin/python',
];
if (noTree) {
while (expected.length > 1) {
expected.pop();
}
}

watcherKinds.forEach((watcherKind) => {
watcherExperiment.forEach((experiment) => {
test(`When watcher experiment is ${experiment} and watching ${FSWatcherKind[watcherKind]}`, async () => {
inExperimentStub.resolves(experiment);
const watcherKinds = [FSWatcherKind.Global, FSWatcherKind.Workspace];
const watcherExperiment = [true, false];

const testWatcher = new TestWatcher(watcherKind);
await testWatcher.initialize();
const opts = {
noTree,
};

// Watcher should be called for all workspace locators. For global locators it should be called only if
// the watcher experiment allows it
if ((watcherKind === FSWatcherKind.Global && experiment) || watcherKind === FSWatcherKind.Workspace) {
assert.ok(watchLocationStub.calledOnce);
} else {
assert.ok(watchLocationStub.notCalled);
}
watcherKinds.forEach((watcherKind) => {
suite(`watching ${FSWatcherKind[watcherKind]}`, () => {
watcherExperiment.forEach((experiment) => {
test(`${experiment ? '' : 'not '}in experiment`, async () => {
inExperimentStub.resolves(experiment);

const testWatcher = new TestWatcher(watcherKind, opts);
await testWatcher.initialize();

// Watcher should be called for all workspace locators. For global locators it should be called only if
// the watcher experiment allows it
if (
(watcherKind === FSWatcherKind.Global && experiment) ||
watcherKind === FSWatcherKind.Workspace
) {
assert.equal(watchLocationStub.callCount, expected.length);
expected.forEach((glob) => {
assert.ok(watchLocationStub.calledWithMatch(baseDir, sinon.match.any, glob));
});
} else {
assert.ok(watchLocationStub.notCalled);
}
});
});
});
});
});
});
Expand Down